API: The Rest of the Scryfall Surface — /sets, /catalog/* and /symbology - #922
Open
daveycodez wants to merge 84 commits into
Open
API: The Rest of the Scryfall Surface — /sets, /catalog/* and /symbology#922daveycodez wants to merge 84 commits into
daveycodez wants to merge 84 commits into
Conversation
…o the Back Every multi-face printing used to fan out to one row per face sharing one scryfall_id; the upsert's ON CONFLICT then kept whichever face came last — the back. That is how every battle vanished (t:battle matched zero rows corpus-wide), t:sorcery missed every MDFC spell side, front-face oracle text was unsearchable (jbylund#873), and ~1,343 printings carried the wrong types, colors, and stats (jbylund#400's audit). The fix is the "single conceptual card" jbylund#400 discussion pointed at, and it turns out to be the semantically correct shape, not a compromise: Scryfall ANDs search predicates at the CARD level, each satisfiable by any face — measured against api.scryfall.com (2026-08-08): `t:sorcery t:land` returns the MDFC lands, `o:` conjunctions match across faces (Ral, Monsoon Mage), and `c:b` matches Westvale Abbey's back-face-only color. One row per printing carrying any-face unions reproduces that directly; one row per FACE would instead break every cross-face conjunction (no single face satisfies both terms). - preprocess_card still runs each face through the full pipeline, then _merge_processed_faces collapses the rows: the front supplies identity and display (cmc, mana cost, illustration, image, prices, raw blob — matching Scryfall's own top-level fields); card_types/card_subtypes/ card_colors/card_keywords/produced_mana union across faces; oracle_text and flavor_text join with a newline face separator; type_line joins with " // " exactly as Scryfall renders it. The P/T and loyalty stat groups come from the first face that has them, as a group, so numeric and _text columns never describe different faces. - "Battle" joins CARD_TYPES in the parser: it was absent, so t:battle fell through to the subtype arm on both the SQL and engine paths — a guaranteed zero-match the moment battles became visible. No engine change: the store already had TYPE_BATTLE and reproduced the miss in isolation before the vocabulary fix. - The raw blob is the front face's dict (every existing top-level consumer keeps meaning "the front") with the raw card_faces re-attached; copy_images_to_s3 now uploads card_faces[1]'s image under the face-2 key — and stops writing every face to the face-1 key the upload path had hardcoded. - The site gets a Scryfall-style flip button, as progressive enhancement so the no-JS render (parity fixture) is untouched: a card named "A // B" is probed for a face-2 CDN image, and only cards that have one — transform/MDFC, never split/adventure — grow the button, in the results grid, the modal, and the card page. Measured on the rebuilt dev stack after reimport (97,802 cards): t:battle 0 -> 36 cards / 53 printings, exactly Scryfall's 36; t:sorcery t:land 20, exactly Scryfall's 20; the jbylund#873 commander query 21 -> 34 of Scryfall's 43, and all 9 misses are never-legal draft_innovation cards the corpus excludes by policy; Fire // Ice findable by both halves' text; Brutal Cathar matches t:human again. Known residual, tested and documented: when several faces carry a stat group (2/2 // 3/3), only the front's values are searchable — Scryfall also matches the back's pow=3. Per-face numerics are a follow-up if measurement warrants. Suites: api + parsing + scripts 2,193 passed; jest 1,747 passed (6 new flip tests); ruff + format + prettier clean; app.min.js rebuilt.
… as Result Fields
Five card-data fields consumers need to run their own downstream
filtering, added to both serving paths with matching Scryfall JSON
shapes (names and values as api.scryfall.com emits them):
- layout, rarity: plain text (rarity via a Rust mirror of
magic.rarity_int_to_text)
- cmc: integer
- color_identity: WUBRG-ordered letter list -- the SQL path reshapes the
raw JSONB object post-query, the engine decodes its identity bitmap
directly, and both agree by construction on the canonical order
- legalities: {format: status} object; the engine decodes the packed
u64 through the format registry (every known format, absent ->
"not_legal", printing-level word for the legality-divergent cards,
same rule the filters use), which round-trips what
jsonb_obj_to_legality_bits encoded
DEFAULT_RESULT_FIELDS is unchanged -- the usual 9 stay the default and
these are opt-in via fields=.
Validation: cargo test 155 passed; api/tests/test_engine_unit.py 161
passed including new shape tests against the fixture corpus (Lightning
Bolt layout/cmc/rarity/identity/legalities, WUBRG ordering on
multicolor identities); api/tests/test_api_resource.py 109 passed
including new _identity_letters unit tests; ruff + format clean.
The merge sets the merged row's raw_card_blob to the FRONT FACE's dict with card_faces re-attached, so that existing top-level reads keep meaning "the front". Going through those reads, exactly one needs that: image_uris, which this PR already gives a card_faces->0 fallback in copy_images_to_s3.py. Everything else read from the blob (lang, set_type, games, finishes, frame_effects, image_status, reserved, game_changer) is card-level and identical either way. With that covered, the blob can hold the card verbatim, which is worth more than the promotion. Every searchable field is merged onto the row's own columns, so the blob has no derivation left to do, and keeping it as Scryfall sent it is what makes the card answerable later: a card object cannot be rebuilt from a face. card_faces is gone, name and type_line are the front's, and which fields a real card carries at top level varies by LAYOUT -- a split card has mana_cost and image_uris there, a transform card does not -- so no rule strips the promoted keys correctly for both. Measured on a transform and a split card: the blob now differs from the object Scryfall sent by the lifted card_name and nothing else, where before it had 11 extra top-level keys and three that disagreed (object=card_face, the front's name, the front's type_line). prefer_weights.py's seven image reads go through one front_image() helper that coalesces to card_faces->0; both of its SQL templates were run against the 97,808-card corpus. Five new tests in TestMultiFaceRawBlob pin the invariant for both layouts and fail on the previous behavior. Suites: api + parsing + scripts + client 2,514 passed, jest 1,747 passed, ruff clean.
order= accepted eight things; Scryfall accepts fifteen. This adds released, set, artist, color, eur and tix, leaving only penny and review unsupported. dir=auto is new as well, and it is not a constant: measured against api.scryfall.com, auto means DESCENDING for released, rarity, usd, tix and eur and ascending for everything else -- so rarity and usd, both already supported, were sorting the wrong way for any caller that asked for auto. All of it reaches /search, /cards/search and the in-query order:/sort:/direction:/dir: directives at once, because every route resolves to CardOrdering and SortDirection before a search path sees it. The blocker was orderby_to_col, whose catch-all arm sorts by edhrec: an ordering added to the API and not to SortCol does not raise, it makes the engine and the SQL builder return differently ordered pages for identical input. So each ordering is an engine change first. Three of them needed more than an arm: - released packs yyyymmdd to y*372 + (m-1)*31 + (d-1) before the sort key, which rounds through f32 and is exact only below 2^24. A raw 20260809 is past that, so dates a day or two apart collide -- a failure that reads as "nearly sorted" rather than as an error. released_at_int is unchanged, since the date and year filters compare against it. - color is eleven buckets, measured over 923 cards spanning every colour shape: W U B R G, then multicolour by HOW MANY colours (guild pairs tie), then colourless, then lands. Two of those are not what a bitmask gives -- colourless sorts last rather than first, and lands after it -- so both paths carry the same explicit bucketing rather than an expression over the colours. - set and artist get dense ranks on the printing, assigned post-load exactly as name_rank already is. Neither sorts from what the sort key can reach: a set code is a string, and card_artist_vid is intern order, not alphabetical. ARCHIVE_FORMAT_VERSION bumps so an existing store rebuilds rather than misreading the new fields. Cost is 8 bytes a printing, ~4 MB against the ~305 MB build floor. None of the six has a precomputed sort permutation, so they take the general sort path; colour is card-level and could have one later. AUTO resolves in _search, before either path sees it, so the two always receive the same concrete direction and the cache keys on what actually ran. Tests: test_order_vocabulary.py iterates CardOrdering rather than a hand-written list, so a member added without its counterpart fails. The engine half is behavioural -- a fallen-through ordering returns exactly the edhrec page -- against a 40-card store built so no two orderings agree by accident. test_card_ordering.py gains the same completeness check for the SQL map. The colour expression was run against the 97,808-card dev corpus and produces the eleven buckets in order. Suites: api + parsing + scripts + client 2,591 passed, cargo test 156 passed, ruff clean.
`$ Low` and `$ High` in index.html send `usd-low` / `usd-high`, but PreferOrder spells them `usd_low` / `usd_high`, so picking either has always answered: Invalid value for 'prefer': 'usd-low' (expected one of: default, oldest, newest, usd_low, ...) Confirmed against the live site, not only locally. Two of the six prefer options are unusable from the UI and always have been. Predates the order-vocabulary work on this branch. It surfaced because the guard test in the next commit compares every dropdown against its enum, and this is the first thing that comparison finds. The hyphenated spellings are not wrong everywhere — jbylund#893 accepts `prefer:usd-low` as an in-query directive alias, because that is how Scryfall-shaped queries write it. They are wrong as a query PARAMETER, which is what the dropdown sends.
This branch added artist, color, eur, released, set and tix to CardOrdering but left the dropdown at the original eight, so none of them were reachable from the UI. The API accepted them the whole time, which is exactly why nothing failed. test_frontend_vocabularies.py closes the gap in both directions, for all three dropdowns: - an enum member with no <option> is unreachable from the UI - an <option> with no enum member is a 400 the moment someone picks it Neither shows up anywhere else. The API is happy either way, so the drift is invisible until a user tries it — and it stayed invisible long enough for two prefer options to be broken in production (fixed in the previous commit, found by this test on its first run). Gates: pytest api/ 2,439 passed, 19 xfailed; jest 1,741 passed; ruff clean.
Every transform/MDFC printing on the CDN is wrong today: img/bot/6/1/745.webp --
the FACE-1 key, which the site renders as the front -- is Slicer, High-Speed
Antagonist, the back. Face 2 is a 403. The front image is nowhere.
The cause predates this branch. Until the face merge, a multi-face row WAS its
last face, so raw_card_blob is the back face's dict; Scryfall omits top-level
image_uris on a multi-face card and puts one on each face, so
`raw_card_blob->'image_uris'` resolved to the BACK's images, and process_card
wrote them under a hardcoded face-1 key.
The previous two commits do fix this for rows written from now on: the blob is
the card verbatim, so top-level image_uris is absent on a transform card and the
coalesce falls through to card_faces->0 as intended. What they cannot fix is the
corpus that already exists. Every row imported before them still holds the old
blob, where the first coalesce branch is non-NULL and wrong, so a sync run today
keeps uploading the back face as the front until a full reimport has rewritten
every row.
Reading columns instead removes the dependency entirely:
- the image URL is a pure function of scryfall_id, which is NOT NULL and
uniquely indexed;
- whether a back face exists is card_layout, which is indexed and
lowercase-checked, and correctly excludes split/adventure/flip -- they share
a "//" name and have no second physical face.
That is right on the existing corpus with no reimport, and stays right for rows
whose blob never captured the faces at all. Verified against Scryfall: both
derived URLs for bot/6 return 200, and the front one is Slicer, Hired Muscle.
ONE-TIME REPAIR REQUIRED, and this part no reimport fixes either. The S3 diff is
`db_cards - s3_cards`, so it only uploads what is ABSENT -- and the wrong face-1
objects are present. They will never be replaced by a normal run. Run once with
--no-skip-existing to overwrite them; the flag already exists.
That same append-only diff means a card Scryfall re-scans keeps its stale art
forever. image_updated_at is the natural key for it. Left alone here: a real
bug, but a different one.
The new test pins the query text, which is unusual and deliberate: every other
test in the file mocks fetchall, so all of them pass just as happily against a
query reading the wrong column. That is exactly how this shipped.
The button worked and looked borrowed. Measured against the control it is modelled on -- .card-grid-item-transform-button in Scryfall's own stylesheet -- every value was wrong: position top:1.6em right:1.6em -> over the art's right edge, ~26% down size 2.2em (font-relative) -> 44px fill --color-card-background -> #fff border none -> 2px solid #343242 rest opacity .85 -> opacity .6 hover rotate(180deg) -> opacity 1, no transform Position was the substantive one, and it needed more than copying their numbers. Scryfall's container (.card-grid-item) holds the image and nothing else, so top:26% lands on the art. Ours (.card-item) holds the image PLUS the name, mana, type, text and set rows, so the same percentage measures a box roughly twice as tall and drops the button into the card body. Moving the button inside .card-page-link would fix the geometry and nest a <button> in an <a>, which is invalid and breaks keyboard traversal -- so the tile becomes a query container instead and the offsets are expressed against the image: a card's art is a fixed 488x680, so its height is always 1.3934x the tile's content width, and 26% of that is 36.23cqw. The modal keeps plain percentages, its wrapper being image-only. Theme colours were the other real problem. The control sits ON the artwork, which is neither theme: against a black card border in dark mode it nearly vanished, and against bright art in light mode it read as a hole punched in the card. A white disc with a dark ring resolves against both because it depends on neither. The glyph is now an SVG. A text character's ink box is not its em box, so flex centring aligns the em box and leaves the arrow visibly off-centre -- by an amount that changes with whichever fallback font renders it, so nudging it would have been tuning for one machine. The outline strokes with currentColor, which also gives the inverted state its icon colour for free. 22px inside the 44px disc: Scryfall insets theirs 7px, but their artwork carries padding inside its own viewBox where this one strokes to the edge, so matching the inset would make a visibly heavier mark than theirs. Matching the ink is what reads the same. And it carries STATE now, not just an action: while the back face is showing the button inverts to a dark fill with a white ring (Scryfall's `.spooky`). Once the art has changed, the button is the only thing on screen that still says which face you are looking at. Toggled on click rather than after the transition, so the control answers the press, and mirrored into aria-pressed. The 180-degree hover spin is gone. The glyph already says "turn over"; spinning the control implied the control was the thing that rotates. Sizes and centring were compared by rendering the button at its real proportions rather than judged from the markup. prettier clean.
The button's offsets are percentages of the card image, but nothing it was anchored to IS the image. A grid tile (.card-item) is the image plus the name, mana, type, text and set rows, so a percentage measured from it lands well below the art. The modal's .modal-image-wrapper is a flex area far wider than the picture centred inside it, so on a single large card the button sat out in the margin BESIDE the card rather than on it. The frame cannot be the <a> that already wraps the image: a <button> inside an <a> is interactive content nested in interactive content, invalid and a break in keyboard traversal. So the JS that injects the button now also injects a plain <div class="card-image-frame"> around the link, with width: fit-content so the box is the image's box. Both call sites go through one helper. Injected rather than templated, deliberately: the server render and its parity fixture are untouched, and the frame exists only on the cards that actually get a button. An earlier attempt derived the image's height from the fixed 745x1041 aspect and placed the button with container units. That was right for the grid and would have been wrong in the modal, where the image is sized by the available HEIGHT rather than by width. A box that hugs the image needs no arithmetic and cannot disagree with the image. prettier clean.
The engine serves searches and Postgres is the fallback for when it errors, so a field that lives
only in `raw_card_blob` is a field the primary path cannot answer: a jsonb column is not in the
store, and the store is all an engine request reads. That is the whole reason a Scryfall-compatible
`/cards/*` had to be SQL-only.
Two columns close the gap, both shaped like the jsonb grab-bags this table already has
(card_frame_data, card_is_tags, card_legalities):
card_compat_blob -- every Scryfall key that neither has a column of its own nor is a pure
function of one. Measured on a real card object (Lightning Bolt, 66 keys,
5,881 bytes): the residue is 26 keys and 680 bytes. The blob is
overwhelmingly redundant with columns we already store, plus URLs that are
deterministic in the card's id.
card_faces -- each face's own fields, front first, so a face is something the engine can
read rather than something recoverable only by re-parsing a blob.
The residue is defined SUBTRACTIVELY, in both the migration and _COMPAT_BLOB_EXCLUDED: it is
whatever is left once every stored and derivable key is removed. A Scryfall key nobody has seen yet
therefore lands in the blob by default, instead of being silently dropped the first time it appears.
`prices` stays whole despite price_usd/eur/tix having columns, because usd_foil, usd_etched and
eur_foil do not.
This also retires the residual _merge_processed_faces documents. The merged row keeps only the
FRONT face's stat group (Brutal Cathar's 2/2 // 3/3), while Scryfall matches either face; the
per-face records carry both, so the back is searchable again. Pinned by
test_back_face_stats_survive_the_merge.
Both columns backfill from raw_card_blob, so this deploys without a reimport -- the same reasoning
as 2026-08-06-01-lowercase-keywords.sql. raw_card_blob is kept: the is: tag sync reads it
(api_resource.py), db_info.py declares it a searchable column, and it costs nothing on the fallback
path.
Nine tests in TestEngineCardObjects. Suites: api + card_engine + scripts 2,376 passed, 19 xfailed,
ruff clean.
The face merge (previous commit) is deliberately lossy about WHICH face said what: unions and
joined texts, so any face satisfies a card-level predicate. That is right for searching and not
enough to answer with a card object, which is why /cards/* could only ever be served from
Postgres -- `card_faces` lived in a jsonb column, and a jsonb column is not in the store.
Two archived structs, split the same way the commit pass already splits a row:
OracleFace -- name, mana cost, type line, oracle text, stats, colors, color_indicator. Face
TEXT is identical on every printing of a card, so it hangs off OracleCard.
PrintingFace -- illustration, artist, flavor text. These differ per printing, so they hang off
Printing, parallel by index to the OracleCard's faces.
Empty for the ~82% of cards with one face, where an empty Vec costs a length word.
This also makes per-face stats reachable, retiring the residual _merge_processed_faces documents:
the merged row keeps only the FRONT's stat group (Brutal Cathar's 2/2 // 3/3) while Scryfall
matches either face. faces[i].creature_power_text_id carries both.
ARCHIVE_FORMAT_VERSION 2026080601 -> 2026081001. The struct layout changed, so the header check in
from_aligned rejects a stale archive rather than reading it wrong -- a rebuild is required, and
that is the mechanism that enforces it.
Note the existing comment on Printing.scryfall_id: UUIDs are packed as u128 "so future lookup-by-id
can match Scryfall's". That lookup is the next commit; this one gives it something worth looking up.
Three tests: faces_survive_the_archive_round_trip pins text and art independently through rkyv,
single_faced_cards_carry_no_faces pins the common case, face_indexes_line_up_across_the_split pins
the parallel-index contract between the two halves.
Gates: cargo test -p card_engine 160 passed / 38 ignored; api + card_engine + scripts 2,377 passed,
19 xfailed; ruff clean.
`Printing.scryfall_id` and `OracleCard.oracle_id` already hold the UUID's exact bits -- the comment on that field says why: "so future lookup-by-id can match Scryfall's". They were simply never findable. Every by-id question therefore meant scanning ~98k printings, which is why the routes that ask them are served from Postgres instead of from the store. Two sorted permutations close that, and they are permutations rather than (id, index) tables because the ids are already on the rows: 4 bytes each, ~390 KB and ~127 KB at corpus scale, binary searched. printing_by_scryfall_id -- printing space, ordered by scryfall_id oracle_by_oracle_id -- card space, ordered by oracle_id Both are exposed where a caller can reach them: card_by_scryfall_id(id, fields) -> the one printing, or None printings_of_oracle_id(id, fields) -> every printing of that card, stored order `find_by_sorted_id` rejects 0 before searching. 0 is parse_uuid_or_hash's null and no real id maps to it, so an absent id cannot collide with a stored one even though it sorts first. Three tests: printings_are_findable_by_scryfall_id searches deliberately unordered, non-dense ids and pins both the absent-id and null-id answers; cards_are_findable_by_oracle_id does the same in card space; the_id_permutation_is_a_permutation asserts every row appears exactly once, because an index that silently drops a row 404s a real card. Gates: cargo test -p card_engine 163 passed / 38 ignored, cargo check clean with and without default features; api + card_engine + scripts 2,377 passed, 19 xfailed; ruff clean.
card_compat_blob is written and backfilled but nothing read it, so the fields a Scryfall card
object needs still existed only in Postgres. This lands them in the archive.
Packed rather than kept as a blob, because the ratio is the whole argument: a real card object is
5,881 bytes across 66 keys, of which 25 are already columns and 15 are pure functions of the card's
id/set/oracle id. The residue is 26 keys, and as typed fields it is ~50 bytes a printing --
~5 MB corpus-wide against ~575 MB of blobs, which is the difference between fitting in the store
and not.
- six marketplace/client ids + penny_rank + image_updated_at as Option<u32>
- usd_foil / usd_etched / eur_foil as integer cents, the same convention as the price columns
- lang, image_status, set_type, security_stamp interned (small closed vocabularies)
- games and finishes as bitsets over closed vocabularies
- twelve booleans as one u16, since they are only ever read together
- set_id as u128: per printing rather than per set, because the archive has no set table. ~1.5 MB
and the largest single item here.
Default is hand-written, not derived. absent_compat_values_stay_absent caught the derived version:
it zeroes the interned ids, and vocab id 0 is a REAL string, so a card with no `lang` would report
whatever sits at slot 0. Absent has to be VOCAB_NONE. That mattered more than it looks -- the
missing-blob path returns Default, so every card lacking a compat blob was affected.
The same distinction runs through the extraction: Scryfall OMITS a key rather than sending null, so
"was not there" and "was empty" have to stay separable or a reconstructed object sprouts nulls
Scryfall never sent, and a client comparing shapes sees a difference on every row.
Two tests: compat_fields_survive_the_archive_round_trip pins the values and checks bitset members
independently; absent_compat_values_stay_absent pins the sentinel.
Gates: cargo test -p card_engine 165 passed / 38 ignored, cargo check clean both feature shapes;
api + card_engine + scripts 2,377 passed, 19 xfailed; ruff clean.
# Conflicts: # card_engine/src/lib.rs
PR 1 put the residue and the faces in the store; nothing could read them back out. These are the FIELD_TABLE entries that let a caller ask for them, in Scryfall's own field names, so a card object can be assembled from the engine instead of unwrapped from a jsonb column. 31 entries: lang, image_status, set_type, security_stamp, set_id, the six marketplace ids, penny_rank, image_updated_at, the three foil/etched prices, multiverse_ids, promo_types, frame_effects, games, finishes, the twelve booleans, and card_faces. Absent stays absent throughout. Scryfall OMITS a key it has no value for, so these emit None rather than a zero, and `coll_str_opt` exists precisely because `coll_str` has no absent case -- every collection element is a real vocab entry, while a compat field routinely is not. A card that sprouts nulls Scryfall never sent differs from Scryfall on every row, which is the one thing a drop-in replacement cannot do. card_faces emits text from the oracle card and art from this printing, and omits `object` and `image_uris`: the first is the constant "card_face", the second a pure function of the card's id and the face's position. Both are re-emitted by the caller rather than stored. A printing carrying fewer art records than the card has faces leaves those faces without art rather than borrowing the wrong face's. mask_to_color_letters is new because colors were only ever filtered on, never emitted: answering `c:r` needs the mask, saying what a face IS needs the inverse. Gates: cargo test -p card_engine 165 passed / 38 ignored, cargo check clean; api + card_engine 2,358 passed, 19 xfailed; ruff clean.
Scryfall routes /cards/multiverse|mtgo|arena|tcgplayer|cardmarket/:id, and the store could answer none of them: the ids arrived with the compat residue but nothing indexed them. Sparse (namespace, id, printing) triples rather than five dense columns -- most printings carry two to four of the seven ids, and multiverse_ids is a LIST so one printing contributes several entries. ~350 KB at corpus scale, and one binary search serves all five routes. mtgo_foil_id folds into the mtgo namespace and tcgplayer_etched_id into tcgplayer, because Scryfall resolves those to the same printing rather than to a different card. find_printing_by_external_id walks back to the FIRST match after landing. Etched and nonfoil rows do collide on a TCGplayer id, and binary_search lands on an arbitrary one of them; printings are stored in descending prefer order, so the lowest index is the printing the rest of the API would show. Pinned by a_shared_external_id_resolves_to_the_first_printing. Two tests, including that namespaces are separate keyspaces -- id 200 as an mtgo id must not resolve as an arena one. Gates: cargo test -p card_engine 167 passed / 38 ignored, cargo check clean; api + card_engine 2,358 passed, 19 xfailed; ruff clean.
Scryfall's `all_parts` is on ~41% of cards -- meld halves, meld results, combo pieces, tokens -- and a card object without it differs from Scryfall's on every one of them. Each entry carries its own id, name and type line rather than an index into our cards. That is not redundancy: most of these point OUTSIDE the corpus. `preprocess_card` filters Token and Card type lines, and a token is exactly what a `token` component references, so an index would resolve to nothing for the most common case. The reference has to stand alone. Oracle-level, taken from the group's first row like face text, because a card's relations do not vary by printing. `component` is interned rather than an enum: Scryfall has added components before, and an unknown one should pass through rather than fail an import. Two tests, including that order survives -- it is meaningful for melds (the two parts, then the result). Gates: cargo test -p card_engine 169 passed / 38 ignored, cargo check clean; api + card_engine 2,358 passed, 19 xfailed; ruff clean.
`?fuzzy=` and `/cards/autocomplete` were the two routes with no engine equivalent at all: both leaned on Postgres, fuzzy on pg_trgm's similarity() and its GIN index. trigram_similarity reimplements pg_trgm exactly -- split on non-alphanumerics, pad each word " word ", window 3 bytes, deduplicate, score |intersection| / |union|. Exactness is the point: the SQL path is a fallback, and if the two scored differently the same query would resolve to different cards depending on which one served it. Nothing is stored. The name vocabulary is ~31,700 oracle names, so scoring the whole corpus per request is a few milliseconds, against a trigram index that would cost archive space permanently for one low-traffic route. Same reasoning for autocomplete's prefix scan. fuzzy_name_match keeps jbylund#912's semantics: clear the floor, and lead the next DISTINCT name by the lead margin. Distinctness is load-bearing -- printings_of_one_card_do_not_look_ambiguous pins it, because without the rule several printings of one card tie with themselves and EVERY fuzzy lookup reports ambiguous. Ambiguous stays a distinct outcome rather than collapsing into a miss: Scryfall reports it, and a 404 would tell the client the card does not exist. Five tests, including trigram_similarity_matches_pg_trgm with hand-computed values ("abc" vs "abd" is 1/3) and the punctuation case that proves separators are separators. Gates: cargo test -p card_engine 174 passed / 38 ignored, cargo check clean.
… a Client Changes The engine already answers Scryfall's query syntax, and /search returns it in this project's own response shape -- selected fields, a cards array, limit/offset -- which is what the frontend wants and is untouched here. A Scryfall-shaped client wants something different: Scryfall's response objects and pagination, plus the per-card addressings it reaches for alongside search. api/scryfall_compat/ adds that as a second surface -- every route Scryfall documents under /cards, answering with Scryfall's own response objects, its 175-per-page pagination, and its error bodies: /cards, /cards/search, /cards/named, /cards/autocomplete, /cards/random, POST /cards/collection, /cards/:id, /cards/:code/:number(/:lang), the five external-id namespaces, and /cards/:id/rulings with its four sibling addressings. format=text and format=image on the single-card routes, format=csv on the list routes, pretty= everywhere. The handlers are a mixin on APIResource -- iter_marked_routes scans inherited attributes, so they register like any other route while the compatibility surface stays in files of its own. The router needed no change: it matches a full path before falling back to the first segment, so the five named sub-routes claim their exact paths and the other nine shapes reach one handler positionally. The card object comes straight out of raw_card_blob, which is what Scryfall sent modulo three keys the importer adds and an empty-string flavor_text it normalizes -- both exactly reversible. That holds for a multi-face printing only as of the merged-row change this builds on; a row not yet rewritten by an import still holds a promoted front face, and is served as its card rather than as a card_face, for the one import cycle that lasts. Rulings are new: magic.rulings, loaded from the bulk rulings file the fetcher already knew about, as a whole-table replace in one transaction. Lookups are index-backed on the 97,808-card corpus, verified by EXPLAIN, including three folded-name expression indexes so named?exact= -- which matches either face of a "Front // Back" card -- plans as a BitmapOr rather than a sequential scan. /cards/random is exempted from both cache layers: _run_query keys on SQL text plus parameters, neither of which varies between draws, and CachingMiddleware would have pinned one card per import generation. Divergences, chiefly that the corpus is a filtered subset of Scryfall's, are recorded in docs/issues/local-scryfall-cards-api.md.
Rebased onto jbylund#877 and jbylund#913, and the routes this commit's title describes are now actually engine-first. Dropped, because jbylund#877 and jbylund#913 supply them and stacking on both made them literal duplicates: - the five FIELD_TABLE entries layout/cmc/rarity/color_identity/legalities, which were duplicate keys in the same table. `colors` stays, being new here. - `rarity_int_to_text` -- jbylund#877's match arm supersedes the array-index copy. - `legality_bits_to_pydict`. jbylund#877's is the one to keep, and not only for precedence: the copy here shifted twice -- let code = (bits >> (u64::from(*shift) * 2)) & 0b11; -- when format_shift_or_assign already stores shifts.len() * 2, so every format past the first decoded wrong. The follow-up commit that fixed it is dropped with it, there being nothing left to fix. - the cherry-picked copy of jbylund#913's order vocabulary, which is what jbylund#913 is. `mask_to_color_letters` and jbylund#877's `identity_letters` had identical bodies under two names; kept jbylund#877's and widened its visibility. FIXED, and this is the substantive part -- the title was not true of most of these routes: - `_card_by_external_id` was defined TWICE in the class. Python keeps the last, so the SQL-only one shadowed the engine-first one and every /cards/multiverse|mtgo|arena|tcgplayer|cardmarket request went straight to Postgres, making the whole of "Address a Card by Its External Ids" unreachable from the API. Deleted, and `_EXTERNAL_ID_KEYS` with it: a strict subset of `_EXTERNAL_ID_COLUMNS` that nothing else read. - `_card_by_scryfall_id` and `_card_by_oracle_id` had no callers anywhere. `_resolve_path_card` and `_resolve_identifier` called `_fetch_one_card` directly. Wired up, so /cards/:id and the collection POST ask the engine before Postgres. Of the five engine-first helpers this commit advertises, only `_cards_by_ids` was reached before.
CompatFields sits on EVERY printing, so its size is multiplied by ~98,000. It was 160 bytes, taking APrinting from 176 to 336 -- the residue cost more than the printing it hangs off. Two changes take it to 128, and APrinting to 304: 32 bytes a row, ~3.1MB corpus-wide. Option<u32> -> Option<NonZeroU32> for the eleven ids and extra prices. rkyv niches a NonZero, so the None lives in the value instead of a separate tag word: 8 bytes becomes 4, eleven times over. None of these can legitimately be 0 -- not a marketplace id, not a penny rank, not a price in cents -- so the niche costs nothing in expressiveness, and `0 reads as absent` is the same rule the producer already applied. set_id: u128 -> an interned u16. There are ~1,000 sets against ~98,000 printings, so storing each printing's set UUID inline was paying 16 bytes to repeat one of a thousand strings. Interning it costs 2. It also removes the only 16-byte-aligned member of the struct, which is where the rest of the saving comes from: that alignment was padding every printing. This matters most where the archive is not free. On Cloudflare Workers the store is served from KV in 26MB chunks and every chunk past the third is an extra serialized round trip on cold load, against a budget with 3.18MB of headroom -- a 15.6MB residue is the difference between fitting and not. It is not only a Workers concern: the same bytes are read into memory everywhere the engine runs, and a smaller archive loads faster on every host. Deliberately NOT done: moving the sparse ids into a side table keyed by printing. It would save another ~4MB, since most printings carry two to four of the six, but it puts a lookup between the field table and its value and cannot be justified until the remaining size is actually the constraint. 174 tests pass. Sizes measured through size_of::<ArchivedCompatFields>() rather than estimated.
`order=usd|eur|tix` under `unique=card` ranked each card by whatever its
`prefer_score`-preferred printing happened to cost, which is not what the
ordering means and not what Scryfall returns.
Measured against api.scryfall.com on 2026-08-11, `unique=cards`:
Birds of Paradise order=usd -> msc jbylund#170 $9.60 (min; the dearest is leb jbylund#187 $1000)
Counterspell order=usd -> brb jbylund#15 $2.30 (min; the dearest is leb jbylund#55 $919.92)
Gandalf the White order=usd -> ltr jbylund#19 $10.22 (min; the dearest is ltr jbylund#299 $2999.99)
Gandalf the White order=tix -> ltr jbylund#470 0.02 (min; ltr jbylund#19 at 0.03 is NOT chosen)
Juzám Djinn order=usd -> arn jbylund#29 $1827 (its only priced printing)
`dir=asc` and `dir=desc` return the same printing and only reverse the list, so
this is a property of the ordering rather than of the direction.
Two failure shapes fall out of the old behaviour, both reproducible:
- Juzám Djinn's preferred printing is its oversized 90s promo (o90p), which
has no USD price at all, so the card carried no price into the sort and
vanished from `order=usd` entirely. Scryfall ranks it first at $1827.
- Gandalf the White's preferred printing is the serialized ltr jbylund#299 at
$2999.99, which floated it to rank 1 of `t:creature order=usd dir=desc`.
Scryfall ranks it by its $10.22 printing, nowhere near the top.
Neither is a `prefer_score` tuning problem: the cheapest printing is
systematically the *least* canonical one (world-championship decks, bulk
reprints), so no weighting reaches this answer. Juzám Djinn's promo outscores
its Arabian Nights printing by exactly the +15 of the `frame` component
(1997 vs 1993) and is otherwise identical, and correcting that would still not
select the cheap printing a price ordering has to return.
`prefer_for_sort` therefore fills in the representative from the ordered column
when the caller named no `prefer` of their own. `Prefer::UsdLow` already spelled
this rule out by hand, so `usd` reuses it and `EurLow`/`TixLow` join it for the
two columns that had no equivalent; they are internal-only and
`prefer_from_str` cannot produce them. An explicit `prefer=` is the caller
naming the printing they want and still wins.
Missing prices lose to real ones (the `*Low` arms negate, so an absent price
scores -inf), and a card with nothing priced keeps its ordinary representative
rather than silently changing which printing it shows.
Confined to selection: no stored value, archive layout or sort key changes, and
the three price columns already have no sort permutation — their keys depend on
the prefer-chosen printing — so they were already on the gathered/walk paths
this affects.
This branch adds `order=eur` and `order=tix` but not the values behind them, so a caller can rank a page by either currency and then have no way to read the number it was ranked on. `price_usd` has been a result field all along; these are its two missing twins. Both are real `magic.cards` columns (api/db/2025-09-29-great-reset.sql:190-191, each with its own partial index) and both are already in ENGINE_COLUMNS, so this is a mapping entry on the SQL side and a FIELD_TABLE arm on the engine side -- no schema change, no reimport, no archive change. The engine arm is a copy of `price_usd`'s, including its exact-f64-from-stored-cents conversion, so the three currencies round-trip identically rather than one of them being lossy. Deliberately NOT added to DEFAULT_RESULT_FIELDS: `fields=None` resolves against DEFAULT_FIELDS in card_engine, and the default response shape is not the place to start returning three prices nobody asked for. The ordering-coverage test is written over CardOrdering rather than over a literal list, so a currency added to the ordering vocabulary without a matching result field fails there rather than shipping a page nobody can interpret. Both new tests fail without the api_resource change.
`assign_artist_ranks` parks the artistless printings in a trailing rank block, keyed on `(name.is_none(), name)` — its doc says that is "matching how the absent side of every other order sorts". Reporting that block to `sort_key_bits` as a VALUE rather than as absent broke exactly that: a real rank REFLECTS under `desc` and an absent sentinel does not, so artistless sorted last ascending (right) and FIRST descending (wrong). It was the one ordering whose absent side moved with the direction. `SortCol::Artist` is now nullable, like every other nullable column. No change to `assign_artist_ranks`: its trailing block is disjoint from every named rank, so it simply stops being read — no stored bytes move and no rebuild is implied. `SortCol::Set` beside it is deliberately left alone: `card_set_code` is non-null, so its dense rank is always a real value. 7 printings on the production corpus, so this is a correctness tidy rather than a visible reshuffle. The test asserts artist's absent side RELATIVE to an already-nullable column rather than against a fixed direction. That pins the property that matters — artist behaves like everybody else — without pinning which side that is, so it holds both before and after jbylund#919's nulls-sort-lowest change and fails if only one of the two is ever flipped. cargo test: 162 passed.
…o Card-Object Fields Had No Accessor
Found by porting this branch to the Cloudflare Workers deployment, which has no
SQL to fall back to -- so every path that quietly went to Postgres here had to
be built on the engine there, and building it is what surfaced these.
FIXED, and this is the same defect the duplicate `_card_by_external_id` had:
- `_fuzzy_similarity_candidate` and `scryfall_cards_autocomplete` never called
the engine. Both ran SQL, so the whole of "Fuzzy Name Match and Autocomplete,
Computed Not Stored" -- trigram_similarity, fuzzy_name_match,
autocomplete_names -- was unreachable from the API, exactly as
card_by_external_id was before the previous commit. Both now ask the engine
first and fall back to SQL, like every other lookup on this surface.
The fuzzy path keeps the SQL route when a `set` filter is present: the
engine matches on names alone, and answering the unrestricted match would be
a different card.
- `autocomplete_names` returned `card_name_lower`. A catalog entry is
something a client hands straight back to `/cards/named?exact=`, and
"lightning bolt" is not the name Scryfall prints. Invisible while the route
went to SQL (which selects `card_name`); wiring the route up is what would
have put it on the wire.
- `to_scryfall_card` reads `border_color` and `frame`, and neither was in
CARD_OBJECT_FIELDS or FIELD_TABLE. On the ENGINE path every card object
therefore carried `border_color: null` and no `frame` at all, where Scryfall
always sends both. Both values were already stored; only the accessors were
missing. `frame` is recovered from card_frame_data by vocabulary rather than
by position, since the import folds `frame` and `frame_effects` into one
collection and a card with an effect but no frame would otherwise report the
effect as its frame.
- `named?exact=` ranked a face match level with a whole-name match. It matches
either half of a "Front // Back" name, which is right -- Scryfall resolves
`exact=Delver of Secrets` -- but ordering on prefer_score alone means a
two-faced card outranks the card actually named that whenever its score is
higher. On the current corpus `exact=Lightning Bolt` answers
"Emeritus of Conflict // Lightning Bolt". A whole-name match now sorts first.
MEASURED, correcting two numbers this branch asserts:
- The niche in "Halve What the Compat Residue Costs a Printing" was not
happening. rkyv 0.8 does NOT niche an `Option<NonZeroU32>` without
`#[rkyv(with = NicheInto<Zero>)]` -- measured at 8 bytes per field, so
CompatFields was 128 rather than 84 and the 44 bytes a row that commit
describes were never saved. The 160 -> 128 it reports came from set_id's
u128 alignment going away. The attribute is added here and the size is
pinned by a test; 84 bytes over ~98,000 printings is 4.2 MB.
- `build_external_id_index`'s "~350 KB at corpus scale" is 8.34 MB. 347,625
entries, and rkyv archives a `(u8, u64, u32)` as 24 bytes because the u64
forces 8-byte alignment on a 13-byte payload. Documented rather than
changed: `(u32, u32)` plus a namespace offset table would take it to 2.78 MB,
which matters for an archive served from a size-capped store and not for
Postgres.
Gates: cargo test -p card_engine 175 passed / 38 ignored; the card-object unit
tests 32 passed; ruff clean.
Every route on this surface except `random` answered without a Cache-Control header, so nothing downstream cached any of them. `CachingMiddleware` is an internal response cache and says nothing to a CDN or a browser, which means a client and any proxy in front of this service re-fetched every card object, every search page and every autocomplete on every request. The values are api.scryfall.com's own, measured against it rather than chosen: /cards/search, /cards/named, /cards/autocomplete, /cards, /cards/:id, /cards/:code/:number, /cards/:namespace/:id public, max-age=57600 POST /cards/collection max-age=0, private, must-revalidate Set BEFORE each handler runs, so the tier rides on the errors raised inside it, which is also what Scryfall does -- an empty-query 400 comes back with the route's own max-age rather than uncached, and so does a `named` miss. `/cards/random` keeps its `no-store` rather than taking Scryfall's `no-cache`: the draw must not be replayed by either layer, and no-store is the one that also defeats the internal cache. The comment there already explains why. Scryfall sends 172800 on `named` specifically; 57600 is used for it here because a card object embeds `prices`, and 48 hours lets a client hold prices from two import cycles ago with nothing in the response to say so. `_set_cards_cache` is local rather than `api_resource.set_cache_header`, which is the same line of code: these routes are a mixin ON `APIResource`, so `api_resource` imports this module and importing back is a circular import that fails at startup. Verified by importing it. Gates: ruff clean; api/tests/test_scryfall_compat_objects.py 32 passed; `import api.scryfall_compat.routes` clean.
`rust-test` runs `cargo clippy -- -D warnings`, and this branch has been failing
it since before the compat-residue work -- 23 errors in lib.rs and 4 more in the
test target. Every one is a lint, not a behaviour change:
16 needless_borrow `&face` / `&part` where the callee takes a reference
10 useless_conversion `u8::from(x)` where x is already u8, because an
archived u8 IS a u8
1 collapsible_if
Applied with `cargo clippy --fix`, so the edits are the ones the lint itself
proposes. 175 tests pass unchanged.
Worth saying why it went unnoticed: the gate this work has been running is
`cargo test` plus `cargo check`, and neither runs clippy -- so every local run
was green while CI was red, and the PR's own checks were never looked at. A
gate that does not include what CI includes is not the gate.
`rust-test` runs `cargo clippy -- -D warnings`, and the three price-ordering tests fail two lints on their fixture literals: zero_prefixed_literal `0_50` inconsistent_digit_grouping `1827_00`, `1775_40` Allowed rather than rewritten. Prices are integer CENTS and these are written `dollars_cents`, so `1827_00` reads as $1827.00 at a glance; clippy's suggestions -- `182_700` and `50` -- are exactly what the column holds and say nothing about what the money is. The grouping is the documentation, so the allow is scoped to the three tests that use it and carries the reason. 162 tests pass unchanged; clippy is clean. Same note as the sibling fix on jbylund#912: the gate being run locally is `cargo test` plus `cargo check`, neither of which runs clippy, so these were green here and red in CI the whole time.
`useless_vec` on `build_external_id_index(&vec![a, b])`. The previous commit
missed it because local clippy is 0.1.90 and this workflow pins 1.97.1, and the
lint does not exist in 0.1.90 -- so the fix ran clean locally and CI stayed red.
Which is the drift the workflow's own comment warns about:
This exact drift bit PR jbylund#760 (local 1.96 was clean, CI's 1.97 flagged
byte_char_slices).
Re-verified by installing 1.97.1 and running the workflow's exact command
rather than an approximation of it:
cargo clippy --manifest-path card_engine/Cargo.toml --all-targets -- -D warnings
0 errors, 175 tests pass.
Second trap worth recording: `cargo clippy` re-run after `cargo clippy --fix`
reports nothing from cache, which reads identically to a clean pass. Checking a
lint gate needs a forced re-lint, not a repeat invocation.
# Conflicts: # README.md # api/api_resource.py # api/card_processing.py # card_engine/src/legality.rs # card_engine/src/lib.rs # card_engine/src/tests.rs
… Not Six `image_uris` carries eleven keys. Scryfall added `thumb`, `grid`, `display`, `art` and `crop` — webp renditions of the same five pictures the jpeg sizes already point at — and `_image_uris` emitted none of them, so every card object this module built differed from Scryfall's, on the top-level dict and on every face. The five are unconditional. Across all 540,484 printings in the 2026-08-16 all_cards bulk, `image_uris` is either wholly absent — 8,444 cards and 7,641 faces, the layouts whose picture lives on the other level — or exactly these eleven keys in exactly this order. No card, face, layout or `image_status` carries a partial set, so there is no absence to round-trip. They are derived rather than stored, so this costs no storage: the same scan confirms all eleven URLs are the one pure function of the id and the face on every one of the 548,604 objects that has them. `art` is a different size of `art_crop`'s path, not a stored companion to it. IMAGE_VERSIONS stays SIX, which is the trap in the change. `version=` did not grow with `image_uris`: `?format=image&version=thumb` redirects to the LARGE jpg, byte for byte the fallback `version=bogus` gets, and the same for the other four. The two constants were the same list and are not any more, so each now says in its own comment why it is not the other. Widening IMAGE_VERSIONS to match would have changed five 302 targets and added five columns to the CSV export, which Scryfall's own export does not have — it serves a single `image_uri` column. Verified byte-exact against api.scryfall.com over 39 `image_uris` objects spanning all 23 layouts, cache-busters included, and the new test pins Scryfall's key set and order rather than this module's, on both placements. card_engine's `write_image_uris` carries the identical table, so the Rust and Python builders still agree key for key.
…Resource, app_context Accessors, a Closed Test Brace Three adjustments the merge of main (via jbylund#913) needed beyond the conflict hunks themselves: - The rulings import rode the import path, which jbylund#963 moved to AdminResource: import_rulings and _import_rulings_quietly now live there, the quiet refresh still sits between the oracle-tag import and the engine reload, and the test patches api.admin_resource._import_rulings and reaches the helper through stub_api_resource.admin. - ScryfallCardsRoutes read self._conn_pool / self._engine off the class it is mixed into; those are self.app_context.reader_pool / .engine now, and the compat tests use the same accessors upstream's own tests do. - The tests.rs merge seam landed jbylund#913's price-ordering tests inside the last face test's body; the brace is closed again.
…ration, set_statement_timeout, reload_engine, _clear_caches The last of jbylund#963's moved names that ScryfallCardsRoutes and the compat test fixture still reached through APIResource: the statement-timeout helper is db_utils.set_statement_timeout, the cache generation and engine reload live on app_context, and _clear_caches moved to AdminResource with the import flow.
…bology-upd # Conflicts: # README.md # api/api_resource.py
…und#963 Put the Import Flow import_sets / import_catalogs / import_symbology and _import_reference_quietly move with the rest of the import sequence onto AdminResource; the quiet refresh still runs beside the rulings one, and the tests patch the admin module. The route-table test names the four public reference routes and the three admin imports.
…und#966's Full Listing for an Admin jbylund#966 made the 404 listing depend on the caller: admin routes hidden from an anonymous caller, the full listing for one carrying the shared secret. This branch had already stopped sending the listing to an unresolved path at all (a client pointed here instead of api.scryfall.com parses code and details, and a listing carries neither). The two compose: an unauthenticated unknown path answers Scryfall's not_found object, an authenticated one falls through to upstream's _raise_not_found and gets the full listing. The jbylund#966 tests for the anonymous cases now assert that object — trivially, no admin route leaks through a body that names no routes.
# Conflicts: # card_engine/src/lib.rs
# Conflicts: # card_engine/src/lib.rs
…he Bulk Corpus
`card_object.rs` was written on this branch and vendored into the Cloudflare port, which then kept
fixing it against the live API while this PR waited. Those fixes never came back, so the two copies
have drifted by about six hundred lines. This carries back everything that does not depend on
archive fields this branch has not got, and keeps `api/scryfall_compat/objects.py` in lockstep so
the SQL path and the engine path cannot answer differently.
Every item was measured against the 2026-08-16 all_cards bulk (540,484 printings) or fetched live;
the reasons are in the comments rather than here.
- **`slug()` was folklore.** "Non-alphanumerics collapse to hyphens" gives `erayo-s-essence` and
raw UTF-8 `jötun-grunt`; Scryfall DELETES `' " , . /` and the curly quotes, hyphenates only runs
of spaces, trims nothing, and percent-encodes. Rewritten to the real rule and verified against
the `scryfall_uri` of all 540,484 printings, zero mismatches. The Rust and Python implementations
are cross-checked to produce identical output on the awkward names.
- **`scryfall_uri` gains the language segment** for the ten print localizations; the glyph
languages (ph, qya) correctly get none.
- **Per-face `image_uris` is gated on the LAYOUT, not the face count.** Only the five two-image
layouts have a second picture; a split, flip, adventure or prepare printing is one piece of
cardboard, and giving its faces their own URLs invented a `.../back/...` link with no image
behind it on every one of them. Zero exceptions in either direction across the bulk. The same
gate moves `colors`, `card_back_id` and `illustration_id` onto the faces where they belong.
- **A one-image multi-face card keeps its cost at top level**, joined `" // "` between the faces
that HAVE one — flipped Erayo is `{1}{U}`, not `{1}{U} // `. Checked against all 3,654.
- **`reversible_card` omits `oracle_id`, `cmc` and `type_line`** at top level and carries the
card's `oracle_id`/`cmc` on both faces. All 81, 0 disagreeing.
- **`related_uris` leads with `gatherer`**, built from the first multiverse id, `printed=true` for
non-English. 540,430 of 540,484; the 54 exceptions are foreign-only promos whose Gatherer entries
carry no translation, which is not derivable from the row.
- **`edhrec` links the FRONT face** except on the split-likes — eight cases verified live. The
`tcgplayer_infinite_*` links keep the joined name, so the two are deliberately different strings.
- **`purchase_uris` always emits all three keys**, falling back per KEY to a name search where the
printing has no id for that marketplace. Emitting nothing left the object EMPTY on 426,416
printings. The search text is the front-face name, because the joined string matches no product.
- **...and is omitted entirely for a printing no marketplace sells** — the rule is `paper or mtgo`,
not `digital`: prm/80925 is digital and HAS the key, ymid/59 and khm/A-198 are Arena-only and do
not.
- **An empty string is a VALUE where Scryfall sends one** (Rust only — `dict.get` never had this
bug). A basic land's `mana_cost` is `""` on 61,908 printings, `oracle_text` on 7,266 and `artist`
on 965, and all three were coming out as `null`. Safe because the keys are always present where
emitted at all.
- A misplaced `prices_are_two_decimals_or_null` docstring moves to the test it documents.
DELIBERATELY LEFT OUT, because they need archive fields this branch does not store: the printed
name/type/text triple and `flavor_name`, the vanguard modifiers, `produced_mana` and a card-level
`color_indicator`, the printed-slug half of the foreign `scryfall_uri`, and omitting the top-level
`watermark` on faced cards — that last one would make the watermark vanish, since `PrintingFace`
has no watermark field here. They stay on the port until the schema catches up.
`cargo clippy` at the pinned 1.97.1 and `cargo test` (199) are clean; `make test-unit` is 3387
passed with one pre-existing failure unrelated to this change
(`test_a_worker_that_only_mmapped_the_archive_still_decodes_legalities`, red on the branch before
this commit).
…ixture The Scryfall card object is built twice — `card_engine/src/card_object.rs` on the engine path and `api/scryfall_compat/objects.py` on the SQL path — and both answer `/cards/*`, so a difference between them is a difference a client can see. Nothing compared them. The Rust suite and the Python suite are separate CI jobs that never meet, and `objects.py`'s own docstring already names the hazard: "Two builders would be two chances to disagree with Scryfall." That is not hypothetical here. The commit before this one carried eleven fixes across from the Cloudflare port, and each had to be written twice by hand; the slug rule alone was only proved identical by dumping both implementations over a name list and diffing them. `api/scryfall_compat/fixtures/card_object_parity.json` holds ten engine rows and the object BOTH implementations produce from each. Rust asserts it in `card_object.rs`, Python asserts it in `test_scryfall_compat_objects.py`, so whichever side drifts turns its OWN job red, with no cross-language invocation and nothing new in CI. The cases are chosen where the two have most room to disagree: a plain card, a slug that deletes rather than hyphenates, a foreign printing's language segment, a two-image layout (faces own the pictures and `colors`), a one-image multi-face card (joined top-level cost), a flip whose back face has no cost, a reversible printing's three dropped keys, the per-key `purchase_uris` name-search fallback, an Arena-only printing that omits `purchase_uris`, and a basic land whose empty-string `mana_cost`/`oracle_text`/`artist` are values rather than absences. Values and key PRESENCE are what this pins; key ORDER is not, since both sides compare parsed objects. The wire order stays pinned by the position assertions already in `card_object.rs`, which is the only place it is observable. Both halves were checked against a deliberately injected regression rather than assumed to work: re-gating per-face `image_uris` on the face count fails the Python half on exactly the two one-image multi-face cases, and dropping `gatherer` from `related_uris` fails the Rust half. clippy 1.97.1 clean, `cargo test` 200 passed, `make test-unit` 3398 passed with the one pre-existing unrelated failure.
…ets-catalogs-symbology
Both sides had spent 2026082301/2026082302 on different archive layouts, so the merged layout takes ARCHIVE_FORMAT_VERSION 2026082701. Upstream's jbylund#1056 legality sort cache supersedes this branch's equivalent (format_order -> format_shifts_sorted). The jbylund#1041 query budget reaches /cards/search through _search unchanged; /cards/random parses q itself, so it now answers QueryBudgetExceeded and InvalidRegexPatternError with the same messages _search sends, inside the Scryfall error object.
…nt 01 and 02 on Other Layouts
…cryfall-sets-catalogs-symbology
… — Remove All 1,355 Artifact Files
Three conflicts, all in the engine, plus one silent gap the textual merge could not see. card_engine/src/lib.rs, ARCHIVE_FORMAT_VERSION: main spent 2026082501 on `SortPermutations`' printing-span prefix sums while this branch holds 2026082703. Both comment histories are kept in date order and the constant stays 2026082703 -- the check is equality with a never-reuse invariant, so the merged layout (which carries BOTH main's prefix sums and this branch's `set_rank`/`artist_rank`/`card_name_folded_id`) needs a value newer than either parent's, and renumbering would hand a live meaning to a number a sibling branch has already staked. card_engine/src/lib.rs, build_sort_permutations: adjacency only. This branch inserts the external-id index and the by-id finders immediately above it; main adds `build_perm_printings_prefix` there and puts `offsets` back on the signature. Both kept, main's signature and its dropped "purely card-space now" comment taken as-is -- this branch never touched that function, so main's side is the whole of the change. card_engine/src/tests.rs: both sides append at the end of the file. Kept in order, face-storage and artist-ordering tests first, then the sigma_bound port's fixture tests and the three-phase divert tests. The gap: `ArchivedSortPermutations::get_printings_prefix` is new on main and its non-streamable arm enumerates `Rarity | PriceUsd`, which is the whole of main's non-streamable set. This branch adds `PriceEur`, `PriceTix`, `Released`, `Color`, `Set` and `Artist` to `SortCol`, none of which have a card-space permutation either, so the merged match was non-exhaustive. Extended to the same list `get`/`get_inv` already carry, returning None -- the printing-keyed columns have no prefix because they have no permutation to take a prefix of. cargo test 208 passed; clippy clean on the pinned 1.97.1; pytest 3719 passed; ruff clean.
…cryfall-sets-catalogs-symbology Takes the re-pinned base, which carries upstream/main's jbylund#1011 and the sigma-decision-rule series along with it. No conflicts on this side: the base's three engine resolutions (ARCHIVE_FORMAT_VERSION kept at 2026082703 with both comment histories, build_sort_permutations' offsets argument restored, the two test blocks kept in order) arrive already settled, and this branch touches none of that. api/admin_resource.py auto-merged: jbylund#1011 rewrites the `hybrid` and `phyrexian` entries of BOOLEAN_IS_TAGS, this branch appends the reference surface's import wiring elsewhere in the file, and the two hunks do not overlap. Both are present. pytest 3897 passed; cargo test 208 passed; clippy clean on the pinned 1.97.1; ruff clean.
…ut Halves
`/symbology/parse-mana` answered 422 on a real mana cost. The 2026-08-16 follow-up on this branch
established that `{W/U/B}` is not a Magic symbol and encoded that as `_HYBRID_HALVES = 2`, which is
right for `{W/U/B}` and wrong for the ten Phyrexian hybrids, which have three halves. Four live
cards put one in their mana cost -- Ajani, Sleeper Agent `{1}{G}{G/W/P}{W}`; Tamiyo, Compleated Sage
`{2}{G}{G/U/P}{U}`; Nahiri, the Unforgiving `{1}{R}{R/W/P}{W}`; Lukka, Bound to Ruin
`{2}{R}{R/G/P}{G}` (`is:phyrexian is:hybrid`, 4 results, 2026-08-28) -- and every one of the four
was a 422 here.
Loosening the count to "two or three" would be wrong in the other direction, so the rule is now the
INVENTORY, which this branch already serves from `/symbology` and had not thought to read: fetched
whole on 2026-08-28 it lists 84 symbols, 36 of them containing a slash, and a hybrid parses if and
only if it is one of those 36. That also picks up the six colorless hybrids `{C/W}`..`{C/P}`, which
the halves rule rejected for a different reason -- it priced colours, digits and `P`, and `C` is
none of the three -- and it still rejects `{W/U/B}`, `{3/W}` and `{C/W/P}`, none of which Scryfall
lists.
Spelling is measured, not assumed. A two-part hybrid may be written either way round and comes back
canonical (`{U/W}`->`{W/U}`, `{W/2}`->`{2/W}`, `{P/W}`->`{W/P}`, `{W/C}`->`{C/W}`); a three-part one
may not -- `{U/W/P}` and `{P/W/U}` are both 422s where `{W/U/P}` parses.
Accepting the symbols exposed the emission order, which was a colour-rank sort and is really the
catalog order: `{G}{G/W}{W}` answers `{G/W}{G}{W}`, so a hybrid comes out ahead of a plain pip of
its colour, and `{G/W}{W/U}` answers `{W/U}{G/W}` -- the later colour first, which no colour-rank
sort can reach. The five plain colour pips are the one exception and keep the canonical colour order
`RUW`->`{U}{R}{W}` pins. Ranking them inside their own block of the catalog says both at once, which
also collapses the colored/colorless buckets into one list: `{W}{C/P}` answers `{C/P}{W}`.
Two smaller things the same fetch settled. `H` is not a prefix over any colour -- the catalog lists
`{HW}` and `{HR}` and no others, and `?cost={HB}` is a 422. And the fragment a rejection names
strikes out exactly the ten one-character symbols: `P`, `H` and digits SURVIVE, so `{U/W/P}` reports
`“{//P}”` and `{3/W}` reports `“{3/}”`, where this struck them because the parser priced them. The
2026-08-16 rule ("the input with everything Scryfall could read struck out") is unchanged; what it
strikes was inferred and is now measured.
61 parse-mana requests, one per row, all of them now goldens; the parser reproduces all 61 byte for
byte, 422 messages included. Full non-container suite 3,956 passed; ruff check and format clean.
…Back Face, and Both Name Routes Compared Folded Where Scryfall Compares COLLATED
`POST /cards/collection` resolved a `{"name"}` identifier with
lower(card_name) = lower(%(name)s)
OR lower(split_part(card_name, ' // ', 1)) = lower(%(name)s)
which is wrong against api.scryfall.com in four separate ways, and `named?exact=`
carried two of the same mistakes in its own predicate and in `folded_name_matches`.
MEASURED against api.scryfall.com on 2026-08-31, ONE IDENTIFIER PER REQUEST — a
collection response's `data` is not in identifier order, so a batched probe
attributes its answers to the wrong needles, which is how the first reading of
this rule came out saying flavor names resolve:
{"name":"Delver of Secrets"} Delver of Secrets // Insectile Aberration
{"name":"Insectile Aberration"} the same card — a BACK face names it
{"name":"Delver of Secrets // Insectile ..."} not_found <- and `exact=` answers it
{"name":"Fire // Ice"} / {"Wear // Tear"} /
{"Bonecrusher Giant // Stomp"} not_found, all three
{"name":"Who // What // When // Where // Why"} und/75 — a FIVE-part name IS a key
{"name":"Who"} not_found — and so are `exact=Who`,
`exact=What`, `exact=Why`
{"name":"Godzilla, King of the Monsters"} not_found <- `exact=` answers Zilortha
{"name":"limduls vault"} Lim-Dûl's Vault (c13/197)
{"name":" Lightning Bolt "} Lightning Bolt (msc/806)
{"name":"Delver of Secrets","set":"mid"} mid/47 — set FILTERS the lookup
{"name":"Elves"} Elves (ffdn/9) — the card NAMED that,
not one of the hundreds containing it
So a card answers to its two FACE names when its name splits in EXACTLY two, and
to its whole name otherwise — never both. `exact=` adds the joined name of a
two-faced card and the flavor names; that is the whole of the difference. One scan
with a scope, `name_key_tier` deciding which keys the caller may have.
WHAT THE COLLECTION PREDICATE GOT WRONG, each verified by running the branch's own
code before the change:
- It never looked at the BACK face. `{"name":"Compat Aberration"}` missed a card
Scryfall resolves; `split_part(..., 2)` was simply absent.
- It accepted the JOINED name, which is not a key there.
- It read a five-part name as having a front face, so `{"name":"Who"}` answered
und/75 where Scryfall reports not_found.
- It compared `card_name` AS POSTED — not folded, not collated, not trimmed — so
no accent, punctuation or spacing difference resolved, and
`{"name":" Lightning Bolt "}` missed on its own whitespace.
- And it had no ranking at all: `_fetch_one_card` with no `rank_first` orders on
prefer_score alone, so a needle that is one card's whole name and another's face
answered whichever scored higher.
TWO THINGS THE ENGINE SCAN GOT WRONG FOR `exact=` TOO, fixed here because they are
one predicate.
COLLATED, not folded. `exact=delverofsecrets`, `exact=limduls vault`,
`exact=Lightning-Bolt`, `exact=Kongming Sleeping Dragon` and
`exact=whowhatwhenwherewhy` all resolve on api.scryfall.com and all answered 404
here, on both surfaces.
A FACE KEY EXISTS ONLY WHEN THE NAME SPLITS IN EXACTLY TWO. `split_once(" // ")`
read *Who // What // When // Where // Why* as "Who" plus the rest, so `exact=Who`
answered und/75 against Scryfall's 404 — while the five-part name itself is a key
on both surfaces, and `exact=Why` (the LAST part, no more a key than the first) was
already 404 either way.
THE NARROWING. `name_trigram` is built over `card_name_folded`, and a collated
needle shares no trigram with the punctuated name it matches — `limduls vault`
against `lim-dul's vault` narrows to nothing. So `name_best` treats the index as a
FAST PATH and re-runs a MISS as a full scan: every needle spelled the way the corpus
stores the name still costs the 4 us the index was added for, and only a miss pays
the 884 us. What that leaves is stated in the code rather than hidden — two distinct
cards whose folded names collate alike but are spelled differently would be ranked
against each other only on the full-scan path. Closing it means rebuilding the index
over a collated name, which is an archive-format change and not this one's.
FLAVOR NAMES are the one part of `exact=`'s rule this cannot express: no column in
this corpus carries a printing's flavor name, so `exact=Godzilla, King of the
Monsters` answers Zilortha there and 404s here. Pre-existing, not opened by the scope
split — and it happens to leave the COLLECTION surface right, since a flavor name is
not a collection identifier's key either.
The SQL fallback says the same thing as the engine, expression for expression
(`_COLLECTION_NAME_MATCH`, `_EXACT_NAME_MATCH`, `_WHOLE_NAME_FIRST`), because a
worker with no store loaded answers from it and a rule expressed in only one of the
two is a rule this API applies only sometimes. The one place the two can still drift
is the character class: Rust's `char::is_alphanumeric` against the server's
`[:alnum:]`, which agree over ASCII and so over every folded name this corpus holds.
The route tests run every case through BOTH, via `by_name_paths`.
That fixture immediately earned itself: a five-part fixture name spelled "Compat"
five times (70 bytes) resolved through SQL and NOT through the engine, because
`card_name_folded` is an `InlineStr<61>` and anything longer is silently truncated.
36 names in the real corpus are over that bound. Left alone here — it is an archive
layout question, not a key-rule one — and the fixture name shortened so this pins the
divergence it is about.
cargo test 211 passed; clippy clean on the pinned 1.97.1, both crates; pytest 3776
passed (the 3 test_engine_property errors are pre-existing on the untouched branch —
a `devotion without identity` panic in its random-store fixture); ruff clean.
…es — `Snake // Zombie` Was Searched For as `Snake`
`write_purchase_uris` did `quote_plus(front_face_name(name))`, so all three
marketplace SEARCH fallbacks cut the front face off a joined name on EVERY layout.
Its Python twin `_purchase_uris` did `name.split(" // ", 1)[0]` for the same reason.
Scryfall does not: the search string splits by LAYOUT, and by exactly the split
`related_uris.edhrec` already used.
MEASURED on api.scryfall.com 2026-08-31, over `unique=prints`, on the first printing
of each card whose marketplace ids are MISSING so the SEARCH form is what gets
emitted — a printing WITH the id gets a product link and tells you nothing — and
reading the tcgplayer term out of the `u=` parameter of Scryfall's own partner
redirect rather than off the wrapper:
split Bind // Liberate cmb2/88 cardhoarder `Bind // Liberate`
reversible_card Mechtitan // Mechtitan sld/1969 cardhoarder `Mechtitan // Mechtitan`
double_faced_token Snake // Zombie cc2/9 tcgplayer, cardmarket AND
cardhoarder, all `Snake // Zombie`
split Who // What // When // Where // Why und/75 cardhoarder the whole
five-part name
adventure Champions of Archery // Join the … ph19/4 cardmarket, cardhoarder
`Champions of Archery`
flip Curse of the Fire Penguin // … unh/73 cardhoarder `Curse of the
Fire Penguin`
art_series Aang and Katara // Aang and Katara atle/8 all three `Aang and Katara`
transform Delver of Secrets // Insectile … sld/2367 cardhoarder `Delver of Secrets`
The first three are `EDHREC_JOINED_LAYOUTS` exactly, so the constant is renamed
`JOINED_SEARCH_LAYOUTS` and the string it selects is `search_name` rather than
`edhrec_name`: it was never an EDHREC quirk, it is what a SEARCH LINK spells here.
`front_face_name` is now applied at ONE site — the `search_name` derivation at the
top of `write_scryfall_card` — and applying it a second time inside
`write_purchase_uris` is the whole of the bug.
`related_uris`' two `tcgplayer_infinite_*` links remain the exception and keep the
joined name on every layout, split or not. Verified in the same probe: atle/8 and
cc2/9 spell the joined name there while their marketplace searches follow the layout
rule, and mom/230/es already pinned the front-face half.
BOTH COPIES, because a card object is built twice here: `card_object.rs` serves the
engine path and `objects.py` serves the SQL path, and both answer `/cards/*`. The
shared parity fixture moved on exactly two of its ten cases — `Fire // Ice` (split)
and `Propaganda // Propaganda` (reversible_card) — and on nothing else, which is the
rule stated as a diff. Patched in place rather than regenerated, so the file keeps
its key order and the change is six lines.
The Rust test's fixture carried a joined `name` with NO `card_faces`, which
`search_name` reads as a single-faced card: it would have passed the joined
assertions while testing nothing. Every fixture in it now carries faces and a layout.
FOUND THROUGH und/75, whose collection identifier the commit before this one made
resolvable: its cardhoarder link searched for `Who`.
cargo test 211 passed; clippy clean on the pinned 1.97.1; pytest 3784 passed (the 3
test_engine_property errors are pre-existing on the untouched branch — a `devotion
without identity` panic in its random-store fixture); ruff clean.
…ames) into scryfall-sets-catalogs-symbology This branch was forked from the base at 3a47724 and so still carried the base's OLD code for two rules the base has since fixed. Taking the base again brings both fixes in as the SAME commits they are on scryfall-cards-api, so a later merge of either branch sees one sha per rule rather than two shas touching the same lines. dcc0b6b -- a `{name}` collection identifier has its OWN keys, and both name routes compared FOLDED where Scryfall compares COLLATED. Replaces `folded_name_matches` -- three references here, all of them the old rule -- with `name_key_tier` under a `NameScope`, adds `collection_card_by_name` beside `exact_card_by_name`, and rewrites the `{name}` identifier's SQL (`_COLLECTION_NAME_MATCH`) and `named?exact=`'s (`_EXACT_NAME_MATCH`, `_WHOLE_NAME_FIRST`) to say the same thing the engine says. 36e2d5b -- `write_purchase_uris` and its Python twin `_purchase_uris` cut the front face off a joined name on EVERY layout, where the three marketplace SEARCH fallbacks split by layout exactly the way `related_uris.edhrec` does. `EDHREC_JOINED_LAYOUTS` becomes `JOINED_SEARCH_LAYOUTS` and the string it selects becomes `search_name`, derived once at the top of the card writer. NO CONFLICTS. The only commit this branch has added since 7494275 took the base at 3a47724 is 02d868c, which is confined to api/scryfall_compat/mana.py and api/tests/test_scryfall_mana.py -- neither file is touched by either incoming commit. The eight merged files land at the base's content exactly; the residual diff against scryfall-cards-api is this branch's own surface and nothing else (`catalog_object`'s optional `uri`, the `ScryfallResponder` mixin the `/sets`, `/catalog` and `/symbology` routes share with `/cards/*`, and the dispatch 404 that replaced the raised HTTPMethodNotAllowed on `GET /cards/collection`). routes.py auto-merged across both: dcc0b6b's name-key block and this branch's extraction of `_scryfall_respond` into api/scryfall_compat/responder.py do not overlap. card_object_parity.json arrived as the base's six changed URL lines across its two moved cases -- `Fire // Ice` and `Propaganda // Propaganda` -- with the key order intact; it was merged, not regenerated. ARCHIVE_FORMAT_VERSION stays 2026082703, which is what both sides already held. cargo test 211 passed and clippy clean on the pinned 1.97.1, both crates; pytest 4024 passed plus the 22 testcontainers integration tests, both green; ruff check clean.
Contributor
Author
Conflict notes: land after #912 (the chain is in this PR's description)The full chain is the one this PR's description already states — #894 → #913 → #912 → #893 → #927 → #928 / #929, with #922 any time after #912. What follows is about the CONFLICTS that order produces, not a replacement for it. Full ordering and conflict-resolution rules are in #912 (comment) — the short version for this PR: This branch contains #912's commits (it is the only true ancestor relation among the five: It carries the same generation as #912 (search fallbacks present, |
daveycodez
added a commit
to daveycodez/sylvan_librarian
that referenced
this pull request
Sep 1, 2026
…m, and the Cut Reached Every Name Surface at Once `OracleCard.card_name_lower` is an `InlineStr<61>`, whose `from_str` truncates in SILENCE, and the comment beside it said "61 bytes covers every card name in the Scryfall dataset". It does not: 36 names in `all_cards` are longer — 34 doubled `X // X` art-series and reversible names, "Curse of the Fire Penguin // Curse of the Fire Penguin Creature" (63 bytes), and the 141-character Unhinged elemental. ONE TRUNCATION, EVERY SURFACE. The folded name is DERIVED from the already-cut inline, and both name routes collate off the folded name, so a single silent cut reached the `!` operator, `name:`, `named?exact=`, the collection identifier and the autocomplete catalog together. Measured on a deployment of this engine 2026-08-31, against api.scryfall.com: !"Curse of the Fire Penguin Creature" 0 here 1 there name:"fire penguin creature" 0 here 1 there named?exact= of the same string 404 the card !"Curse of the Fire Penguin Creatu" 1 HERE 0 there The last line is how it was found: the 61-byte cut spelling answered, and the name the card actually prints did not. `CardRow` NOW CARRIES THE FULL `String` for both names. A parse-time row is never archived, so the only thing the inline bought there was the truncation — and the archived `card_name_folded_id` is decided by comparing those two fields, so for an over-long name it was comparing two cuts and answering about a string neither field held. THE OVERFLOW ID IS FREE IN THE ROW HERE, which is the one place this branch differs from the same fix elsewhere. `card_name_lower_id` holds the whole string whenever the inline does not, `NONE_STR` on all but 36 cards — and `card_name_lower` KEEPS its 61 bytes, because `size_of::<Archived<OracleCard>>()` measures 256 both before and after. `oracle_id` gives the row 16-byte alignment and the trailing round-up already held more than four spare bytes; a sweep over the inline width puts the cliff at `InlineStr<65>`, where the row goes to 272 (~500 KB of archive over ~31,700 cards). So nothing is narrowed to pay for the id, and `the_overflow_id_is_free_in_the_row` pins all of it rather than leaving it to be rediscovered. EVERY READER GOES THROUGH `lower_name` NOW — `folded_name` and `folded_name_of` fall back to the whole lower name rather than the inline prefix (which is what carried the cut into `name:`, both name routes, the trigram index and the autocomplete catalog), `TextField::NameLower` and `FilterExpr::ExactName`'s predicate read it directly. The builder writes the pair through `split_lower_name`, one function, so the bound cannot be agreed on in one place and forgotten in another. THE NAME PERMUTATION IS THE DELIBERATE EXCEPTION, and it needed its own two lines. `assign_name_ranks` keys on the INLINE field and `narrow_rec`'s `ExactName` arm binary- searches that same permutation, so re-keying one without the other would search a permutation sorted by something else. The arm is exact again instead: it DECLINES a needle the bound would cut (the unnarrowed walk then verifies through `lower_name` and finds the card, where the search answered the empty set TIGHTLY before), and it drops spilled cards from the block it lands in (a spilled card matched the needle on its cut, so the needle is a strict prefix of what the card is called — that is the `…Creatu` false positive above). `tight` stays honest either way, which the `Not` arm depends on. TESTS. `a_name_past_the_inline_bound_survives_every_name_surface` resolves the whole joined name, both faces, the collated spelling of the back face and a collection identifier, and asserts the 61-byte cut resolves to NOTHING on either surface — that last one is the assertion that fails on the old build by answering the card. `exact_name_narrowing_survives_a_name_past_the_inline_bound` covers both halves of the narrowing plus the text-exact reader, and the autocomplete test grows the real 141-character elemental with a needle at byte 102, which the inline could not reach at all. The fixtures build their stores through `split_lower_name`, the same function the builder uses, so a fixture cannot quietly store the cut and test the bug. ARCHIVE_FORMAT_VERSION 2026082703 -> 2026083101. The row does not grow, but every field after `card_name_lower` moves, so this is a real layout change and the header's size check would not catch it on its own. The value is dated TODAY rather than continuing the 2026-08-27 sequence on purpose: the five stacked branches hold …708 (jbylund#929), …707 (jbylund#927/jbylund#928) and …703 (jbylund#912/jbylund#922), the check is equality with a never-reuse invariant, and the highest value has to win at merge — a value above all four is above them whichever order they land in. Verified under the pinned 1.97.1, not the local default: `cargo test` 214 passed / 0 failed on card_engine and clean on shared_cache, `cargo clippy --all-targets -- -D warnings` clean on both. Each of the four fix sites was reverted in turn to confirm the new assertions actually fail without it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Stacked on #912. Merge #912 first. GitHub will not let this PR use
scryfall-cards-apias itsbase — that branch lives in the fork and a base branch must exist in this repository — so the diff
below shows #912's commits until it lands. The commits that belong to this PR are the last three.
Motivation
#912 made every route under
/cardsanswer the way api.scryfall.com does. That left the half of theAPI that describes Magic rather than returning cards: what sets exist, what the game's vocabularies
are, what the mana symbols mean. A client pointed at this host got its cards and then 404d on
/sets. This adds the remaining twenty-six routes, after which the surface is complete.What
GET /setsGET /sets/:code,/sets/:idGET /sets/tcgplayer/:idGET /catalog/:nameGET /symbologyGET /symbology/parse-mana?cost=Three new tables, an importer wired into the existing sequence beside the rulings load, and a second
routes mixin.
Mirrored, not derived — the part worth reviewing
Everything except
parse-manais served from a table mirrored off Scryfall rather than computed frommagic.cards. That is not the obvious choice — this project already derives/cards/autocompletefrom its own corpus — so the reasoning matters:
tcgplayer_id,mtgo_code,arena_code,icon_svg_uri,block,block_code,parent_set_code,printed_size./sets/tcgplayer/:idisnot degraded without the first of them, it is unimplementable from the corpus — a set's
tcgplayer_idis a TCGplayer groupId (24766fortrk) while a card's is a productId(
706132–706191for cards in that same set). Different namespaces, no derivation between them.card_countcounts Scryfall's printings. This corpus is a deliberate subset, so a derivedcount would report a number no other Scryfall client agrees with.
svg_uriorgatherer_alternatesfrom.parse-mana, and two rules nothing documents
Both measured against the live API rather than inferred:
RUWanswers{U}{R}{W}. Every canonical ordering (allied pairs, enemy pairs, shards, wedges, four-colourruns, WUBRG) is a walk around the colour wheel at a constant step: one for anything contiguous,
two for anything not. Step 1 before step 2, starting points in WUBRG order, reproduces Scryfall
for all 31 colour combinations.
{C}regardless of input order, generic summed —2XWUanswers{X}{2}{W}{U},1{1}answers{2}.Also: an empty cost is
nullbutcost=0is"{0}", and an unparseable fragment is a 422.These are exactly what a hand-written test would re-assert from the implementation, so
test_scryfall_mana.pypins 79 goldens captured from the live API instead, covering all 31colour subsets written forwards and backwards.
Cache headers are mirrored too
These routes do not carry the card routes'
public, max-age=57600:/sets,/sets/:code,/sets/tcgplayer/:id,/catalog/*,/symbologypublic/symbology/parse-manamax-age=0, private, must-revalidateThe first cut of this branch applied the card tier to all six, which meant telling shared caches to
hold
parse-manafor sixteen hours where Scryfall revalidates. Both upstream values are mildlysurprising and are mirrored rather than endorsed — reasoning in the design doc. An A/B against the
live API returns 4/4 byte-identical.
Validation
Full suite 2,970 passed, 19 xfailed (up 142 from 2,828 on #912's head);
ruff checkclean andruff format --checkclean across every.pyinupstream/main...HEAD.End to end against live Scryfall on 2026-08-11, in a fresh container:
import_setsimport_catalogsimport_symbologyparse-manaWorth flagging: the first cut of the
_import_reference_quietlytest stubbed only the failing step,leaving the other two to perform real Scryfall fetches and real writes into the shared test
database — while passing. The helper now stubs all three and says why.
Known divergences
card_countis Scryfall's number, not this instance's./catalog/card-namesand/catalog/artist-nameslist cards and artists this instance may nothold, for the same corpus reason API: Every Scryfall /cards/* Route, So the Base URL Is the Only Thing a Client Changes #912 already owns.
unknown name is not served as a silently empty catalog.
uri/search_uri/icon_svg_urion a Set object point at Scryfall, matching API: Every Scryfall /cards/* Route, So the Base URL Is the Only Thing a Client Changes #912's rule thatpayload URIs are not rewritten.
Full reasoning and follow-ups in
docs/issues/local-scryfall-sets-catalogs-symbology.md.
Merge safety
The migration is additive — three new tables, no change to any existing one — and sorts after
2026-08-10-01. Nothing outside the new mixin reads the new tables, and the only edits to #912'sfiles are the
ScryfallResponderextraction (two methods moved, no behaviour change), the mixinbeing added to
APIResource's bases, and oneruff formatblank line.Follow-up (2026-08-16): six answers measured against api.scryfall.com and corrected
This surface was written from Scryfall's documentation and from what looked reasonable. Put to
api.scryfall.com one request at a time, six of its answers were not Scryfall's. Every change below
is a measurement, and the commit messages carry the request/response pairs.
/symbology/parse-manawith nocost400 "You must provide a cost parameter to parse."200 {"cost": null, ...}— the same body?cost=gives?cost={W/U/B}200, a three-coloured ManaCost422— a hybrid has exactly two halves?cost=xyzzy{X}{Y}{Z}{Z}{Y}(writing order){X}{Y}{Y}{Z}{Z}— sorted and grouped?cost=!!!“{!}”— the first fragment, re-braced“!!!”— every fragment, as written/catalog/Card-Types200404— catalog names are case-sensitivepublicno-cache(a 404 about the DATA,/sets/zzzz, keepspublic)Plus two that are not per-route:
/sets/:code/extraanswered "No Magic set found for the given codeor ID" about a set that was fine, and every error body is indented on api.scryfall.com — measured
across the whole surface, and it does not negotiate (
Accept: application/json,Accept: text/html,a bare wildcard and an explicit
?pretty=falseall give the same 130-byte indented not-found, whileevery data body is compact). This rendered both compact, so a client comparing bytes saw a different
document for every 4xx it received.
The unparseable-fragment wording turned out to follow one rule rather than five, and the six new
goldens pin it: what comes back is the input with everything Scryfall could read struck out, adjacent
bare characters merged into one run, and a braced token keeping its braces.
{W/U/B}answering“{//}”is what made the rule visible.One measurement is recorded rather than reproduced. A 133-character nonsense cost comes back
naming 51 characters. One data point does not determine a truncation rule, so nothing was guessed.
Two existing tests asserted the behaviour Scryfall does not have (
test_the_name_is_case_insensitive,test_a_missing_cost_is_a_400) and are inverted rather than deleted, since the URLs they cover stillneed covering.
Found by a differential sweep of the peripheral API surface — response formats, the reference
endpoints and HTTP-level behaviour — against api.scryfall.com, run from the Cloudflare port
(daveycodez/sylvan-librarian-cloudflare,
scripts/parity-sweep.ts). That harness now carries 200cases over this surface, so these stay covered.
Follow-up (2026-08-28):
{W/U/P}is a printed symbol, and the two-halves rule rejected itThe correction above ("a hybrid has exactly two halves") is right about
{W/U/B}and wrong about theten Phyrexian hybrids, which have three.
/symbology/parse-manawas answering 422 on a realmana cost:
is:phyrexian is:hybridfinds four cards, and all four were rejected.{1}{G}{G/W/P}{W}{2}{G}{G/U/P}{U}{1}{R}{R/W/P}{W}{2}{R}{R/G/P}{G}"Two or three halves" would be wrong in the other direction, so the rule is now the inventory this
branch already serves:
GET /symbology, fetched whole on 2026-08-28, lists 84 symbols of which 36contain a slash, and a hybrid parses if and only if it is one of those 36. That also fixes the six
colorless hybrids
{C/W}…{C/P}, which the halves rule rejected for a different reason — it pricedcolours, digits and
P, andCis none of the three — and{W/U/B},{3/W}and{C/W/P}arestill 422s, because Scryfall lists none of them.
Spelling is measured, not assumed: a two-part hybrid may be written either way round and comes back
canonical (
{U/W}→{W/U},{W/2}→{2/W},{P/W}→{W/P},{W/C}→{C/W}), while a three-partone may not —
{U/W/P}and{P/W/U}are both 422s where{W/U/P}parses.Emission order (rule 2 above) is superseded. Accepting the symbols exposed it: it is not a
colour-rank sort but
/symbologycatalog order, with the five plain colour pips as the onlyexception — they keep the canonical colour order rule 1 pins.
{G}{G/W}{W}{G/W}{G}{W}{G/W}{W/U}{W/U}{G/W}{G/U}{W/B}{W/B}{G/U}{HR}{HW}{HW}{HR}{W}{C/P}{C/P}{W}{S}{C}{C}{S}Every row was also requested written the other way round and answered the same, which is what makes
it a sort rather than the writing order. Ranking the plain pips inside their own block of the catalog
expresses both rules at once, and collapses the colored/colorless buckets into one list.
Two smaller things the same fetch settled.
His not a prefix over any colour — the catalog lists{HW}and{HR}and no others, and?cost={HB}is a 422. And the fragment a rejection namesstrikes out exactly the ten one-character symbols:
P,Hand digits survive, so{U/W/P}reports
“{//P}”and{3/W}reports“{3/}”. The 2026-08-16 wording rule is unchanged; what itstrikes was inferred and is now measured.
61 parse-mana requests, one per row, all of them now goldens — the parser reproduces all 61 byte for
byte, 422 messages included. Full non-container suite 3,956 passed;
ruff checkandruff format --checkclean. The same fix is in the Cloudflare port(daveycodez/sylvan-librarian-cloudflare,
src/routes/scryfall-compat/mana.ts), which is where thetwo-halves rule was ported to.