Import+Frontend: Merge Card Faces Into One Searchable Row, and Flip to the Back

[daveycodez]

Closes the core of #400 and #873: multi-face cards become one searchable row carrying every face's data, the "single conceptual card" direction from the #873 discussion. Based directly on main; no overlap with the other open PRs (#872/#877/#879/#888/#889/#890/#893).

The bug being fixed

preprocess_card() fans each multi-face card out to one row per face — all sharing one scryfall_id — and the upsert's ON CONFLICT (scryfall_id) keeps whichever face came last: the back. Hence the #400 audit's findings: t:battle matched zero rows corpus-wide, t:sorcery missed every MDFC spell side, front-face oracle text was invisible (#873), and ~1,343 printings carried the wrong types/colors/stats. Split cards were hit too (Fire // Ice was findable by Ice's text but not Fire's).

Why one merged row, not one row per face

Measured against api.scryfall.com (2026-08-08): Scryfall ANDs search predicates at the card level, each satisfiable by any face.

• t:sorcery t:land returns the MDFC lands — no single face is both
• o:"flip a coin" o:"instant and sorcery" matches Ral, Monsoon Mage — the texts live on different faces
• c:b matches Westvale Abbey, whose only black face is the back
• pow=3 matches Brutal Cathar via its back face

One row per printing with any-face unions reproduces those semantics directly. One row per face — the original #400 plan — would instead break every cross-face conjunction (no face-row satisfies both terms) on top of colliding on the primary key. The merged row isn't a compromise; it's the correct shape.

What the merge does (_merge_processed_faces)

Field class	Policy
Identity/display: cmc, mana_cost, illustration_id, image, prices, raw blob	front face — matches Scryfall's own top-level fields (verified on its card objects)
card_types, card_subtypes, card_colors, card_keywords, produced_mana, illustration_ids	union of faces, order-preserving
oracle_text, flavor_text	faces joined with \n//\n — both texts substring-searchable. The newline keeps . from crossing a face, and that turned out to be all it keeps out; see the regex update below
type_line	"Battle — Siege // Creature — Moonfolk Ninja", exactly Scryfall's rendering
P/T + loyalty	copied per group from the first face that has them, so numeric and _text columns never describe different faces

Each face still runs through the full existing pipeline first; the merge is ~40 lines over the processed rows. No schema migration, no engine changes, no SQL-gen changes — the engine inherits everything on reload.

Plus one vocabulary fix this exposed: "Battle" was missing from the parser's CARD_TYPES, so t:battle routed to the subtype arm on both paths — a guaranteed zero-match the moment battles became visible. The engine was innocent (its TYPE_BATTLE bit worked in isolation once the parser sent card_types).

Frontend: back-face flip button

Scryfall-style flip, implemented as progressive enhancement so the no-JS server render (parity fixture) is byte-identical: cards named A // B are probed for a face-2 CDN image, and only cards that have one — transform/MDFC, never split/adventure — grow the button (results grid, modal, and card page). copy_images_to_s3.py now uploads card_faces[1]'s image under the face-2 key (the raw blob keeps the front's image_uris at top level for every existing consumer, with the raw card_faces re-attached), and the upload path stops hardcoding face 1 in the S3 key. Ops note: backs appear on the site after a reimport + image sync run.

Validation (rebuilt dev stack, full reimport, 97,802 cards, 2026-08-08)

Check	Before	After	Scryfall
t:battle	0	36 cards / 53 printings	36
t:sorcery t:land	—	20	20
#873's commander query	21	34	43, of which 9 are never-legal draft_innovation cards the corpus excludes by policy — 34/34 of the achievable set
is:permanent on battles	0	36	✓
Fire // Ice by front-half text	0	1	✓
!"Nissa, Vastwood Seer" t:creature	0	1	✓
!"Westvale Abbey" c:b	0	1	✓
!"Brutal Cathar" t:human	0	1	✓

Known residual (tested + documented): when several faces carry a stat group (Brutal Cathar's 2/2 // 3/3), only the front's values are searchable; Scryfall also matches the back's pow=3. Per-face numerics (and back-face mana:) are a measured follow-up if wanted.

Tests

• test_card_processing.py: rewritten DFC tests + a TestFaceMerging class — battle/MDFC unions, front-priority and back-fallback stat groups, *-power group consistency, text joining, all-faces-filtered, one-row-per-printing (the PK-fight regression).
• test_pyparsing_parser.py: t:battle routes to card_types, t:siege stays a subtype.
• test_copy_images_to_s3.py: face-2 emission with its own PNG URL; single-faced cards stay front-only.
• app.test.js: 6 new flip tests (face URLs, probe gating + caching, flip toggling, single-injection). The no-JS parity suite passes untouched.
• Suites: api + parsing + scripts 2,193 passed; jest 1,747 passed; ruff + format + prettier clean.

---

Update: raw_card_blob is now the card Scryfall sent (956f8a6)

The merged row's blob was the front face's dict with card_faces re-attached, so existing top-level reads keep meaning "the front". Going through those reads, exactly one needs that — image_uris — and the copy_images_to_s3.py change already in this PR gives it a card_faces->0 fallback. Everything else read from the blob (lang, set_type, games, finishes, frame_effects, image_status, reserved, game_changer) is card-level and identical either way; backfill_prefer_scores.sql never touches image_uris.

With that covered, the blob can hold the card verbatim, which buys something the promotion cannot. Every searchable field is merged onto the row's own columns, so the blob has no derivation left to do, and keeping it as Scryfall sent it is what makes the card answerable later:

merged_row["raw_card_blob"] = copy.deepcopy(card) | {"card_faces": card_faces}

Why a reader could not have undone the promotion instead. A card object cannot be rebuilt from a face: card_faces is gone, name and type_line are the front's, and which fields a real card carries at top level varies by layout — a split card has mana_cost and image_uris there, a transform card does not. No rule strips the promoted keys correctly for both.

Measured against the object Scryfall sent, before and after:

layout	before	after
transform	11 extra top-level keys; object, name, type_line all disagree	card_name only
split	same	card_name only

card_name is the combined name the importer lifts before splitting faces — one key, stripped by any reader that wants the original.

Also in this commit: prefer_weights.py's seven image reads go through one front_image() helper that coalesces to card_faces->0, so a transform printing no longer silently drops out of the art_crop IS NOT NULL tuning queries. Both of its SQL templates were run against the 97,808-card dev corpus.

Tests: five new in TestMultiFaceRawBlob, covering both layouts, verified to fail on the previous behavior. Suites: api + parsing + scripts + client 2,514 passed, jest 1,747 passed, ruff clean.

Why now: #912 needs the card object to serve Scryfall-identical /cards/* payloads, and was carrying a cards.scryfall_json column to hold a second copy for multi-face rows. With this change that column is gone — the blob is already the answer. #912 is now stacked on this PR.

---

Update: art tags follow every face the row shows (a9bfc56)

Folded in from #930, which is now closed. I opened it as a separate stacked PR first and that was the wrong call: this repairs damage this PR does, so merging the two separately would mean merging a regression and its fix as two units, neither of which has landed. One PR that was always correct is strictly better.

What the merge broke

card_art_tags is attached by joining on magic.cards.illustration_id:

_sync_card_tags(conn, "illustration_id", "card_art_tags", illustration_id_to_tags)

On main that is right, and it is right because faces are separate rows: a double-faced card's back face is its own row carrying its own illustration_id, so a tag on the back art lands on a row and arttag: finds the card.

This PR gives the merged row the front's illustration_id — correctly, matching Scryfall's own top-level field. But then the back's illustration exists nowhere on the row, and a tag that lives only on the back art becomes unreachable by any query. Measured against api.scryfall.com, 2026-08-16:

query	Scryfall	this PR, before a9bfc56
arttag:snow e:khm	75	73
-art:human e:khm t:creature	135	136

The two missing from the first are Birgi, God of Storytelling // Harnfel, Horn of Bounty and Esika, God of the Tree // The Prismatic Bridge — their snow is on the back face's art. The surplus in the second is Valki, God of Lies // Tibalt, Cosmic Impostor: Tibalt is the human, and Tibalt is the back.

The rule, checked against the corpus rather than assumed

Replaying the 2026-08-16 art-tags dump over the bulk cards, the union predicts Scryfall's arttag:snow e:khm answer set-for-set — 75 names, no surplus and no shortfall — where the front-only reading predicts 73. On the same bulk, 9,368 printings show more than one illustration and 5,491 of them gain at least one tag from a non-front face.

Mechanism

A row that merges its faces has to carry every illustration it shows, so illustration_ids jsonb does — front first, deduped. illustration_id stays the front's for display.

The merge-side change is one tuple entry, because each face row already computes its own single-element list and _merge_processed_faces already unions front-first with dedupe:

_FACE_LIST_UNIONS = ("card_types", "card_subtypes", "illustration_ids")

A split or adventure card — one illustration_id on the card, none on its faces — has both face rows inherit the same id and collapse back to one entry, which is the right answer: there is one piece of art to tag.

jsonb and not uuid[], for a mechanical reason as much as a stylistic one: api/db/bulk_upsert.py resolves a column through _PG_TYPE, where jsonb extracts with obj-> and an ARRAY column falls through to text and fails the insert. Matching card_types/card_subtypes keeps bulk_upsert untouched.

The sync is keyed on scryfall_id, not illustration_id — the part I'd most want a second opinion on. An illustration id no longer determines a row's whole value, and it cannot be batched that way either: _sync_card_tags batches at 5,000, and a batch of illustration ids can split one card's illustrations across two statements where a batch of cards cannot. So _union_art_tags resolves the union in Python and _sync_card_tags is unchanged — still shared byte-for-byte with the oracle path, still diffing, batching and IS DISTINCT FROM-guarded. The costs, stated plainly:

• Memory: flat. A single-illustration card references the dump's existing tag dict rather than copying it, so allocation is bounded by those 9,368 multi-illustration printings, not by the corpus.
• Wire: one record per tagged card instead of per tagged illustration. That is the real price. It buys a scryfall_id join on the unique index instead of idx_cards_illustration_id, and a clear step that is exact per row.

Doing the union in SQL over the array instead needs either a whole-table UPDATE with a per-row aggregate — the shape #910 was about — or a temp table plus an unbatched statement. Neither seemed worth giving up the batching for.

The migration backfills from raw_card_blob->'card_faces', which on an already-imported row is the only record of the back's illustration. Without it the column is empty until the next card import and _sync_card_tags clears every card's art tags in the meantime.

prefer_score reads the union too, deliberately

art_style in backfill_prefer_scores.sql reads card_art_tags, so this moves representative selection. Measured rather than guessed: exactly three printings corpus-wide flip art_style under the union.

• Tribute to Horobi // Echo of Death's Wail neo/356, English and German — back art is anime
• Thaumatic Compass // Spires of Orazca pxtc/249 — back art is line-art

I fed the union to prefer_score rather than keeping a front-only reading for it, and that is a choice, not a consequence. For it: art_style asks what art a printing shows, and a printing shows all of it — a card whose back is anime is partly anime. Against: two readings would protect three rows from a demotion #720's evidence didn't rule on. Three rows is small enough that I'd take either answer; say the word and it's a one-line split.

artwork_printings — still right given the merge, and one thing it quietly fixes

The other card_art_tags-adjacent consumer in that file is artwork_printings, which groups on (illustration_id, card_name) and feeds illustration_count. It is untouched here, and I checked that it is still correct rather than merely unchanged:

• For a transform/MDFC card, each face previously had its own illustration_id and so its own group. Post-merge the group keyed on the front illustration has exactly one member per printing — the same membership the front-face row had before — so no surviving row's illustration_count moves.
• For split, adventure and flip cards it does move, and in the right direction. Both face rows inherited the card's single illustration_id and the same card_name, so they landed in the same group and each printing was counted twice. Post-merge each printing counts once. That is ~836 English printings (Scryfall: 345 split, 449 adventure, 42 flip), and the correction is uniform across every printing of such a card, so it cannot flip a representative — prefer_score ranks printings within a card, and a monotonic shift applied to all of them preserves the order.

So grouping on the front id remains the right key: it measures how many printings show this artwork, and the row shows the front. The residual, unchanged by this PR and a different question, is that two printings sharing a front illustration but differing on the back count as one artwork here while Scryfall's unique=art treats them as two.

Tests

Against real Postgres, not mocks (test_tagging_integration.py, test_upsert_cards.py):

• a back-face tagging lands on the merged card — this is the test that proves the fix pins anything: reverting the keying to illustration_id turns it red, which I ran as a negative control rather than assuming
• the front-face tagging still lands, unchanged
• a dump with no taggings clears the row, so the diff-and-clear path stays exact under the new key
• a merged row stores both faces' illustrations, a single-faced row stores its one
• the migration's backfill lands where preprocess_card would, on rows written by the real import path

And as units: front-first ordering, a split card's faces collapsing to one illustration, a face with no art contributing nothing, an untagged illustration omitted rather than mapped to {}, and the no-copy property of the single-illustration path.

python -m pytest --ignore=api/tests/test_integration_testcontainers.py: 2,551 passed, 19 xfailed. That file errors 22 times for me identically with and without this commit — its class-scoped container binds a fixed host port and collides with the session container locally — so I've left it to CI. ruff check and ruff format --check clean.

---

Update: a joined column is matched per face now — the separator was answering queries (9298416)

Joining the face texts is what makes every face searchable, and it is also the one thing in this PR that creates a query defect that cannot exist on main. _FACE_TEXT_SEPARATOR ("\n//\n") is a string this branch invented. Scryfall stores nothing like it and never joins — it matches each face separately — so every character of that separator is a position in our haystack and in nobody else's, and patterns were answering from it. On main, where each face is its own row, there is nothing to match at.

The table cell above claimed the newline kept regexes from spanning faces. It keeps . from spanning faces. It keeps nothing else out, and a pattern that spells the separator's own characters was matching it happily. Measured against api.scryfall.com, 2026-08-28 (its answer, against what a joined column answers):

query	Scryfall	joined column	what the difference is
o:/\ndraw/	381	389	the 8 extras every one a two-face card whose back face opens with "Draw" — the separator ends in a newline, so "\n//\nDraw…" contains "\ndraw"
o:/\sdraw/	3,604	3,611	the same eight, \s reaching that newline
o:/\nwhenever/	3,839	3,885	and 46 more of them
o:/\/\//	1	849	the one real card is SP//dr
o:"//"	1	849	the same query after lower_literal_regexes lowers the literal
ft:/\/\//	0	262	flavor text is joined the same way
t:/\/\//	930	930	and the type line is not ours to undo — see below

The fix is to split the stored value back at match time and run the pattern, or the needle, per segment. No store change, no reimport, no schema change, no ARCHIVE_FORMAT_VERSION-style bump: the bytes were always right and only the reading of them was wrong. A single-face card has no separator, so the split yields one segment and costs one scan for a needle that is not there.

The substring path is not the exotic half, it is the half users reach

o:/\/\// is a plain literal, so lower_literal_regexes rewrites it into the contains-predicate o:"//" before the engine sees a pattern at all. A fix applied only to the regex arm would have left the largest row of that table exactly where it was.

Five sites in card_engine/src/filter.rs, because a text predicate on a joined column reaches an answer five different ways and all five had to agree:

• the two evaluation arms, TextRegex and TextContains
• the two bind-time FlavorMatch rewrites — ft: never reaches evaluation at all, so a fix applied only there would have left the 262 where it was
• the verify beneath the memoized OracleMatch set

The #734 trigram narrow above that verify is untouched and stays sound either way — a face is a substring of the join, so every literal factor the narrow requires of a survivor is still present in the joined text it indexed — but the verify is the answer, and a needle straddling the separator must not survive it.

The SQL fallback needed the same semantics

Otherwise _search_sql and _search_engine answer one query two ways, which is its own bug. oracle_text/flavor_text leaves now compile to a per-face EXISTS over string_to_array(col, separator):

• the : form keeps the whole-value LIKE as its leading conjunct, so gin_trgm_ops still drives the scan and only survivors pay for the split. Sound because a face is a substring of the join, so that conjunct can only ever be a superset.
• the regex form gets no such prefilter. ~* is not newline-sensitive, so after the split ^/$ bind at each face's ends and a face can match where the join does not; ANDing the join's answer in would drop real matches.
• CASE WHEN col IS NULL THEN NULL ELSE EXISTS (…) END keeps the leaf three-valued. EXISTS is two-valued, and flavor_text is nullable: without the guard, -flavor:/x/ would start matching every printing that has no flavor text at all.

Verified row-for-row against a real PostgreSQL on the fixture below: engine and SQL return identical sets for all 18 queries tried, negations and the NULL column included.

Anchors bind per face now, and that is the one place this adds matches

compile_search_regex builds with (?i) and no (?m), so ^/$ bound at the ends of the whole joined value. Splitting binds them at each face's ends, which is what Scryfall does — it anchors inside the face it is matching. o:/^draw/ therefore starts matching a card whose back face opens with "Draw", which is correct. It still does not bind per line the way Scryfall's Ruby anchors do; that divergence is older than this PR and untouched by it.

Three columns join faces and only two separators are ours

type_line joins with " // ", which is Scryfall's own top-level field for a split card ("Instant // Instant") — t:/\/\// answers 930 there, so splitting it would be inventing a bug rather than fixing one. It is also not a search column here at all (t: reads card_types). That asymmetry is why the rule is a per-column decision, FACE_JOINED_TEXT_COLUMNS, asserted in both directions.

The separator itself moved from a private constant in api/card_processing.py to api/parsing/db_info.py, where the SQL generator can reach it, and the engine's copy is exported to Python so a test can pin the two together. Nothing else would have noticed them drifting: the importer would keep joining with one string and the matcher would keep splitting on the other, silently answering from the seam again, with every other test still green.

Tests

A three-card fixture separates the bug from everything that must not move: the two-face card whose back opens on "draw", the single-face control whose own newline precedes it, and the two-face card with a real internal "\ndraw" that splitting must not throw out.

• three engine tests over that fixture — the eval walk, the ft: bind rewrites (asserting the rewrite actually fired, so the test cannot silently measure the wrong path), and the memoized o: verify
• one end-to-end class in api/tests/test_engine_unit.py that goes through parse_scryfall_query, because o:/\/\// never reaches the engine as a regex and a hand-built filter would miss the path users take
• the type_line negative control, in both directions, and the Rust/Python constant-parity check
• every SQL-shape test in test_sql_gen.py / test_regex_patterns.py updated to the new shape, with a named face_split_like() helper so the intent reads rather than the string

All three engine tests were run against the unsplit code as a negative control and fail there.

Suites: cargo test 169 passed / 56 ignored; cargo clippy --all-targets -- -D warnings on the pinned 1.97.1 clean; pytest 3,447 passed (testcontainers excluded); ruff check clean.

Also in this push: upstream/main merged in (through #1011), so the branch is current.

---

Update: the merged row's printed cost is every face's, and the card object is not the evidence (9a83786)

The table at the top of this PR says the merge takes mana_cost from the front face because that "matches Scryfall's own top-level fields (verified on its card objects)". The card objects are right and the conclusion is wrong: Scryfall's mana: haystack is not its top-level mana_cost field.

Reading the objects gives a clean layout table, and it holds over all 5,193 faced printings in the 2026-08-28 default_cards bulk with 0 exceptions — the field is PRESENT on the one-image layouts (split 350, adventure 459, prepare 95, flip 45) and ABSENT on the two-image ones (art_series 2,650, transform 1,065, modal_dfc 328, double_faced_token 120, reversible_card 81). Search does not follow it. Probed against api.scryfall.com, 2026-08-28, each as the card's own !"…" ANDed with the pattern so the corpus filters cannot confound the count:

card	layout	probe	Scryfall
Extus, Oriq Overlord // Awaken the Blood Avatar	modal_dfc — no top-level mana_cost at all	mana:/\/\//	1 — the decisive row
		mana:/{b}{b} \/\/ /	1 — a pattern SPANS the seam, so it is one string and not a set
		mana:/{r}$/	1 — ending in the BACK face's last pip
Fire // Ice	split	mana:/^{u}$/	0 — the back half alone is not a value of its own
Delver of Secrets // Insectile Aberration	transform	mana:/^{u}$/	1 — an EMPTY back face contributes nothing…
		mana:/^{u} /	0 — …not even a separator
Westvale Abbey // Ormendahl, Profane Prince	transform, costless on both faces	mana:/^$/	1 — so an all-costless card is EMPTY…
		mana:/\/\//	0 — …never " // "

Corpus-wide the same day: mana:/\/\// is:mdfc is 40 of 100, and mana:/^$/ is:artseries is 2,243 of 2,243.

The change

mana_cost_text joins the faces' non-empty costs with " // ". That reproduces Scryfall's own field byte for byte wherever it sends one (949 of 949) and supplies the same string on the layouts where it sends none.

It joins by going into _FACE_JOINED_TEXTS beside type_line — and, like type_line, not into FACE_JOINED_TEXT_COLUMNS, because the separator is Scryfall's rather than this branch's and matching must leave it whole. The key == "type_line" test in the merge becomes a _JOINED_WITH_SLASHES tuple, so the two-group split is a named constant asserted in both directions rather than a string comparison.

mana_cost_jsonb and devotion do not move: both are computed per face from that face's own cost and the merge keeps the front's, paired with the card's own cmc. Only the printed STRING joins.

is:hybrid had to be pinned to the front face, and it is exactly two cards wide

BOOLEAN_IS_TAGS["hybrid"] reads cards.mana_cost_text, so this change would have moved it. Scryfall's is:hybrid asks only about the front face. Replaying all three candidate readings over the 2026-08-28 default_cards bulk and diffing against all 603 of api.scryfall.com's is:hybrid, fetched card for card:

reading	answers	misses of the 603
the front face's cost	617	0
the joined cost	619	0, plus two extras
layout-gated (front on a two-sided layout, whole cost otherwise)	617	0

The two extras are Abigale, Poet Laureate // Heroic Stanza and Lluwen, Exchange Student // Pest Friend — prepare printings whose only hybrid pip is on the back, and Scryfall calls neither hybrid. The layout-gated reading is the rule stated exactly and answers the same 617, because no split, adventure or flip card in the corpus carries a hybrid pip only on its back; so the expression coalesces to raw_card_blob->'card_faces'->0->>'mana_cost' instead — the same set for less SQL, and available only because this PR already made the blob verbatim. (The 14 cards both readings add over 603 are from sets outside Scryfall's default filters.)

phyrexian is deliberately left reading the column: the same replay gives 76 either way, missing 0 of Scryfall's 73, because its /P} pips never sit on a back face alone and its oracle_text half already spans every face.

Tests

Three in TestFaceMerging, all three failing on the previous behaviour:

• the split join, with mana_cost_text in _FACE_JOINED_TEXTS and not in FACE_JOINED_TEXT_COLUMNS asserted in both directions — the same pair the type_line test already pins
• the costless-face drop in both its shapes: one empty face ("{U}", not "{U} // ") and both empty ("", not " // ")
• mana_cost_jsonb staying the front's while the string joins

Suites: pytest --ignore=api/tests/test_integration_testcontainers.py 3,450 passed; cargo test 169 passed / 56 ignored; cargo clippy --all-targets -- -D warnings on the pinned 1.97.1 clean; ruff check clean.