Import: Skip the Memorabilia Sets Scryfall Hides, and Oversized Printings With Them

[daveycodez]

Based on main, independent of every open PR.

What Scryfall does

set_type: memorabilia printings — World Championship decks, Collectors' Edition, 30th Anniversary, the oversized promos, 99 sets — are hidden from any search that does not name their set. Measured against api.scryfall.com on 2026-08-11:

!"Ancestral Recall"      9 printings returned, 18 exist
!"Birds of Paradise"    42 printings returned, 43 exist

set:cei / e:cei / s:cei  the hidden printing, on its own
-set:lea                 all 17 others — every hidden set reappears
(set:cei or t:instant)   all 18
is:reserved              14  } together exactly 18, so BOTH polarities
-is:reserved              4  } run against the unhidden corpus
is:oversized             the oversized printings

border:black  year<=1994  r:rare  cn:48  number:48  in:cei  st:core
frame:1993  prints>1  lang:en  game:paper  is:promo  is:digital
is:foil  is:nonfoil  is:fullart  is:booster
                         no unhiding — only the 9 visible printings

Why it matters here

Importing them makes ordinary queries disagree. Concretely, they supply the cheapest printing for 184 cards — exactly the printing a price ordering is defined to return. order=usd was ranking Birds of Paradise by a $4.98 World Championship copy where Scryfall ranks it by a $9.60 one, and Counterspell by a $1.29 wc00 where Scryfall uses brb at $2.30.

Why at import, not at query time

This is the load-bearing decision, and it was made on measurement — I built the query-time version first and threw it away.

To be correct, a query-time predicate has to live in the filter tree. The representative walk picks the prefer best among printings that pass the residual, so a flag outside the tree lets a hidden printing stand for its card; and total is derived five different ways across the plans, so a predicate the filter does not carry has to be reimplemented in each.

But a conjunct present on every query breaks four of the six physical plans:

plan	gate it fails
PlanePopcountOrder	residual must be literally FilterExpr::True
all_match_known (both match kernels)	same
CardRangePopcount	range must be BARE
PrintingRangeScan	same

None can be taught to look through it. The two range fastpaths emit straight from a value-index slice and report the slice width as their total, so looking through would emit hidden printings and miscount them in one step. Measured cost of the query-time version: +59–115 µs per query on a 112,932-printing store, with PlanePopcountOrder — the plan serving the default unique=card plane shape — lost on every query.

Import-time exclusion costs none of that, and makes the store smaller rather than larger.

What it gives up

set:cei and its 98 siblings return nothing, where Scryfall returns them.

No card is lost. Measured over a 31,724-card corpus: 0 cards are printed only in memorabilia sets. So this changes which printing represents a card, never whether the card is findable. 2,672 of 97,803 printings (2.73%) stop being imported.

Oversized printings are excluded too

Per the review question, resolved by measurement in the decision comment (2026-08-27 against api.scryfall.com):

• Never paper-legal. Legalities are name-level: of 726 oversized prints, the 240 showing legal/restricted are all oversized copies of real cards whose normal printings carry the actual legality.
• Never the only printing that matters. 343 of 546 oversized names exist only oversized — every plane (207/207), every scheme (102/102), every paper Vanguard avatar (32/32), Garruk the Slayer — but all 343 are not_legal in every format, so the existing legality gate already refuses each of them before this check runs.
• Scale. After the legality/paper/funny gates, exactly 240 oversized printings survive today; 230 are memorabilia, already excluded above. The oversized exclusion removes the last 10: the p09/p10/p11 oversized box-topper promos (Lightning Bolt, Wurmcoil Engine, Sun Titan, …), each of which has normal-sized printings. 0 cards lost here either.

An earlier revision added an is:oversized tag instead; with oversized never imported the tag would be permanently empty, so it is gone.

Tests

test_preprocess_card_filters_memorabilia_sets, test_preprocess_card_keeps_ordinary_sets, and test_preprocess_card_filters_oversized_printings — the keep-side test paired so a predicate that dropped everything fails rather than looking like a working filter.

The memorabilia test pins the literal set_type rather than MEMORABILIA_SET_TYPE, with the constant asserted separately. Driving both sides off the same constant made it self-referential — it passed with the constant set to anything at all. Caught by flipping the constant and watching it stay green.

pytest api/tests/test_card_processing.py api/tests/test_parsing_errors.py → 72 passed. ruff check api/ clean.

[jbylund]

Should we just exclude oversized printings from import entirely? are those ever paper legal? are they every the only printing of a card?

On memorabilia sets - excluding them at import makes them never recoverable, always filtering on a printing varying condition makes every scan walk printings instead of cards, which is rough.

[daveycodez]

Rebased onto main @ ccb562a (the admin split moved the tag sync into admin_resource.py; the oversized entry rides BOOLEAN_IS_TAGS there now).

One thing worth knowing before this merges: the Cloudflare port tried this exact approach — drop memorabilia at import — and then reversed it. An absent row cannot reproduce a query-time gate in either direction: /cards/named?exact=Counters answered 404 where Scryfall answers fmsc/9, and include_extras=true had nothing to include. What ships there now imports every row, tags the extras class at import, and ANDs -is:extra at query time unless the caller or a set term asks otherwise. The is:oversized half of this PR is wanted either way; the import filter half may deserve the same second look.

[daveycodez]

Ran your two questions against api.scryfall.com (2026-08-27):

Ever paper-legal? No. Legalities are name-level, not printing-level: of 726 oversized prints, the 240 showing a legal/restricted format are all oversized copies of real cards (Vintage Championship trophies, Commander's Arsenal oversized commanders) whose normal-sized printings carry the actual legality. Dropping oversized loses no legality information.

Ever the only printing? In Scryfall's universe, yes — 343 of the 546 oversized names exist only oversized: every plane/phenomenon (207/207 — even the physically normal-sized planar cards in recent commander decks carry oversized: true), every scheme (102/102), every paper Vanguard avatar (32/32), plus Garruk the Slayer (ppc1) and one token. But all 343 are not_legal in every format, so preprocess_card's legality gate already refuses each of them before set_type is ever consulted. Within this importer's corpus, an oversized printing is never the only printing of a card.

Scale: after the existing legality/paper/funny gates, exactly 240 oversized printings survive today, and 230 of those are memorabilia — already excluded by this PR. Excluding oversized entirely removes exactly 10 more printings: the p09/p10/p11 oversized box-topper promos (Lightning Bolt, Wurmcoil Engine, Sun Titan, …), each of which has normal printings.

So agreed — excluding oversized at import entirely is safe here, and it makes is:oversized permanently empty, so I'm updating the PR to do both: drop oversized printings at import alongside the memorabilia exclusion, and drop the is:oversized tag with them. One heads-up so the divergence is deliberate: our Cloudflare port keeps importing them — full Scryfall parity is the goal there, including the never-legal classes (its is:oversized answers Scryfall's own 546 names) — so the two codebases intentionally differ on this one.

Footnote while measuring: st:memorabilia now matches 4 cards in Scryfall's default mode (the three oafr oversized dungeons + oclb Undercity // The Initiative) — drifted from the "matches nothing" I measured on 08-11.