Engine+Parsing: Compile the Full Regex Dialect, and Stop t:/…/ Matching Nothing

[daveycodez]

Goal

Bring the in-memory engine to full capability on regex, so the SQL fallback stops being load-bearing and becomes what it was meant to be — a crash net.

_search routes to PostgreSQL whenever _search_engine raises. That handler reads as a backstop, but several ordinary, documented queries land in it, which means the SQL path is quietly answering things the engine was expected to answer. This PR closes the regex portion of that set.

Motivation: the Cloudflare Workers port runs this engine compiled to wasm. There is no PostgreSQL there, so every one of these declines has to be absorbed by a D1/SQLite mirror of the corpus that exists purely to catch them — an expensive thing to maintain for queries the engine should be answering itself. That is the motivation, but the bugs below are upstream bugs and the fixes stand on their own here.

What was declining

query	Scryfall	before	after
o:/draw (?!two)/ t:instant	435	decline → SQL	engine
o:/(?=.*sacrifice)draw/	33	decline → SQL	engine
o:/(?<=draw )a card/	2,542	decline → SQL	engine
name:/\yizzet\y/	20	decline → SQL	engine
s:/^m1/, layout:/^trans/, wm:/^guild/, cn:/^1/	(rejected)	decline → SQL	engine

Lookaround. The regex crate omits it by design — it is what buys the linear-time guarantee — but lookahead is on the documented feature list in docs/changelog/2025-02-02-regex-search.md, and Scryfall answers it.

ARE escapes. PostgreSQL spells word boundaries \y/\Y/\m/\M and end-of-string \Z. Scryfall accepts name:/\yizzet\y/; the regex crate rejects all of them.

Attributes with no TextField. Regex compiled onto four fields while the parser emits a RegexValueNode for nine.

And one that was worse than a decline

t:/…/ never reached the engine as a regex at all. kwargs() ran self.rhs.value.strip().title() and emitted the result as a literal subtype, so t:/^drag/ became the subtype "^Drag" and t:/goblin|elf/ became "Goblin|Elf" — types no card has.

The query returned nothing, silently, on both the engine and SQL paths. No error meant no fallback either, so this one was invisible from every direction:

query	Scryfall	before, measured
t:/goblin|elf/	1,269	0
t:/^legendary/	4,316	0
t:/dragon.*spirit/	18	0
t:/elf (warrior|shaman)/	180	0

test_parser_parity.py structurally cannot catch this — it compares the two parsers against each other, and both mangle it identically.

Bare literals were never affected: lower_literal_regexes already rewrites t:/dragon/ to t:dragon, and I measured that identical against Scryfall (445 = 445, likewise for creature/elf/aura/legendary/equipment). t:/^legendary/ at 4,316 against t:legendary at 4,348 is the case that shows the two are not the same predicate: the anchored regex matches type lines starting with the supertype, the lookup matches it anywhere.

Approach

Two-tier compilation (card_engine/src/regex_compat.rs). CompiledRegex tries the regex crate first and reaches for fancy_regex only on patterns it rejects. Everything that compiles today keeps the linear engine — and with it the #734 trigram narrowing, which reads the pattern back through regex_syntax::parse. A backtracking pattern yields no literal factors and scans, which is already the behavior for any regex without a usable factor.

Escape translation. \y→\b, \Y→\B, \Z→\z are exact equivalents, so those patterns stay linear; \m/\M have none and become lookaround. Bracket expressions are copied through untouched — inside […] these are ordinary escapes, not constraints.

TextField::TypeLine reads the interned type_line_id already present on AOracleCard, card-level like Layout, since Scryfall's type line is oracle data.

Cost model

One constant, measured rather than assigned. bench_backtrack_engine is self-contained (no real.store needed) and reports two things:

• On patterns both engines accept, they are the same speed — 1.00x.
  fancy_regex delegates to regex whenever a pattern needs nothing more, so routing through the backtracking arm costs nothing as a dispatch choice.
• Lookaround itself costs 77x the linear engine per candidate (6,535 vs
  85 ns/card, mean over negative lookahead / positive lookahead / lookbehind).

So REGEX_BACKTRACK_NS100 = 380_000 prices lookaround, not dispatch. It dwarfs every other tier deliberately: these are the one node kind the trigram narrow cannot read a literal factor out of, so ordering them last in an And is the only lever the model has. This wants re-fitting against the real corpus with the calibration bench — I do not have one built.

Verification

Correctness work, so per docs/workflows/performance-pr-workflow.md the acceptance is differential rather than a benchmark story. 164 engine tests and 1,676 parser tests pass; clippy clean.

Run over a real corpus: 30,658 oracle cards, built from Scryfall's oracle_cards bulk data through this repo's own preprocess_card, one row per oracle id so card-level fields are unambiguous. The reference applies Python's re to the field the query names and counts distinct oracle ids — it shares no code with the engine, and for the PostgreSQL-only escapes it uses the independent equivalent spelling (\y → \b) rather than checking the translation against itself.

query	before	after	reference
o:/draw (?!two)/	decline → SQL	3,630	3,630
o:/(?=.*sacrifice)draw/	decline → SQL	247	247
o:/(?<=draw )a card/	decline → SQL	2,900	2,900
o:/^\{T\}:/	decline → SQL	1,035	1,035
name:/\yizzet\y/	decline → SQL	14	14
name:/\mizzet/	decline → SQL	14	14
t:/goblin|elf/	0, silently	1,157	1,157
t:/^legendary/	0, silently	3,926	3,926
t:/dragon.*spirit/	0, silently	16	16
t:/elf (warrior|shaman)/	0, silently	171	171
t:/^drag/	0	0	0
t:/dragon/	402	402	(lowered to t:dragon)

13 of 13 match. The before column is measured the same way, with this branch's engine and the pre-change parser.

One thing I deliberately did not do is compare absolute totals against the Scryfall API. The import drops digital-only and funny-set cards — 18,889 of the 116,694 rows in default_cards — so the corpus is a subset by design and the totals should not line up. Scryfall settles what these queries mean; the reference settles whether the engine evaluates them correctly.

One question for you

Regex on set code / layout / watermark / artist / collector number is a superset of Scryfall, which rejects all five with 400 All of your terms were ignored. I kept them, because the parser already accepts the syntax and the SQL path already answers it — declining in the engine only re-creates the fallback. But if you would rather match Scryfall, the right fix is rejecting them at the parser, and I am happy to switch.

Design notes: docs/issues/local-engine-regex-decline-parity.md (named local- per the conventions; rename to an issue number if you file one).

---

Follow-up: two more predicates that answered nothing, silently

A broad differential sweep against api.scryfall.com (328 queries) found two more members of the
class this PR is named for — a query that returns zero with no error, no warning and no fallback —
and both belong here rather than on a branch of their own.

^ and $ were not line anchors

o:/^Whenever you cast/ e:khm missed Firja, Judge of Valor, whose oracle text is
"Flying, lifelink\nWhenever you cast your second spell each turn, …". The pattern was compiled
(?i), so ^ could only mean the start of the whole string. Measured 2026-08-16:

query (e:khm)	Scryfall	before
o:/^Whenever you cast/	11	6
o:/lifelink$/	4	1
o:/^Flying$/	20	0
o:/Flying.Whenever/	0	0
o:/Flying\nWhenever/	6	6

^/$ are line anchors and . still stops at a newline — exactly PostgreSQL ARE's
newline-sensitive mode, the (?n) the SQL path this dialect tracks would spell. In the regex
crate that pair is (?im) with s left off. Oracle text is the only multi-line column, so nothing
else moves.

The flag string is now a constant, because two callers recover the raw pattern by removing that
exact prefix — regex_tier and #734's literal-factor extraction — and a prefix that stops matching
costs the trigram narrow silently (a case-folded HIR yields classes, not literals, so no factor
comes back). query_regex_flags_stay_strippable is the test that would catch it.

A quoted multi-word type was a subtype nobody has

t:"artifact creature" reaches build_filter as ONE title-cased token, "Artifact Creature" —
neither a type nor a subtype, so TypeCmp gets an empty mask and the subtype containment gets a
value no card carries. 360 rows on Scryfall (cmc<=2), zero here. Scryfall's rule, measured:

query	Scryfall
t:"artifact creature" cmc<=2	360
t:"creature artifact" cmc<=2	0 — order matters, so it is not and-over-words
t:"tifact creat" cmc<=2	360 — neither side is word-anchored
t:"artifact creature" cmc<=2	360 — whitespace runs collapse
t:"creature — human" e:dsk / t:"creature - human" e:dsk	44 / 0 — the em dash is text
t:"human wizard" e:dsk / reversed	6 / 0 — subtype pairs, same predicate

It compiles to a regex over the escaped literal, so TextField::TypeLine — added by this PR —
carries it, with case folding free and the #734 trigram narrowing intact. Containment operators
only; =/</> compare sets and a substring is not one. Single-word values keep the membership
path and its indexes.

Why here. TextField::TypeLine exists only on this branch, and both fixes are the same defect
class this PR opened with. The SQL twin of the quoted-type rule (card.type_line ~* over the same
escaped literal, same whitespace rule) is on #926 with the rest of that sweep's parser work.

Verified on the Cloudflare port against a policy-clean set (e:dsk, unique=prints):
t:"human wizard" e:dsk 6 = 6, t:"snow land" e:khm 17 = 17, t:"legendary creature" e:dsk
34 vs 35 (one printing that corpus excludes), o:/^flying, lifelink$/ e:khm 1 = 1.

A SINGLE-WORD t: is a substring too — and nothing was narrowing any of them

The commit above fixed the quoted phrase and left the single token on the membership path, where
t:creat cmc<=2 e:khm answered 0 against Scryfall's 39. There is one rule for both. Measured
against e:khm (323 prints), 2026-08-16:

query	Scryfall	what it shows
t:creature / t:creat / t:reature / t:eatur	151 each	unanchored on both sides
t:snow / t:no	47 / 47	"no" inside "Snow"
t:elf / t:lf	22 / 25	"lf" also inside "Wolf"
t:legend / t:legendary	42 / 42	supertypes are in the line
t:CREAT / t:Creat / t:creat (cmc<=2)	39 each	case-insensitive
t:— / t:"—"	227 / 227	the em dash is ordinary text
t:"// creature" / t:"creature //"	182 / 0	so is the face join
-t:creat	172, and 151 + 172 = 323	a plain complement
t=creature / t="legendary creature"	151 / 32	= is the same substring — set equality answers 0 for the second
t:artifactcreature	0	not a token-set test

So :, >= and = are all the substring. <, <=, > and != keep this project's
set-comparison meaning: Scryfall returns zero rows for those shapes, so there is nothing measured
to follow and the superset stays. The = change is a deliberate divergence from the SQL twin
and is called out here rather than buried: t=creature means "the type array is exactly
[Creature]" in SQL and "the type line contains 'creature'" in the engine after this.

The narrowing is the substantial half. Routing a type predicate to TextField::TypeLine was
correct and had no narrowing arm at all — there is no trigram index over type lines and the #734
factor arm is guarded to name/oracle — so t:"artifact creature" fell through narrow_rec to the
catch-all and ran a regex over every card. Extending that to t:creature would have made the most
common filter in the syntax a corpus scan.

The type line is a tiny vocabulary and that is the whole design: 526,865 real rows carry 3,965
distinct type lines, 127 KB of text. CardIndexes gains type_lines (dense line id → global
string id, their lowercase copies, and a CSR to the cards carrying each) and bind_type_lines
evaluates the predicate once per distinct line — the same shape bind already uses for artist and
flavor text. A card's line id IS its type-line identity, so the winners expand to the EXACT card
set: the candidate arrives tight, with nothing to verify per card.

Measured on a 3,281-card store, best of 30, against an 88 us paging floor and a 1,668 us full scan:

query	with the index	CARD_ENGINE_NO_TYPE_LINE_INDEX=1
t:creature (1,875 rows)	108 us	367 us
t:creat	108 us	351 us
t:elf (65 rows)	39 us	280 us
t:"artifact creature"	68 us	445 us
t:zzzz (bind only)	6 us	234 us
t:creature c:r cmc>=4	112 us	146 us

Two details earn their bytes. The index's lowercase copies: matching (?i)creature against
mixed-case lines costs ~160 us per query because the prefilter fires on every case variant of the
first byte, where memmem over pre-lowercased bytes costs ~5 us. And broad results come back as a
bitmap on the same 1/32 rule HybridTagIndex stores by — t:creature is 410 bytes that way
against 7.5 KB of ids gathered by random access, which is the shape TypeCmp used to hand over
from the type bit planes and why replacing the mask costs ~20 us rather than ~150 us. A user's own
t:/…/ still runs against the original-case line and now narrows through the same index.

ARCHIVE_FORMAT_VERSION → 2026081601: no struct size moves, so an archive without the index would
answer every t: query with zero rows and nothing would say so.

o: searches oracle text with the reminder text taken out

Reported on this PR before it was fixed; it is fixed here now, because it is the other half of the
o:/lifelink$/ e:khm case above (2 vs 4 even after $ learned to anchor per line — only the
stripped form ends a line on lifelink).

Measured 2026-08-16, o: against fo::

query	o:	fo:
o:"damage dealt by this creature also causes"	0	71
o:"you may pay an additional"	0	268
o:"level up only as a sorcery"	0	25
o:"mana abilities can't be targeted"	0	24
o:/\(/	0 corpus-wide	fo:/\(/ e:khm 148

The last row is the general statement: not one parenthesis survives, so it is EVERY parenthesized
run and not only the ones on their own line. ft: is untouched (ft:/\(/ still 47).

The whitespace rule decides which side of the parenthesis goes, and it is measured too:
o:/\{e\}\sequal/ matches Aetherflux Conduit ("…an amount of {E} (energy counters) equal to…")
and o:/\{e\}\s\sequal/ does not, so exactly one space survives a mid-line reminder; and
t:saga o:/^$/ returns 233 — every Saga — so the empty line a leading reminder leaves behind is
still there. Eating the whitespace after the ) would have lost both.

oracle_text_lower_id is the whole change: it is read by TextField::OracleTextLower,
TextSearchField::OracleTextLower and the oracle trigram index, and emitted by nothing, so the
search surface moves without touching a byte of any card object. It makes the archive smaller —
over 526,865 rows the column goes 30,259 distinct texts / 5,196,005 bytes → 30,063 / 4,199,590, so
973 KB less, since the stripped form is shorter and a card with no reminder text interns to the id
it already had.

ARCHIVE_FORMAT_VERSION → 2026081602 for the same reason as above: the strings, the trigram index
and its word dictionary are different bytes for the same card, and no struct size says so.

Not implemented here: fo: itself. Scryfall's full-oracle operator has no column, alias or
index in this project, so adding it is a db_info entry plus a second searchable text — a separate
change, and one the SQL path can serve from oracle_text directly.

Verified on the Cloudflare port, local mirror against api.scryfall.com, 2026-08-16:
t:creat cmc<=2 e:khm 39 = 39, t:creature cmc<=2 e:khm 39 = 39, t:reature e:khm 151 = 151,
t:snow/t:no 47/47 = 47/47, t:elf/t:lf 22/25 = 22/25, -t:creat e:khm 172 = 172,
o:"damage dealt by this creature also causes" 0 = 0, o:/lifelink$/ e:khm 4 = 4.

...and fo:/fulloracle: gets the text o: stopped reading

The commit above is only half an answer on its own: once o: stops searching reminder text,
nothing searches it, and Scryfall has an operator that does. The parser half (both spellings
resolving to oracle_text's column) is on #926; this is the engine half.

query	o:	fo:
"damage dealt by this creature also causes"	0	71
"you may pay an additional"	0	268
draw e:khm	39	57
/\(/	0 corpus-wide	fo:/\(/ e:khm 148

OracleCard gains oracle_full_lower_id. The cost was measured before it was paid: only
9,769 of the corpus's 30,259 distinct oracle texts differ from their stripped form, so the
interner charges 2.17 MB for the whole store and a card with no reminder text shares the id it
already had — against the 973 KB the strip itself gave back.

Deliberately index-free. o: carries the trigram index because it is the common operator; a
second one over the unstripped text would cost ~5 MB to serve a rare one. fo: evaluates per card,
exactly as o: did before its index existed, and its field falls through narrow_rec to the scan
on purpose rather than by oversight.

Verified on the Cloudflare port against a local mirror, 2026-08-16: fo:draw e:khm 57 = 57,
o:draw e:khm 39 = 39, fo:lifelink e:khm 8 = 8, fulloracle:lifelink e:khm 8 = 8,
fo:"level up only as a sorcery" 25 = 25, and ft:/flavor: unchanged at 7 = 7 (flavor text is
not stripped on either side — ft:/\(/ returns 47 on Scryfall).

A correction to an earlier note on this PR. It said the parser "unescapes a backslash in a
regex literal", because fo:/\(this creature/ reaches the engine as (this creature. That was a
misreading of a wire tree and there is no such bug: a backslash before a NON-word character IS that
character, so a pattern of nothing but literal text and escaped punctuation is a substring search,
and lower_literal_regexes rewrites the leaf accordingly. Nothing is dropped — the leaf changes
TYPE, from RegexValueNode to StringValueNode, because the query no longer needs a regex engine.

Checked rather than argued, on a local mirror against api.scryfall.com (2026-08-16), including the
pairs that discriminate a literal from a metacharacter:

query	Scryfall	mirror
o:/target./ e:khm	111	111
o:/target\./ e:khm	5	5
name:/./ e:khm	323	323
name:/\./ e:khm	0	0
o:/\{t\}/ e:khm	48	48
o:/\+1\/\+1/ e:khm	51	51
name:/\,/ e:khm	34	34
cn:/\d/ e:khm	323	323

A pattern that keeps its regex leaf keeps its backslashes byte for byte — \(a.b, ^\(x,
[\]], \bfoo all survive intact, on all eleven regex-capable operators. #926 now pins BOTH
halves as properties rather than a value table (the old table would have passed just as well if
\( were being dropped instead of resolved), and the parity fixtures cover escaped metacharacters
per operator so the two implementations cannot drift on which half a pattern is in.

Added: a t: value that names a type matches the type word (a98fae9)

t:god is 96 on api.scryfall.com and this engine answers 104. The 8 extra are every
Demigod in the corpus — the substring rule this PR measured is right about most needles and wrong
about the ones Scryfall indexes as types. (It is also an anchor that quietly poisoned three probes
in a later negation sweep, whose "extra rows" were all Demigods: a wrong anchor, not an empty one.)

The rule is both, split on Scryfall's own published type catalogs. Measured 2026-08-17:

query	Scryfall	substring rule	which
t:god	96	104	anchored
t:ape	45	273	anchored — "ape" is inside Shapeshifter, Spellshaper
t:bat	54	92	anchored — and inside Wombat, Incubator
t:ir	1,906	1,906	substring — Plane — Ir is a real type line, in no catalog
t:las	43	43	substring — Plane — Las Vegas; Bolas still matches
t:art	4,171	4,171	substring — Creature — Art Lizard; Artifact still matches

The three substring rows are why this is a fixed catalog — the union of the nine /catalog/* type
lists, 531 names — and not a vocabulary derived from the corpus's own type lines. Plane types are
printed, are in no catalog, and Scryfall does not anchor on them: a corpus-derived rule would have
anchored t:ir and answered 0 where Scryfall answers 1,906.

The boundary is a type-word boundary, so the three characters the catalog spells inside a name
bind: t:urza does not reach Land — Urza's Mine (the type is Urza's, and Urza is a separate
planeswalker type), while t:worker, which is in no catalog, still substring-matches
Assembly-Worker exactly as it does there.

Nothing else moves. Every row this PR already recorded still answers the same number, including the
ones that look like they should shift: t:creature 18,753, e:khm t:snow 47 = t:no 47, t:elf
22 against t:lf 25, t:legend 42 = t:legendary 42, t:gob 563 = t:goblin 563.

What it still cannot do: Scryfall matches the type ARRAY, which holds subtypes it never prints.
t:warrior is 1,298 there against the printed line's 1,294, because Burakos, Party Leader answers
to all four party classes while its type_line reads Legendary Creature — Orc on both sides. One
card per party class, not derivable from any published field.

A catalog is a snapshot: a creature type printed after that date matches as a substring until the
list is refreshed, which is the pre-fix behaviour and the safe direction to be stale in. The
catalog's sortedness, size and lowercasing are asserted in a test, since a silently truncated list
would quietly return t: to substring-matching every creature type at once.

cargo test --lib 170 pass, 0 fail. No stored data changes; this is query-side only.

Rebased onto the regex execution budget, and two things that fell out

#security-regex-execution-budget landed while this was open. Both sides had independently reached
for fancy-regex — that work for a backtrack budget and error taxonomy, this branch for dialect
coverage — so the merge is an integration rather than a text reconciliation. CompiledRegex stays
the compiled form and compile_search_regex became the seam that wraps it in the budget and the
error prefix; the failure latch moved to CompiledRegex::try_is_match, and this branch's
1,000,000 backtrack ceiling gave way to the budget's measured 8,192.

Two collisions auto-merged cleanly and would have been a red build, since neither side's
additions overlapped by line: a duplicate fancy-regex key in card_engine/Cargo.toml, which
cargo refuses to parse, and REGEX_BACKTRACK_NS100 defined twice in filter.rs.

The budget's parse-time validator rejected four of the operators this PR documents.
regex_budget.py measures a pattern with Python's re._parser, which does not speak ARE, so
\y/\Y/\m/\M raised bad escape \y — o:/\yizzet\y/ became a user-visible
InvalidRegexPatternError one layer before the engine, rather than the 20 rows it answers on
Scryfall. Neither side's tests covered the intersection. Fixed by running the engine's own
translate_are_escapes over a copy for measurement only: the stored pattern still reaches
PostgreSQL, which speaks ARE natively. The Rust and Python copies were cross-checked to agree on
every case, including the ones both must refuse ([\y]lit).

The backtrack-exhaustion test was vacuous under the merged design. (?=a)(a+)+b stops
exhausting the budget once a linear engine exists — fancy-regex hands the plain (a+)+b tail
straight to it. Confirmed empirically at n=20/40/100/200, then moved the lookahead inside the
repetition (((?=a)a+)+b), which trips the limit as intended.

Both fixes are a separate commit rather than buried inside the merge, since one of them changes a
security module.

One residual, left deliberately. The budget rejects backreferences outright while
CompiledRegex can execute them, so the engine now supports a construct the parser will never hand
it. That is a security policy rather than a dialect question, so the budget's position stands.