Specializer wave B: regexp family — corpus 484 -> 505 (TASK-53)

Cargo.toml

@@ -16,5 +16,6 @@ cranelift-frontend = "0.126"
 cranelift-jit = "0.126"
 cranelift-module = "0.126"
 pyo3 = { version = "0.29", features = ["abi3-py314"] }
+regex = "1"
 sqlparser = "0.62"
 target-lexicon = "0.13.5"

backlog/tasks/task-53 - Specializer-wave-B-—-regexp-family-regexp_-functions-~-~-SIMILAR-TO-deferred-star-forms.md

@@ -0,0 +1,65 @@
+---
+id: TASK-53
+title: >-
+  Specializer wave B — regexp family (regexp_* functions, ~/!~/SIMILAR TO,
+  deferred star forms)
+status: Done
+assignee: []
+created_date: '2026-07-26 23:50'
+updated_date: '2026-07-27 00:35'
+labels: []
+milestone: m-7
+dependencies:
+  - TASK-52
+priority: high
+type: feature
+ordinal: 47000
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+Post-wave-5 pool (census at master ba8bf98: 484 match / 192 unsupported / 2 known-divergent): the regexp bucket is the largest coherent workable slice (~25-30 cases + unlocks). Scope:
+
+- regexp_matches / regexp_full_match / regexp_extract / regexp_replace (scalar forms; list/table-returning forms classify clean as non-scalar)
+- operators ~ and !~ (DuckDB binds them to regexp_full_match), SIMILAR TO on values
+- deferred wave-5 star forms: * SIMILAR TO 're' (unanchored name search + the pinned NOT asymmetry) and COLUMNS('re')
+
+Engine decision: Rust `regex` crate (RE2-lineage, pure Rust; NOT in the tree yet — new direct dependency). DuckDB uses RE2 — the pins fleet must run a DIFFERENTIAL battery (duckdb vs rust-regex side by side) to pin exactly where they disagree (\d/\w/\s Unicode-ness, (?i) fold scope, empty matches in replace, error classes for invalid patterns); divergent corners classify clean-unsupported rather than serving wrong answers.
+
+Pins-first; wave-3 over-generalization precedent applies — every claim needs an executed query/program recorded.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [x] #1 Pins-first: wave-B pins spec (md + JSON) committed before implementation, incl. the duckdb-vs-rust-regex differential battery and the crate decision
+- [x] #2 regexp_matches / regexp_full_match / regexp_extract / regexp_replace serve per pins (options strings, group semantics, backrefs, NULL handling); list/table forms classify clean
+- [x] #3 Operators ~ / !~ and SIMILAR TO on values serve per pins (anchoring semantics measured, not assumed)
+- [x] #4 Deferred star forms serve: * SIMILAR TO name filter (incl. the pinned NOT asymmetry) and COLUMNS('re') expansion
+- [x] #5 Patterns whose semantics differ between RE2 and rust-regex classify clean-unsupported (guard at bind time from the pinned divergence list) — never a wrong answer
+- [x] #6 Corpus replay: three-outcome contract holds, zero FAILs, match count reported
+- [x] #7 Gate green both backends, clippy clean, serving-bench parity gate passes
+<!-- AC:END -->
+
+## Implementation Plan
+
+<!-- SECTION:PLAN:BEGIN -->
+Stages (each lands with tests + corpus replay green):
+1. Regex infrastructure: regex = "1" dep (RegexBuilder::octal(true), default Unicode — unicode(false) measured to BREAK (?i) parity); translation module (perl-class rewrite \d->(?-u:\d) etc incl. in-class variants, reject list: \B, (?<>), dup group names, bounds>1000, stacked quantifiers, \u, \Q\E; options parser c/i/l/s live, m/n/p no-ops, g replace-only; replacement translation \N->${N}, $->$$, with the out-of-range->identity and bad-escape global/non-global asymmetry resolved at BIND time); Program-level regex table; IR ops ReMatch/ReExtract/ReReplace + print/parse/verify + both backends.
+2. Functions + operators: regexp_matches (search) / regexp_full_match / regexp_extract (''-on-no-match, flat 0..9 group check) / regexp_replace; ~ / !~ = full match (NOT Postgres search!); SIMILAR TO = raw-pattern full match (no % translation); NULL rules incl. the options-arg asymmetry.
+3. Star forms: * SIMILAR TO (unanchored search) / NOT SIMILAR TO (NOT full match — independent predicates) via rewrite.rs marker codes; COLUMNS('re') interception + declared-order expansion.
+4. Census + bench parity + close-out + PR.
+Spec: docs/superpowers/specs/2026-07-27-waveB-regexp-pins.md (pins committed b003122 before implementation).
+<!-- SECTION:PLAN:END -->
+
+## Final Summary
+
+<!-- SECTION:FINAL_SUMMARY:BEGIN -->
+Wave B shipped: the regexp family, corpus 484 -> 505 bit-exact of 678 (zero wrong answers, zero replay FAILs). Pins-first: 6-agent fleet (5 DuckDB areas + the RE2-vs-rust-regex DIFFERENTIAL battery) committed as docs/superpowers/specs/2026-07-27-waveB-regexp-pins.md + pins-waveB/*.json before implementation.
+
+The engine is the rust `regex` crate (new direct dependency) behind the measured parity recipe: RegexBuilder::octal(true) + default Unicode mode (unicode(false) was measured to BREAK (?i) folding parity — the 'obvious' fix is the wrong one), a bind-time Perl-class rewrite in retrans.rs (\d -> (?-u:\d) etc, in-class variants included) closing the whole RE2-ASCII-vs-rust-Unicode gap, a reject list for irreconcilables (\B unservable in DuckDB itself, (?<name>), duplicate group names, bounds > 1000, stacked quantifiers a*+ — silently WRONG in rust, not just error-shaped different, \u, \Q\E), and replacement-template translation with RE2's invalid-rewrite quirks resolved at bind (out-of-range backref = identity; global bad escape = consume-with-prefix). With that applied all 98 differential battery entries were byte-identical or identically-rejected.
+
+Infrastructure: Program grew a prepare-time regex table (print/parse round-trips it; verify checks indices and rewrite presence); ReMatch/ReExtract/ReReplace on both backends (interp closures own their compiled Regex; cranelift helpers read a CompiledRe table through Cx). Semantics per pins: regexp_matches = unanchored SEARCH, regexp_full_match / ~ / !~ / SIMILAR TO = FULL match (~ is NOT the Postgres search — the DuckDB binder error names regexp_full_match; SIMILAR TO translates NO wildcards), regexp_extract '' on no-match with the flat 0..9 group check and NULL-group -> '', regexp_replace backslash backrefs with $-literals and the NULL-options -> NULL asymmetry, options alphabet c/i/l/s live with m/n/p as measured no-ops, constant patterns compile at prepare (pinned eagerness). Star forms: * SIMILAR TO = unanchored name search, * NOT SIMILAR TO = NOT full-match (independent predicates — pinned non-complement asymmetry), bare COLUMNS('re')/COLUMNS(*) expand in declared order with alias stamping into the wave-5 dedup. List-valued forms (extract_all, split_to_array) and COLUMNS-in-expression stay clean-unsupported.
+
+Gate: 154 Rust + 557 py green, interp backend identical (548 minus the backend-identity guards), clippy clean on wave files, serving-bench parity gate passed with spec in the usual 1.4-2x band. Remaining pool after the wave: aggregation 45 + table-fns 24 (out of scope), lists/structs 25 (wave C), dynamic self-joins 10, small tails.
+<!-- SECTION:FINAL_SUMMARY:END -->

docs/known-limitations.md

@@ -0,0 +1,130 @@
+# Known limitations — deliberate, named, and loud
+
+This is the user-facing contract of the SQL specializer (`DuckDBInferFn`):
+what it refuses to serve, **why**, and what you see when you hit a limit.
+Its executable twin is [`tests/test_known_limitations.py`](../tests/test_known_limitations.py) —
+every limitation below is asserted there, so lifting one breaks a test and
+forces this document to change with it.
+
+**The contract.** For any SQL you hand it, the engine does exactly one of:
+
+1. **Serve it bit-for-bit identical to DuckDB** (verified continuously
+   against DuckDB's own test corpus: 505 of 678 statements as of wave B), or
+2. **Refuse loudly at BUILD time** — `DuckDBInferFn(...)` raises a
+   `ValueError` naming the construct. Nothing is ever silently wrong or
+   silently dropped at inference time.
+
+There is no third mode. Every limitation here is a *measured decision*
+recorded in a pins spec (`docs/superpowers/specs/`), not an accident.
+
+---
+
+## 1. The specialization bargain (inherent to the engine model)
+
+The specializer's speed comes from doing ALL general work at build time:
+parse once, bind once, compile once, freeze the static tables into the
+code. Anything that would require re-doing general work per row is
+rejected, permanently by design:
+
+| Limitation | You'll see | Why |
+|---|---|---|
+| Regex patterns must be constants (`regexp_matches(s, pattern_col)` rejects) | `unsupported: non-constant regex pattern (compiled at prepare in v0)` | Regexes compile at prepare; DuckDB compiles per row. Per-row compilation is the opposite of specialization. |
+| Replacement strings / regex options / extract group indexes must be constants | `non-constant regexp_replace replacement` etc. | Same. |
+| Static (join) tables must be provided at build time, with unique keys and no NULL values | `duplicate map key`, `has a NULL in value column` | Joins are frozen hash maps baked into the function. Duplicate keys mean 1:N join multiplicity — a designed extension (stage B, TASK-50 notes) that is deliberately not built yet. |
+| The dynamic table cannot be joined to itself | `joining the dynamic table to itself` | The batch is the probe side; using it as a build side too needs stage-B machinery. |
+| Exactly one row table drives the query | `the specializer takes exactly one row table`, `must be the dynamic table` | The serving contract is rows-in → rows-out for one entity stream. |
+
+## 2. Out of scope for row-serving (by decision, not difficulty)
+
+The engine serves **row-at-a-time feature transforms**. Whole-relation
+constructs are out of scope because their output shape is not
+one-row-in/one-row-out:
+
+- Aggregation: `GROUP BY`, `HAVING`, `sum`/`count`/`avg`/... →
+  `aggregate function ... (no aggregation in v0)`
+- `ORDER BY`, `LIMIT`/`OFFSET`, `DISTINCT` — row-independent transforms
+  don't reorder or deduplicate.
+- CTEs (`WITH`), `UNION`/`INTERSECT`/`EXCEPT`, subqueries, multiple
+  statements.
+- Table functions in FROM (`range(...)`) — there is no base table.
+- `rowid` pseudo-column — rows have no stable identity in a stream.
+- `FULL OUTER JOIN` — emits rows that no input row produced.
+
+## 3. Type-system boundaries
+
+The engine computes in exactly four types: `i64`, `f64`, UTF-8 string,
+bool. Measured consequences:
+
+- **f32 base tables reject** (`engine is f64-only`).
+- **Static-table key/value columns must fit BIGINT** — `UBIGINT`/`HUGEINT`
+  payloads outside i64 reject with a named message.
+- **Lists and structs reject** (`row column 'x' has a non-scalar type`) —
+  this is the designated next capability wave ("wave C"), and it also gates
+  `regexp_extract_all` / `regexp_split_to_array` / the STRUCT form of
+  `regexp_extract` (`list-valued — non-scalar in v0`).
+- **DECIMAL literals are f64** — a documented divergence: DuckDB types
+  `1.5` as `DECIMAL(2,1)` and does decimal arithmetic; we map to f64.
+  Values agree on every corpus case; exact-decimal accumulation semantics
+  are not reproduced.
+- Narrow integer widths don't exist: bitwise ops compute in i64, which
+  matches DuckDB whenever either operand is BIGINT (always true for
+  row-model ints). Explicit narrow CASTs are rejected rather than
+  risking DuckDB's narrow-width overflow behavior.
+
+## 4. Semantics descoped after measurement
+
+Each of these was measured against DuckDB 1.5.5 first (pins in
+`docs/superpowers/specs/`), and rejected because serving it would risk a
+wrong answer or require semantics we can't reproduce exactly:
+
+| Construct | Why it's descoped |
+|---|---|
+| `reverse()` | DuckDB reverses UAX-29 grapheme clusters (incl. regional-indicator pairing), not codepoints. A codepoint reverse would be silently wrong on emoji/accents. |
+| `^` operator | It IS pow in DuckDB, but sqlparser's precedence differs from DuckDB's (`2*x^y` would parse as `(2*x)^y`). Mapping it computes the wrong tree silently. Use `pow()`. |
+| prefix `~`, `#`, `NOT GLOB` | Same class: precedence/parse divergences that would silently mis-associate. `xor()` covers bit-xor; `NOT (x GLOB p)` works. |
+| Regex reject list: `\B`, `\Q…\E`, `(?<name>…)`, duplicate group names, bounds > 1000, stacked quantifiers (`a*+`), `\u` escapes, negated Perl classes inside `[...]` | The RE2↔rust-regex differential battery (98 entries) proved these are the constructs where the engines disagree or DuckDB itself is broken (`\B` crashes DuckDB at runtime on non-ASCII). Everything else is byte-identical. |
+| `SIMILAR TO ... ESCAPE` | Not implemented in DuckDB itself. |
+| `* EXCLUDE (t.key)` on a USING join | DuckDB UNMERGES the coalesced column (it reappears at the right table's position) — measured, not modeled. Unqualified EXCLUDE works. |
+| `BETWEEN`/`IN` mixing non-numeric string literals with numbers | DuckDB converts at EXECUTION time (an empty input succeeds!); a bind-time conversion was measured to be over-eager. Numeric literals convert fine. |
+| `COLUMNS(...)` inside expressions, lambda/list forms | Only bare `COLUMNS('re')` / `COLUMNS(*)` as select items are served. |
+| Bare `NULL` with no typing context, `CASE` where every branch is NULL, `COALESCE`/`NULLIF` of only NULLs | DuckDB's SQLNULL type has no i64/f64/str/bool home. `NULL <op> NULL` IS served with the measured result types. |
+
+## 5. Deliberate contract choices (behavior differs from raw DuckDB surface)
+
+These are served, but with a consciously chosen surface — know them:
+
+- **Duplicate output column names are renamed**, using DuckDB's own
+  boundary-rename algorithm (`id, id, id_1` → `id, id_1, id_1_1`;
+  case-insensitive collision check). Raw DuckDB keeps duplicates at the
+  top level, but a pydantic model or dict cannot — and this rename is
+  bit-identical to what DuckDB itself does at every subquery/CTE/CTAS
+  boundary and in `.df()`. Verified against `.df()` in tests.
+- **`NULL || NULL` types as VARCHAR** in the output model (value is NULL
+  either way; DuckDB's SQLNULL materializes as INTEGER).
+- **Error TEXTS are approximate where noted.** Runtime traps
+  (overflow, shifts, substring range) reproduce DuckDB's message bodies
+  verbatim; some bind-time rejections (star-filter zero-match, regex
+  compile errors) use our own wording with the same error class. The
+  corpus only ever compares successful results, so texts never affect
+  parity.
+- **Two known oracle divergences** (excluded from the corpus by name in
+  `tests/test_corpus_replay.py::_KNOWN_DIVERGENT_SOURCES`): DuckDB
+  behaviors that depend on column STATISTICS (e.g. ILIKE's NUL handling
+  selects a different kernel depending on *sibling rows*). A row-at-a-time
+  engine cannot reproduce statistics-dependent semantics even in
+  principle; the engine is NUL-transparent (the ASCII-kernel behavior).
+- **`%`-by-zero NaN bit pattern is platform-libm** — pinned as
+  engine==oracle bit agreement per platform, not a constant.
+
+## 6. How to read a rejection
+
+Every rejection is a `ValueError` at `DuckDBInferFn(...)` construction
+whose message starts with a classification:
+
+- `unsupported: ...` — real SQL, deliberately not served (this document).
+- `parse error: ...` — the dialect surface ends here.
+- `bind error: ...` — the query is wrong against YOUR schema (typo,
+  type mismatch), not a limitation.
+
+If a message you hit isn't in this document or the tests, that's a bug in
+our bookkeeping — file it.