Specializer M-lower: frontend + BTA + lowering for the v0 subset

backlog/tasks/task-43 - Specializer-M-lower-frontend-BTA-produce-consume-lowering-for-the-v0-subset.md

@@ -3,9 +3,10 @@ id: TASK-43
 title: >-
   Specializer M-lower: frontend + BTA + produce/consume lowering for the v0
   subset
-status: To Do
+status: In Progress
 assignee: []
 created_date: '2026-07-25 02:31'
+updated_date: '2026-07-26 05:14'
 labels: []
 milestone: m-7
 dependencies:
@@ -25,9 +26,38 @@ The load-bearing milestone (design doc §5): sqlparser(DuckDB dialect) frontend
 
 ## Acceptance Criteria
 <!-- AC:BEGIN -->
-- [ ] #1 v0-subset queries prepare end-to-end: sql + static tables -> verified imperative IR running on the interpreter backend
-- [ ] #2 All-static subtrees are evaluated at prepare time: a static-tables-only query lowers to a constant emitter with no probe/filter ops in its IR
-- [ ] #3 Differential suite vs DuckDB green on hand-written v0 cases; engine-vs-oracle disagreement follows the xfail-strict + ticket protocol
-- [ ] #4 Corpus replay reports match / clean-unsupported / FAIL counts; zero FAILs; every unsupported rejection is a clean build-time error naming the construct
-- [ ] #5 mise gate-specializer green (corpus replay wired into it)
+- [x] #1 v0-subset queries prepare end-to-end: sql + static tables -> verified imperative IR running on the interpreter backend
+- [x] #2 All-static subtrees are evaluated at prepare time: a static-tables-only query lowers to a constant emitter with no probe/filter ops in its IR
+- [x] #3 Differential suite vs DuckDB green on hand-written v0 cases; engine-vs-oracle disagreement follows the xfail-strict + ticket protocol
+- [x] #4 Corpus replay reports match / clean-unsupported / FAIL counts; zero FAILs; every unsupported rejection is a clean build-time error naming the construct
+- [x] #5 mise gate-specializer green (corpus replay wired into it)
 <!-- AC:END -->
+
+## Implementation Plan
+
+<!-- SECTION:PLAN:BEGIN -->
+Stretch plan (~5-6 stretches estimated, recorded 2026-07-26):
+1. Frontend spine: sqlparser (DuckDB dialect) -> relational IR (scan/project/filter over __THIS__), reusing shape lessons from src/datafusion; end-to-end sql -> imperative IR -> interpreter for arithmetic projections.
+2. 3VL lowering: SQL NULL semantics compiled to flag algebra (comparisons, AND/OR Kleene logic, CASE, IS NULL); type + nullability derivation; the correctness core.
+3. BTA + statics: taint __THIS__, equi-joins to static tables lower to probes, scalar folding; Python materialization through the DuckDBInferFn shell (pa.Table -> StaticData).
+4. IR builtin extensions (workflow fan-out per op): upper/lower/trim/substr/abs/round/concat + :: casts + COALESCE/NULLIF as lowerings; each new instruction lands across verifier/printer/parser/interp with pin tests.
+5.-6. Differential suite vs duckdb-python (pytest backend id "specialized") + corpus replay under the three-outcome contract; grind divergences to zero FAILs; xfail-strict + ticket for genuine oracle disagreements.
+<!-- SECTION:PLAN:END -->
+
+## Implementation Notes
+
+<!-- SECTION:NOTES:BEGIN -->
+Stretch 3 landed (commit 49362ee on claude/specializer-m-lower, PR #26): INNER/LEFT equi-joins to static tables lower to prologue probes (ordered JoinSpec list, no join tree nodes in v0); valid_hit = map hit AND every nullable key flag; f64 probe keys canonicalize -0.0/NaN at compile AND probe (DuckDB DOUBLE `=` pins: NaN=NaN, 0.0=-0.0); join key promotion both directions (i64 dyn promoted via IntToFloat; f64-vs-i64-col converts the build side at materialization per StaticSpec). fold.rs: conservative const folder, both-sides-const only, never drops a potentially-trapping dynamic operand (`FALSE AND a%0` keeps the rem). DuckDBInferFn replaces the stub: one row table (pydantic, ordered fields), pyarrow statics -> StaticData::Map per StaticSpec (NULL-key rows dropped, NULL value col errors, dup keys rejected at compile), output model synthesized via create_model. Differential pytest vs duckdb-python: 13 cases (joins, promotion, error contracts). Deliberate ceilings: per-block probe re-emission (pure, cached per block); all non-key cols become map values (prune later); key cols referenced outside their ON clause -> clean unsupported; supplied output_model trusted as-is.
+
+AC #2 (static-only queries fold to a constant emitter) is NOT yet satisfied: needs DuckDB evaluation at prepare time (Python side) and a constant-emitter program shape (emits N fixed rows regardless of input) — parked for the stretch 5-6 grind. Stretch 4 started: builtin catalogue; `::` casts already landed in stretch 2 (CastKind::DoubleColon). Fan-out adaptation: per-op implement-in-worktree would conflict on the same 7 coupled files (ir/mod, verify, print, parse, interp, frontend, fold), so the workflow fans out DuckDB semantics PINNING (8 parallel agents, duckdb-python 1.5.5) and post-integration adversarial verify; implementation is integrated centrally (one integrator, many measurers).
+
+Stretch 4 landed (commit ac3786a): builtin catalogue implemented strictly from measured pins (8-agent workflow fan-out measured DuckDB 1.5.5; spec at docs/superpowers/specs/2026-07-26-stretch4-builtin-pins.md). New IR insts across all six files + fuzz gen: supper/slower, strim.{both,lead,trail}, ssubstr (virtual-window codepoint arithmetic), iabs/fabs/fround, BinOp::Frem. Frontend catalogue: upper/lower/ltrim/rtrim/abs/round(1-arg)/concat/coalesce/nullif + TRIM/SUBSTRING dedicated AST forms; || is ALWAYS concat (even 1 || 2) with implicit VARCHAR casts; CONCAT skips NULLs (all-NULL -> ''); COALESCE per-row lazy via CASE desugار; NULLIF compares at promoted type, keeps first arg's type. TWO DIVERGENCES IN PREVIOUSLY-LANDED CODE were found by pinning and FIXED in-branch with pin tests (PM: flagging per the disagreement protocol — these were fixes toward the oracle, not semantics patches to pass tests): (1) integer % by zero returns NULL in DuckDB, we trapped -> lowering now CASE-guards the divisor, MIN % -1 still traps like DuckDB; (2) DOUBLE comparisons: DuckDB order is NaN=NaN / NaN above everything / zeros equal, we had IEEE partial -> shared exec::duck_fcmp used by interp AND fold. Known deliberate divergence (strict-xfail + needs a ticket): Rust std lacks Unicode SIMPLE case maps, so upper('ß') gives 'ß' vs DuckDB 'ẞ' and lower('İ') differs; ASCII exact. Deliberate ceilings: round(x, digits) unsupported; decimal literals stay f64 (stretch-1 ceiling). 114 cargo tests + 21 differential pytest + 1 strict xfail; gate green. Adversarial verify fan-out (6 probe agents vs duckdb) running; findings will be triaged fix-vs-xfail before stretch 5 (differential suite backend id + corpus replay in gate).
+
+Stretch 6 landed (commit 0782bf1): AC #2 satisfied — static-only queries become constant emitters, evaluated once at build time by DuckDB itself (Python boundary; no IR is built, so trivially no probe/filter ops). The fallback fires on Unsupported/Parse prepare errors and self-validates: dynamic queries reference the row table, unknown to DuckDB, so evaluation fails and the original clean error surfaces. Bonus: aggregation/ORDER BY/dialect-beyond-sqlparser all work on the static-only path. Final corpus: 53 match / 625 clean-unsupported / 0 FAIL of 678, wired into the gate via pytest. MILESTONE COMPLETE pending review — hard stop before M-cranelift (TASK-44). Two items for PM: (1) deviation note — AC #3's 'backend id specialized' wording: the DataFusion-oracle differential harness (test_diff_*) can't host the specializer (different oracle, e.g. `/` semantics differ by design), so the specializer has its own duck_check differential suite (32 cases) instead; (2) ticket request per the disagreement protocol — Unicode SIMPLE case mapping (upper('ß')/'İ'/'ᾀ' class): Rust std only exposes full maps; strict-xfail in place; candidate fixes are a small static table of the ~100 divergent codepoints or a unicode-data crate dependency.
+<!-- SECTION:NOTES:END -->
+
+## Final Summary
+
+<!-- SECTION:FINAL_SUMMARY:BEGIN -->
+M-lower delivered on branch claude/specializer-m-lower (PR #26), six stretches: (1) frontend spine sql->IR->interpreter; (2) 3VL flag algebra + CASE/CAST/IS NULL; (3) BTA + statics — equi-joins lower to map probes with canonical f64 key bits, DuckDBInferFn Python boundary, 13-case differential suite; (4) builtin catalogue measured by an 8-agent pin fan-out (upper/lower/trim/substr/abs/round/concat/coalesce/nullif, ||, float %, DuckDB DOUBLE comparison order) — 7 new IR instructions across verify/print/parse/gen/interp; (5) adversarial 6-agent probe fleet (~1,400 probes) found and fixed: NULL-divisor % trap, the trap-under-false-flag payload class bug, Zs trim set, vectorized-path substr semantics (backwards negative lengths, ±2^32 guards), DuckDB float->VARCHAR rendering; plus corpus replay wired into the gate under the three-outcome contract; (6) static-only constant emitters (AC #2). Final: gate green (cargo + 603 pytest, 13 xfail), corpus 53 match / 625 clean-unsupported / 0 FAIL of 678. Known strict-xfail: Unicode simple case maps (ticket requested). DuckDB dual-path substr inconsistency documented; oracle arrow-pushdown artifact harness-fixed.
+<!-- SECTION:FINAL_SUMMARY:END -->

docs/superpowers/specs/2026-07-26-stretch4-builtin-pins.md

@@ -0,0 +1,146 @@
+# Stretch 4: builtin catalogue — measured DuckDB 1.5.5 pins + implementation spec
+
+All values below were MEASURED against duckdb-python 1.5.5 (workflow fan-out of 8
+pin agents + local micro-pins, 2026-07-26). DuckDB is the oracle; when in doubt,
+re-measure, never assume.
+
+## Divergences found in already-landed code (fix in this stretch, with pin tests)
+
+1. **`%` by zero**: `5 % 0` → NULL (NOT an error). `MIN % -1` → traps
+   ("Out of Range Error: Overflow in division"). Our irem traps on both; keep the
+   IR inst trapping (MIN % -1 stays a trap ✓) and guard the SQL lowering:
+   `a % b` → `CASE WHEN b = 0 THEN NULL ELSE a irem b END` (skip the guard when b
+   is a non-zero literal). Result becomes nullable when guarded.
+2. **Float `%`**: currently rejected at bind. DuckDB: `5.0 % 2.0` = 1.0,
+   `-5.5 % 2.5` = -0.5 (sign of dividend), `5.0 % 0.0` = NaN — exactly Rust's
+   `%` on f64. Add `BinOp::Frem` (never traps), un-reject at bind. No guard.
+3. **F64 comparison order** (interp.rs Cmp F64 + fold.rs cmp mirror are IEEE —
+   wrong): DuckDB order = IEEE except NaN: `nan = nan` TRUE, `nan > 1` TRUE,
+   `nan > inf` TRUE, `nan <= nan` TRUE, `nan <> nan` FALSE, `1 <= nan` TRUE;
+   `-0.0 = 0.0` TRUE, `-0.0 < 0.0` FALSE. Implementation:
+   both NaN → Equal; one NaN → NaN is Greater; else IEEE partial_cmp.
+
+## New IR instructions (each lands in ir/mod + verify + print + parse + gen + interp)
+
+- `supper` / `slower` (Str→Str): DuckDB uses SIMPLE (1:1) case mapping:
+  `upper('ß')` = 'ẞ' U+1E9E (NOT 'SS'), `lower('İ' U+0130)` = plain 'i' (dot
+  dropped), emoji pass through. Rust std is FULL mapping → v0: per char, use
+  to_uppercase()/to_lowercase() iff it yields exactly 1 char, else keep the char
+  unchanged. KNOWN divergence on ß (we keep 'ß', DuckDB 'ẞ') and İ→lower (we keep
+  'İ', DuckDB 'i'): xfail-strict differential case + ticket note. ASCII exact.
+- `strim.both|lead|trail s, chars` (Str,Str→Str): removes chars in the SET
+  `chars` from the side(s). 1-arg SQL trim = chars " " (ONLY space 0x20 — tab/
+  newline NOT trimmed). Empty set = no-op. NULL either arg → NULL (flag algebra
+  at lowering, inst itself total).
+- `ssubstr s, start, len` (Str,I64,I64→Str): codepoint-based (NOT grapheme —
+  slices inside ZWJ emoji). Algorithm (1-based virtual window):
+  if len < 0 → ""; if start < 0 → start = char_len + start + 1;
+  window [start, start.saturating_add(len)); intersect with [1, char_len];
+  missing SQL len → Lit(i64::MAX) (saturating add makes it "rest of string").
+  Pins: substr('hello',0)='hello', (0,3)='he', (-2)='lo', (-6,3)='he',
+  (-10,8)='hel', (1,0)='', (1,-1)='', (10)=''.
+- `iabs` (I64→I64): traps on i64::MIN ("Out of Range"-style). `fabs` (F64→F64):
+  Rust f64::abs — clears sign bit, abs(-0.0)=+0.0, abs(nan)=nan, abs(-inf)=inf.
+- `fround` (F64→F64): Rust f64::round = half AWAY from zero ✓ (2.5→3, -2.5→-3),
+  nan→nan, inf→inf, 1e300→1e300, -0.4→-0.0 (sign KEPT on double). round(int) is
+  identity at the FRONTEND (no inst); round(x, digits) → clean unsupported
+  (scale-then-round algorithm deferred).
+- `frem` BinOp (F64,F64→F64): Rust `%`. Never traps.
+
+## Frontend dispatch (new SKind variants + lowerings)
+
+- `Expr::Trim` (sqlparser: trim_where BOTH/LEADING/TRAILING, trim_what,
+  trim_characters) + `ltrim`/`rtrim`/2-arg functions → SKind::Trim{side}.
+  All forms NULL-propagating on both args.
+- `Expr::Substring` (SUBSTR and SUBSTRING both route here) → SKind::Substr.
+  NULL-propagating on all three args.
+- `Expr::Function` name dispatch (case-insensitive):
+  - upper/lower → SKind::StrCase{upper} — arg must be Str (DuckDB: upper(123)
+    is a Binder Error, NO numeric coercion).
+  - abs → SKind::Abs; I64 or F64 only (abs('5')/abs(true) = binder errors).
+    Result type = arg type.
+  - round (1-arg) → identity for I64; SKind::Round for F64. 2-arg → unsupported.
+  - concat(a,...) → NULL-SKIPPING: each nullable arg wraps in
+    CASE WHEN x IS NULL THEN '' ELSE cast_to_varchar(x) END, chain SKind::Concat.
+    Result never NULL. concat() zero args → Bind error. Non-string args render
+    via VARCHAR cast (1→'1', 1.5→'1.5', true→'true').
+  - coalesce(a,...) → nested CASE WHEN a IS NOT NULL THEN a ELSE rest END.
+    Lazy per-row (pinned: untaken erroring arm does NOT fire). Args unify under
+    existing promotion (I64+F64→F64); else Bind error. coalesce() → parser
+    error in DuckDB; here Bind error fine.
+  - nullif(a,b) → CASE WHEN a = b THEN NULL ELSE a END; comparison at PROMOTED
+    type, result type = FIRST arg's type (never unified!). nullif(nan,nan) →
+    NULL (needs divergence fix #3).
+- `BinaryOperator::StringConcat` (`||`): ALWAYS string concat in DuckDB — even
+  `1 || 2` = '12', `true || true` = 'truetrue'. NULL-PROPAGATING (unlike
+  CONCAT). Cast non-Str operands to VARCHAR: bool → 'true'/'false' (measured).
+- CONCAT vs ||: CONCAT skips NULLs (all-NULL → ''), || propagates.
+
+## fold.rs
+
+New kinds: fold children only (like Case). Update the f64 cmp mirror to the
+DuckDB order (divergence #3) — fold and interp MUST stay bit-identical.
+
+## Out of scope (clean unsupported), deliberate
+
+round(x, digits); DECIMAL anything (literals stay F64 — known v0 ceiling);
+upper/lower non-simple-map codepoints byte-exactness (xfail + ticket).
+
+## Adversarial-fleet addendum (6 probe agents, ~1,400 probes, 2026-07-26)
+
+Divergences FOUND and FIXED (each now pinned in Rust + differentially):
+
+1. **NULL divisor `%`**: the `b = 0` CASE guard alone is NULL for b NULL —
+   fell through to irem on the garbage zero payload. `b IS NULL OR b = 0`
+   shields it (TRUE OR NULL = TRUE).
+2. **Trap-under-false-flag class bug** (predates stretch 4): computed
+   garbage payloads are unbounded (`(x + MAX) + MAX` with x NULL overflows
+   its payload lane). FIX: `FB::masked` forces nullable payloads to the type
+   default before every trapping instruction (integer arith, iabs, ssubstr
+   positions, ftoi cast input).
+3. **1-arg trim set**: exactly the Unicode Zs space separators (per-codepoint
+   census) — NBSP/ideographic space etc. trim; tab/newline/ZWSP/BOM do not.
+4. **substr**: DuckDB's constant-fold path and vectorized path DISAGREE on
+   negative starts. We implement the VECTORIZED path (what columns, real
+   queries, and the mined corpus use): negative start clamps to 1 after
+   end-resolution (`rs = max(n+start+1, 1)`), start 0 stays virtual, a
+   NEGATIVE length slices BACKWARDS `[rs+len, rs)`, offsets/lengths outside
+   ±2^32 trap ("Out of Range"), the 2-arg form never length-traps (why
+   `len` is `Option`, not a sentinel). Known residual: pure-literal
+   negative-start substr goes through DuckDB's constant path and can differ.
+5. **Float -> VARCHAR**: DuckDB writes an explicit exponent sign with at
+   least two digits (`1e+300`, `1e-05`) and lowercase `nan` (DuckF64 in
+   interp.rs).
+6. **Oracle artifact, harness-fixed, no engine change**: duckdb-python
+   pushes constant filters into REGISTERED-ARROW scans with IEEE NaN
+   semantics, disagreeing with its own native-table order (and violating
+   3VL for `x <= NaN`). duck_check now materializes native tables.
+7. **Simple-case-map divergence extended**: `upper('ᾀ')` (ypogegrammeni
+   titlecase U+1F88) joins ß/İ in the strict xfail.
+
+Also landed with this pass (corpus-driven): implicit numeric->BOOLEAN in
+conditional contexts (WHERE/AND/OR/NOT/CASE WHEN; nonzero -> true incl.
+NaN, NULL -> NULL); `rowid` and DuckDB lateral aliases reject as clean
+unsupported; corpus replay wired into pytest with the three-outcome
+contract at 49 match / 629 clean-unsupported / 0 FAIL of 678.
+
+## Case-mapping divergence: RESOLVED dependency-free (2026-07-26, post-review)
+
+Decision (AmirHossein): no dependency for the ~100 codepoints; do it measured
+and documented. `src/specializer/exec/casemap.rs` (GENERATED by
+`scripts/gen_casemap.py`) carries the exception table over Rust's full maps —
+139 entries (83 upper, 56 lower) from TWO measured classes:
+  1. simple-vs-full mapping divergence (ß/İ/ypogegrammeni block);
+  2. Unicode VERSION skew, both directions: codepoints utf8proc maps that
+     Rust doesn't matter (fallback handles), and Unicode-16 case pairs RUST
+     maps but duckdb 1.5.5's utf8proc predates (identity there — 56 entries
+     Python couldn't even see; the generator's phase 2 measures the actual
+     compiled engine to catch them).
+A unicode-data crate would be the wrong spec twice over: some other Unicode
+version's tables, and blind to utf8proc's vintage. The full-codepoint census
+in tests/test_duckdb_interpreter.py (every scalar value through both engines,
+chunked into long strings) is the standing authority — a duckdb bump that
+shifts utf8proc fails the census, and the generator is idempotent to rerun.
+The former strict xfail is now two passing tests. DataFusion note: per the
+same review, DataFusion is the legacy serving line's oracle only — likely
+retired later, possibly a supported dialect; the specializer ignores it.