Specializer M-lower: frontend + BTA + lowering for the v0 subset - #26
Merged
Conversation
…-lower stretch 1) sql text -> sqlparser(DuckDB dialect) -> bound typed relational IR (plan.rs) -> produce/consume lowering (lower.rs) -> verified imperative IR -> interpreter oracle. v0 ribbon: projection + WHERE over __THIS__; columns (case-insensitive bind, spelling-preserving names), int/float/str/bool literals, + - * / % with DuckDB promotion (measured pins: / is ALWAYS float division, % stays integral, overflow errors), comparisons with int->float promotion, unary minus, NULL propagation as flag algebra (non-nullable expressions carry no flag at all). WHERE keeps a row iff pred IS TRUE — flag && value, one and. Error discipline for the corpus contract: Unsupported names the construct (JOIN, CASE, functions, star, AND/OR, ...) and is disjoint from Bind (unknown column, type mismatch). prepare() verifies every program it returns; per-block column cache keeps one load per column. exec test helpers extracted to exec/testutil.rs for reuse. cargo 76 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d NULL (M-lower stretch 2)
The correctness center of M-lower. lower.rs rebuilt around an env-threading
function builder: CASE and guarded CASTs split the CFG, and strict
block-param SSA means every live value rides branch args — emit() carries a
live stack that block transitions rebind in place.
- Kleene AND/OR lowered branchless from flag algebra (validly-deciding-side
rules); NOT/IS [NOT] NULL; NULL literal typed by context, never folded
through AND/OR (NULL AND FALSE is FALSE)
- CASE: condition chain with join blocks; branch results evaluate ONLY on
their taken path (untaken trapping branches must not trap — tested);
simple form desugars to operand comparisons; int/float branch unification
- CAST/TRY_CAST full v0 matrix; string parses trap on failure via a guarded
trap block (NULL input never traps), TRY folds failure into the null lane;
TRY_CAST(f64->i64) range-guards ftoi through the live stack
- stoi/stof pinned to DuckDB CAST trim semantics (' 5'::BIGINT = 5) — the
M-interp docs deferred exactly this decision to M-lower
- ir::canonicalize(): bijective renumber into text order, run by prepare();
parse(print(p)) == p holds exactly for every prepared program
- float % rejected cleanly (no frem instruction yet)
Every 3VL expectation cross-checked against duckdb 1.5.5 directly. cargo 86.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ng, DuckDBInferFn boundary (M-lower stretch 3) Frontend binds INNER/LEFT equi-joins to static tables (ordered JoinSpec list, no join tree nodes in v0); lowering emits prologue probes with Kleene-correct valid_hit flags; f64 probe keys canonicalize -0.0 and NaN to match DuckDB's DOUBLE equality. fold.rs is a conservative const folder: both-sides-const only, never drops a potentially trapping dynamic operand. DuckDBInferFn replaces the stub: pydantic row model in, pyarrow statics materialized per StaticSpec (NULL keys dropped, NULL values rejected), synthesized output model out. Differential pytest vs duckdb-python covers joins, promotion, and the error contracts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
claude-agent-ahrzb
Bot
force-pushed
the
claude/specializer-m-lower
branch
from
July 26, 2026 04:15
d3bac19 to
49362ee
Compare
… abs/round, COALESCE/NULLIF, || concat (M-lower stretch 4)
Eight parallel pin agents measured DuckDB 1.5.5 exactly (spec:
docs/superpowers/specs/2026-07-26-stretch4-builtin-pins.md); the
catalogue implements what was measured, never assumed.
New IR instructions across verify/print/parse/gen/interp: supper/slower
(simple case mapping), strim.{both,lead,trail} (char-set trim; 1-arg SQL
trim removes ONLY 0x20), ssubstr (DuckDB virtual-window arithmetic on
codepoints), iabs (traps on MIN)/fabs/fround (half away from zero), and
BinOp::Frem (IEEE float %).
Frontend: upper/lower/ltrim/rtrim/abs/round/concat/coalesce/nullif +
TRIM/SUBSTRING forms; || is ALWAYS string concat (1 || 2 = '12') with
implicit VARCHAR casts, NULL-propagating, while CONCAT() skips NULLs;
COALESCE is per-row lazy via CASE; NULLIF compares promoted but keeps
the first arg's type.
Two measured divergences in landed code FIXED with pins: integer % by
zero is NULL in DuckDB (not an error) — lowering now guards with a CASE,
MIN % -1 still traps like DuckDB; and DOUBLE comparisons use DuckDB's
order (NaN = NaN, NaN above everything, zeros equal) — shared duck_fcmp
in interp and fold. Known deliberate divergence, xfail'd strictly:
Rust std has no simple case maps, so upper('ß')/lower('İ') differ.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…vergence fixes (M-lower stretch 5)
Corpus replay (tests/test_corpus_replay.py) wires the three-outcome
contract into the gate: 678 mined cases -> 49 match / 629
clean-unsupported / 0 FAIL. Corpus-driven frontend growth: implicit
numeric->BOOLEAN in conditional contexts (WHERE/AND/OR/NOT/CASE WHEN,
nonzero true incl. NaN); rowid and DuckDB lateral aliases reject as
clean unsupported.
A 6-agent adversarial probe fleet (~1,400 differential probes) found and
this commit fixes, all pinned: NULL-divisor % trapped on its garbage
payload (IS NULL now shields the guard); a trap-under-false-flag class
bug predating stretch 4 — computed garbage payloads are unbounded, so
FB::masked forces nullable payloads to the type default before every
trapping instruction; the 1-arg trim set is exactly Unicode Zs (census);
substr follows DuckDB's VECTORIZED path (negative start clamps to 1,
negative length slices backwards, ±2^32 range traps, 2-arg form never
length-traps — len is Option, not a sentinel; DuckDB's own constant-fold
path disagrees with its vectorized path here, documented); float ->
VARCHAR renders DuckDB-style ('1e+300', '1e-05', lowercase nan).
Oracle artifact found, harness-fixed: duckdb-python pushes constant
filters into registered-arrow scans with IEEE NaN semantics that
contradict its native-table order — duck_check now materializes native
tables. upper('ᾀ') joins the simple-case-map strict xfail.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… (M-lower stretch 6) A query whose driving relation is a static table is evaluated ONCE at build time by DuckDB itself; infer() returns the fixed rows (re-validated through the output model per call) and nothing dynamic remains — no IR, no probes, no filters. The fallback self-validates: it fires on Unsupported/Parse errors and a dynamic query always references the row table, which DuckDB does not know, so evaluation fails and the original clean error surfaces unchanged. Bind errors stay hard. Bonus breadth: aggregation, ORDER BY, and DuckDB-dialect-beyond-sqlparser all work on the static-only path because DuckDB is the evaluator. Corpus: 53 match / 625 clean-unsupported / 0 FAIL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y-free — census-verified exception table Resolves the last strict xfail. casemap.rs is GENERATED by scripts/gen_casemap.py measuring duckdb-python directly: 139 exceptions (83 upper, 56 lower) over Rust's full-map fallback, from two classes — simple-vs-full divergence (ß/İ/ypogegrammeni) and three-way Unicode version skew (Unicode-16 case pairs Rust maps but duckdb's utf8proc predates; Python's tables can't even see 56 of them, so the generator's phase 2 censuses the actual compiled engine). No dependency: a unicode-data crate would be some other Unicode version's tables AND blind to utf8proc's vintage — the wrong spec twice over. The authority is the new full-codepoint census test: every Unicode scalar value through both engines, chunked into long strings. A duckdb bump that shifts utf8proc fails it loudly; the generator is idempotent (carry-forward + engine-residue phases) to regenerate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ahrzb
approved these changes
Jul 26, 2026
claude-agent-ahrzb Bot
pushed a commit
that referenced
this pull request
Jul 26, 2026
…wired, in-tree JIT smoke green (TASK-44) TASK-43 marked Done (PR #26 merged); TASK-44 stretch plan recorded in backlog. cranelift-{jit,module,codegen,frontend} 0.126 added and a smoke test JITs and executes a function on x86_64-pc-windows-msvc — the spike result is now validated in-tree, not just in a note. The real per-row backend (extern "C" fn(ctx) -> i64 ABI, coverage-first with shared helpers, interpreter fallback) lands next per the stretch plan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
{"type":"message","message":"Detected an AI agent environment, printing as NDJSON. Trace logs are written to stderr, while user-facing logs are written to stdout."}
What (TASK-43, milestone m-7 — COMPLETE, all 5 acceptance criteria)
The M-lower milestone, six stretches:
prepare(sql, this_name, in_cols, statics)-> parse (sqlparser DuckDB dialect) -> bind + type derivation -> relational IR -> produce/consume lowering -> verified imperative IR -> the M-interp oracle. Every semantic decision was measured against duckdb-python 1.5.5, never assumed (pins spec:docs/superpowers/specs/2026-07-26-stretch4-builtin-pins.md).Stretch 1 — spine: projection + WHERE over
__THIS__, measured DuckDB promotion (/is float division,%integral, overflow traps), flag-algebra NULLs, case-insensitive spelling-preserving binds. Everything else rejects as cleanUnsupportednaming the construct.Stretch 2 — 3VL core: Kleene AND/OR, CASE (branches run only on their taken path), CAST/TRY_CAST/
::, IS [NOT] NULL, typed NULL literals.Stretch 3 — BTA + statics: INNER/LEFT equi-joins to static tables lower to prologue map probes;
valid_hit= hit AND every nullable key flag; f64 probe keys canonicalize-0.0/NaN; key promotion both directions.fold.rsconservative const folder.DuckDBInferFn: pydantic row model in, pyarrow statics materialized perStaticSpec, synthesized output model out.Stretch 4 — builtin catalogue from measured pins (8-agent workflow fan-out measured; central integration): upper/lower/trim/ltrim/rtrim/substr/abs/round/concat/coalesce/nullif,
||(always concat, even1 || 2), float%. Seven new IR instructions across verify/print/parse/gen/interp. Two divergences in earlier stretches found by pinning and fixed: integer%by zero is NULL (not an error), and DOUBLE comparisons use DuckDB's order (NaN = NaN, NaN above everything, zeros equal — sharedduck_fcmpin interp and fold).Stretch 5 — adversarial fleet + corpus: a 6-agent probe fleet (~1,400 differential probes) found and this branch fixes: NULL-divisor
%trapping on its garbage payload; a trap-under-false-flag class bug (computed garbage payloads are unbounded —FB::maskednow forces nullable payloads to type defaults before every trapping instruction); the 1-arg trim set is exactly Unicode Zs; substr follows DuckDB's vectorized path (negative lengths slice backwards, ±2^32 range traps; DuckDB's own constant-fold path disagrees with its vectorized path — documented); DuckDB-style float→VARCHAR (1e+300,1e-05,nan). Corpus-driven: implicit numeric→BOOLEAN in conditional contexts;rowid/lateral aliases reject cleanly. One oracle artifact (duckdb-python arrow-scan filter pushdown contradicts its native-table NaN order, even violating 3VL) — harness now materializes native tables.Stretch 6 — constant emitters (AC #2): a static-tables-only query is evaluated once at build time by DuckDB itself; nothing dynamic remains. Self-validating fallback — dynamic queries reference the row table, unknown to DuckDB, so the original clean error surfaces.
Verification
mise gate-specializergreen: cargo (incl. IR round-trip fuzz over the new instructions) + 605 pytest, 12 xfailcasemap.rsis a generated, measured exception table (139 codepoints — simple-vs-full divergence + three-way Unicode version skew across Python/Rust/utf8proc); a full-codepoint census test (every scalar value through both engines) is the standing authority, and the former strict xfail is now two passing testsFor review
test_diff_*harness can't host the specializer (different oracle by design — e.g./semantics), so it has its own duck_check suite instead of aspecializedbackend id.🤖 Generated with Claude Code