SQL specializer groundwork: engine-per-module refactor, design docs, pre-loop prep

.gitignore

@@ -21,6 +21,9 @@ wheels/
 # Rust build artifacts
 /target
 
+# DuckDB source clone — reference corpus for the SQL specializer, not a dependency
+/duckdb/
+
 # Compiled PyO3 extension module (built in-place by `maturin develop`)
 sql_transform/_interpreter*.so
 sql_transform/_interpreter*.pyd

Cargo.toml

@@ -7,6 +7,9 @@ edition = "2021"
 name = "_interpreter"
 crate-type = ["cdylib"]
 
+# extension-module is deliberately NOT a default feature: it suppresses the
+# libpython link, which is right for the wheel but breaks `cargo test`.
+# maturin enables it via [tool.maturin] features in pyproject.toml.
 [dependencies]
-pyo3 = { version = "0.29", features = ["extension-module", "abi3-py314"] }
+pyo3 = { version = "0.29", features = ["abi3-py314"] }
 sqlparser = "0.62"

backlog/milestones/m-7 - sql-specializer.md

@@ -0,0 +1,8 @@
+---
+id: m-7
+title: "sql-specializer"
+---
+
+## Description
+
+SQL specializer for ML model serving: prepare-once/run-millions engine built on binding-time separation. Design: docs/superpowers/specs/2026-07-25-sql-specializer-design.md; execution process: docs/superpowers/specs/2026-07-25-sql-specializer-loop-execution-design.md.

backlog/tasks/task-41 - Specializer-M-ir-imperative-IR-—-grammar-types-verifier-text-format.md

@@ -0,0 +1,45 @@
+---
+id: TASK-41
+title: 'Specializer M-ir: imperative IR — grammar, types, verifier, text format'
+status: In Progress
+assignee: []
+created_date: '2026-07-25 02:31'
+updated_date: '2026-07-25 13:46'
+labels: []
+milestone: m-7
+dependencies: []
+documentation:
+  - docs/superpowers/specs/2026-07-25-sql-specializer-design.md
+type: feature
+ordinal: 35000
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+Define the specializer's imperative IR per §6 of docs/superpowers/specs/2026-07-25-sql-specializer-design.md: SSA over typed scalars with a separate null lane (T? vs T are distinct types; ops on T? only via the .opt instructions), StaticRef handles, no allocation vocabulary. Deliverables are the IR definitions, the verifier, and a round-trippable text format under src/specializer/ir/. This is the diagnostic surface for the whole engine and the loop's machine-checkable gate substrate — the boundary must be airtight before anything targets it.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [x] #1 Verifier rejects: non-SSA defs, type mismatches, any arithmetic on a T? value not routed through the null-lane ops, unresolvable StaticRef ids, allocating constructs
+- [x] #2 Text format round-trips: parse(print(ir)) == ir on every test program, including a property/fuzz round-trip test
+- [x] #3 Hand-written IR programs covering every instruction exist as test fixtures
+- [x] #4 mise gate-specializer green
+<!-- AC:END -->
+
+## Implementation Plan
+
+<!-- SECTION:PLAN:BEGIN -->
+1. Freeze IR design decisions (recorded in design doc §6 update): implicit row cursors (no idx params), strict block-param SSA (cross-block value uses forbidden — no dominance analysis needed), acyclic CFG for v0, terminators emit/skip/trap/jump/brif, store-completeness dataflow at emit.
+2. Implement src/specializer/ir/: core types + instructions (mod.rs), verifier (verify.rs), printer (print.rs), parser (parse.rs), shared fixtures (fixtures.rs).
+3. Tests: per-instruction fixture programs (parse+verify+round-trip), negative verifier tests per rule, negative parser tests, deterministic seeded fuzz round-trip (hand-rolled xorshift generator, no new deps).
+4. Adversarial workflow: fan-out attackers on verifier soundness + round-trip edge cases (float/string literals), plus design-conformance and simplification reviewers; fix confirmed findings.
+5. Gate green; stacked PR onto claude/duckdb-native-interpreter-36d016.
+<!-- SECTION:PLAN:END -->
+
+## Implementation Notes
+
+<!-- SECTION:NOTES:BEGIN -->
+Implemented in src/specializer/ir/ (mod, verify, print, parse, fixtures, gen, tests). Key decisions vs the original sketch (design doc §6 updated): implicit row cursor, emit/skip/trap terminators carrying the store contract, strict block-param SSA (no dominance analysis), acyclic v0 CFG, presentation-only names (canonical %vN/bN). Adversarial pass (6 agents) found and I fixed: recursive-DFS process abort on deep CFGs (now iterative, 50k blocks OK), empty map static signatures and non-ident fn names verifying but printing unparseably, non-canonical NaN payloads breaking bitwise round-trip equality (NaN-class equality now), unreachable-island starvation masking join store errors (topo indegrees over reachable sources only), doc/grammar drift, dead uses() code. 45 cargo tests; every verifier rule and parser guard has a rejecting test; 300-seed fuzz round-trip.
+<!-- SECTION:NOTES:END -->

backlog/tasks/task-42 - Specializer-M-interp-closure-compiled-IR-interpreter-the-oracle-backend.md

@@ -0,0 +1,46 @@
+---
+id: TASK-42
+title: 'Specializer M-interp: closure-compiled IR interpreter (the oracle backend)'
+status: In Progress
+assignee: []
+created_date: '2026-07-25 02:31'
+updated_date: '2026-07-25 19:18'
+labels: []
+milestone: m-7
+dependencies:
+  - TASK-41
+documentation:
+  - docs/superpowers/specs/2026-07-25-sql-specializer-design.md
+type: feature
+ordinal: 36000
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+Implement the interpreter backend over the imperative IR (design doc §7): one pre-traversal builds a closure tree, then execution is plain dispatch. This backend is the differential-testing oracle for every future codegen backend and the fallback for uncovered ops — correctness and coverage over speed, never optimized. Depends on M-ir for the IR definitions and verifier.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [x] #1 Every IR instruction executes; the M-ir fixture programs all produce expected outputs
+- [x] #2 Runs only verifier-accepted IR; rejects unverified programs
+- [x] #3 No allocation during execution (arena-only for varlen), asserted by a test
+- [x] #4 mise gate-specializer green
+<!-- AC:END -->
+
+## Implementation Plan
+
+<!-- SECTION:PLAN:BEGIN -->
+1. Runtime substrate in src/specializer/exec/: Batch (typed columns + validity), Arena (bump String, spans as (u32,u32) StrRef), StaticData (Scalar / Map keyed by bit-hashed key tuples) type-checked against Program.statics at compile.
+2. interp.rs: compile(program, statics) runs verify() first (unverified IR is rejected — AC#2), then one pre-traversal builds per-block closure lists + terminator thunks; Frame of Copy registers indexed by Value id, reused across rows; stores push straight into pre-capacitied out builders (verifier's exactly-once contract keeps columns row-aligned).
+3. Semantics pins (documented in interp.rs, oracle-differential at M-lower): checked integer arithmetic traps on overflow; idiv/irem trap on 0 and MIN/-1; fdiv IEEE; fcmp IEEE-ordered (NaN compares false); ftoi.trunc toward zero, ftoi.round half-away-from-zero (DuckDB CAST), both trap out of i64 range; stoi/stof exact parse.
+4. Tests: all 5 M-ir fixtures executed against hand-computed inputs/statics/outputs; unverified-program rejection; counting global allocator asserts zero heap allocs on a warmed second run (arena/regs/builders reused); executor fuzz over gen::gen_program seeds with random inputs (no panics, |out| <= |in|, deterministic).
+5. Adversarial workflow (semantics vs spec, alloc discipline, trap/edge paths, fuzz), fix confirmed findings; gate green; stacked PR onto claude/duckdb-native-interpreter-36d016.
+<!-- SECTION:PLAN:END -->
+
+## Implementation Notes
+
+<!-- SECTION:NOTES:BEGIN -->
+Implemented in src/specializer/exec/ (mod.rs substrate + interp.rs + tests.rs). compile() verifies first; closures built in one pre-traversal; terminators interpreted as a small enum in the row loop; register slots densely remapped at compile. Semantics pins documented and every one tested. Adversarial pass (6 agents) found and I fixed: u32 arena-span wrap silently corrupting output past 4 GiB (offsets now usize), store.opt false-flag storing the live register instead of the pinned type default, unvalidated validity-lane length, sparse value ids inflating the register frame, and an overbroad zero-allocation claim (restated: warmth is per content profile, growth one-time+monotone). RunState.emitted added. cargo 65 tests; alloc-free steady state asserted via thread-local counting global allocator; 150-seed executor fuzz deterministic.
+<!-- SECTION:NOTES:END -->

backlog/tasks/task-43 - Specializer-M-lower-frontend-BTA-produce-consume-lowering-for-the-v0-subset.md

@@ -0,0 +1,33 @@
+---
+id: TASK-43
+title: >-
+  Specializer M-lower: frontend + BTA + produce/consume lowering for the v0
+  subset
+status: To Do
+assignee: []
+created_date: '2026-07-25 02:31'
+labels: []
+milestone: m-7
+dependencies:
+  - TASK-42
+documentation:
+  - docs/superpowers/specs/2026-07-25-sql-specializer-design.md
+  - docs/superpowers/specs/2026-07-25-sql-specializer-loop-execution-design.md
+type: feature
+ordinal: 37000
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+The load-bearing milestone (design doc §5): sqlparser(DuckDB dialect) frontend to relational IR; binding-time analysis that taints __THIS__ and evaluates every all-static subtree in DuckDB at prepare time, materializing Const structures (scalar / dense array / perfect hash / inline); produce/consume lowering of the dynamic frontier to imperative IR. v0 subset per design doc §4. Differential oracle is DuckDB (python pkg); the mined corpus at tests/corpus/duckdb_mined.jsonl replays under the three-outcome contract (match / clean-unsupported / FAIL) documented in scripts/mine_duckdb_corpus.py. Wide-mechanical: run per-operator fan-out via workflows per the loop-execution doc §2.3.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [ ] #1 v0-subset queries prepare end-to-end: sql + static tables -> verified imperative IR running on the interpreter backend
+- [ ] #2 All-static subtrees are evaluated at prepare time: a static-tables-only query lowers to a constant emitter with no probe/filter ops in its IR
+- [ ] #3 Differential suite vs DuckDB green on hand-written v0 cases; engine-vs-oracle disagreement follows the xfail-strict + ticket protocol
+- [ ] #4 Corpus replay reports match / clean-unsupported / FAIL counts; zero FAILs; every unsupported rejection is a clean build-time error naming the construct
+- [ ] #5 mise gate-specializer green (corpus replay wired into it)
+<!-- AC:END -->

backlog/tasks/task-44 - Specializer-M-cranelift-codegen-backend-behind-the-same-interface.md

@@ -0,0 +1,29 @@
+---
+id: TASK-44
+title: 'Specializer M-cranelift: codegen backend behind the same interface'
+status: To Do
+assignee: []
+created_date: '2026-07-25 02:31'
+labels: []
+milestone: m-7
+dependencies:
+  - TASK-43
+documentation:
+  - docs/superpowers/specs/2026-07-25-sql-specializer-design.md
+type: feature
+ordinal: 38000
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+Cranelift-jit backend for the imperative IR (design doc §7; cranelift-jit 0.126 spike-verified on x86_64-pc-windows-msvc 2026-07-25). StaticRef handles resolve to absolute addresses of prepare-time structures owned by the compiled artifact. Interpreter-vs-cranelift differential on random IR programs plus the full corpus; first ns/call numbers per the measurement discipline (design doc §10).
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [ ] #1 Every v0-subset prepared query compiles under cranelift and agrees with the interpreter backend on the corpus and on randomized inputs
+- [ ] #2 Uncovered ops fall back to the interpreter backend rather than failing prepare
+- [ ] #3 p50/p99 ns/call reported at n in {1, 8, 64, 1024} against the interpreter control and the existing native + codegen engines
+- [ ] #4 mise gate-specializer green
+<!-- AC:END -->

backlog/tasks/task-45 - Specializer-M-boundary-generated-row-marshaller-Python-API.md

@@ -0,0 +1,30 @@
+---
+id: TASK-45
+title: 'Specializer M-boundary: generated row marshaller + Python API'
+status: To Do
+assignee: []
+created_date: '2026-07-25 02:32'
+labels: []
+milestone: m-7
+dependencies:
+  - TASK-44
+documentation:
+  - docs/superpowers/specs/2026-07-25-sql-specializer-design.md
+type: feature
+ordinal: 39000
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+Wire the specializer into the Python surface: SpecializedTransform (SQLTransform API minus transformer refs) whose fit() runs prepare and whose infer/infer_batch cross the boundary through a prepare-time-generated marshaller — fixed field order, interned names, packed row structs, model_construct-style output fill (design doc §3 flag 1: input is row-major; no columnar transpose in the hot path). Baseline the boundary with a no-op f through both the generic pydantic path and the generated marshaller so the win is measured, then end-to-end p50/p99 vs the current engines.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [ ] #1 SpecializedTransform fit/infer/infer_batch works end-to-end on the v0 subset with dict and pydantic-model rows
+- [ ] #2 No-op-f boundary baseline reported: generic pydantic path vs generated marshaller, p50/p99 at n in {1, 8, 64, 1024}
+- [ ] #3 End-to-end p50/p99 vs the current native and codegen engines reported
+- [ ] #4 Steady-state hot path allocates nothing per call beyond the output objects; arena reset only
+- [ ] #5 mise gate-specializer green
+<!-- AC:END -->