SQL specializer groundwork: engine-per-module refactor, design docs, pre-loop prep - #23
Merged
Merged
Conversation
Move the existing engine under src/datafusion/ and hoist what is
engine-agnostic to the crate root: Value marshalling (value.rs), the
Base/FieldType/Schema vocabulary (types.rs), InterpError (error.rs).
schema.rs/lookup.rs were already engine-free. datafusion::{expr,plan,
types} re-export the moved names so engine-internal paths still resolve.
src/duckdb/ adds DuckDBInferFn: same Python API as InferFn minus the
transformer callout. Stub only — the constructor parses in the DuckDB
dialect, infer() raises NotImplementedError.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Design doc: partial-evaluation architecture (BTA collapses static subtrees at prepare; row-major AoS ABI with a prepare-time-generated marshaller; DuckDB as frontend/static-evaluator/oracle; interpreter oracle then Cranelift). Loop doc: prep checklist and the /loop-spine + workflow-fan-out execution design. duckdb/ source clone gitignored as a reference corpus. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- duckdb 1.5.5 as dev dep (differential oracle). Substrait spike: extension unavailable for 1.5.5/windows_amd64 (404 all repos) -> frontend flag flipped to the sqlparser fallback; json_serialize_sql noted as parser cross-check. - cranelift-jit 0.126 spike: works on x86_64-pc-windows-msvc. - cargo test made runnable: extension-module moved out of the crate's default pyo3 features (maturin enables it via pyproject); scripts/gate.py puts the uv CPython's python3.dll on PATH. mise task: gate-specializer. - pytest testpaths pinned so the duckdb/ clone's own scripts aren't collected. - scripts/mine_duckdb_corpus.py: 678 replayable cases mined from duckdb's sqllogictests into tests/corpus/duckdb_mined.jsonl (three-outcome contract). - backlog: milestone m-7 seeded, TASK-41..45 (M-ir..M-boundary) chained. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e rule hid it The repo-root DataFusion clone is ignored via .git/info/exclude with the bare pattern `datafusion/`, which matches at any depth and silently swallowed the refactor's src/datafusion/ move (local builds passed off the working tree; CI got a commit without the module). Exclude rule anchored locally to /datafusion/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
src/datafusion/ (Rust module) sits under a ruff src root, so isort's scan started classifying the datafusion pip package as first-party on CI's --all-files run (local pre-commit only checks changed files, so it never fired). duckdb pinned too — the source clone collides the same way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… (M-ir) src/specializer/ir/: SSA over bare typed scalars with the null lane split into explicit i1 flags (.opt instruction forms) so a nullable value cannot reach arithmetic by construction; strict block-param SSA (no dominance analysis needed); acyclic CFG; emit/skip/trap terminators carrying the store-completeness contract (|out|==|in| iff skip unreachable). verify.rs enforces six rule groups, each with a rejecting test; print/parse round-trip exactly (parse(print(p)) == p, bitwise literal equality so nan/-0.0 survive); gen.rs is a seeded xorshift program generator — 300-seed round-trip fuzz in tests, reused later for interpreter-vs-codegen fuzzing. Design doc §6 updated to the implemented form. TASK-41. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… — TASK-41 Six agents (4 attack lenses + 2 reviews) probed the initial IR; every confirmed finding is fixed and pinned by a regression test: - reachability DFS made iterative: a ~8k-block legal CFG stack-overflowed the process; 50k blocks now verify (deep_cfg_verifies_without_crashing) - verified programs must print to parseable text: reject empty map static signatures (grammar can't express `map() -> ()`) and non-identifier function names - Lit::F64 equality treats all NaNs as one class, matching the text form's canonical `nan` token — non-canonical payloads round-trip again - store dataflow runs over the reachable subgraph: an unreachable cyclic island no longer starves a reachable join and masks its store errors - docs aligned: double stores rejected on ALL paths incl. trap (deliberate), grammar EBNF covers store/comments/lenient %names, canonical-form note - dead Inst::uses/Term::uses deleted; 7 missing rule-coverage tests added (entry params, duplicate cols, branch-to-entry, sload-on-map, scalar-static .opt pairing both ways, store.opt on non-nullable) cargo 45 passed; gate green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…notes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Specializer M-ir: imperative IR — grammar, types, verifier, text format
…terp) src/specializer/exec/: runtime substrate (Batch, byte-backed bump Arena with StrRef spans, sorted-vec map statics probed by allocation-free binary search) + interp.rs (compile verifies first — nothing executes unverified IR — then one pre-traversal builds per-block closures; terminators are a small enum in the row loop; branch-arg copies are safe by SSA disjointness). Steady state allocates nothing: registers/arena/out builders live in a reusable RunState; a thread-local counting global allocator asserts zero allocs on a warmed run. Semantics pins documented in interp.rs (checked integer arithmetic, IEEE floats, half-away-from-zero ftoi.round, exact-parse stoi/stof) — provisional until the DuckDB differential at M-lower. All five M-ir fixtures execute against hand-computed expectations incl. the trap path; 150-seed generated programs execute deterministically. TASK-42. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… — TASK-42 Six agents (4 attack lenses + 2 reviews) probed the oracle; confirmed findings fixed and pinned: - StrRef offsets widened u32 -> usize: a single call legitimately exceeding 4 GiB of varlen data silently aliased old spans (Ok with corrupt output — the one thing an oracle must never do) - store.opt with a false flag now stores the type-default payload the spec pins, not the live register - check_input validates the validity lane of nullable columns (a short valid vec silently meant "all valid") - register slots assigned densely at compile: sparse-but-legal value ids can no longer inflate the frame (Value(u32::MAX) demanded a 64 GiB vec) - zero-allocation claim restated honestly: warmth is per content profile (branch paths, probe hits, non-NULL varlen), growth one-time + monotone — refuted-as-written by branch/probe/validity flips at identical shape - RunState.emitted reports the row count (observable with 0 out columns) - Ctx made private; out_decl drops its unread nullable flag; probe hit arm zips entry values directly - semantics pins now all tested: overflow/div traps incl. irem MIN%-1, fcmp NaN table, ftoi ties + range traps, IEEE inf/nan flow, scmp order, exact stoi parse, load.opt garbage normalization cargo 65 passed; gate green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…notes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ahrzb
approved these changes
Jul 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three commits of groundwork for the SQL specializer (partial-evaluation engine, see the two new spec docs):
src/datafusion/; engine-agnostic pieces hoisted to the crate root (value.rs,types.rs,error.rs).DuckDBInferFnstub added (parses DuckDB dialect,infer()raises) — the future pyclass shell for the specializer.2026-07-25-sql-specializer-design.md(architecture, resolved decisions, row-major AoS ABI with prepare-time-generated marshaller) and2026-07-25-sql-specializer-loop-execution-design.md(prep checklist + loop/workflow execution design).cargo testnow runnable (extension-module → maturin-only feature);mise gate-specializer= cargo test + pytest, currently green (2 + 574 passed, 12 xfailed)tests/corpus/duckdb_mined.jsonl(three-outcome replay contract)Verification
mise gate-specializergreen end-to-end (cargo test 2 passed; pytest 574 passed, 12 xfailed)🤖 Generated with Claude Code