Specializer M-cranelift: codegen backend behind the same interface (TASK-44)

Cargo.toml

@@ -11,5 +11,10 @@ crate-type = ["cdylib"]
 # libpython link, which is right for the wheel but breaks `cargo test`.
 # maturin enables it via [tool.maturin] features in pyproject.toml.
 [dependencies]
+cranelift-codegen = "0.126"
+cranelift-frontend = "0.126"
+cranelift-jit = "0.126"
+cranelift-module = "0.126"
 pyo3 = { version = "0.29", features = ["abi3-py314"] }
 sqlparser = "0.62"
+target-lexicon = "0.13.5"

backlog/tasks/task-43 - Specializer-M-lower-frontend-BTA-produce-consume-lowering-for-the-v0-subset.md

@@ -3,10 +3,10 @@ id: TASK-43
 title: >-
   Specializer M-lower: frontend + BTA + produce/consume lowering for the v0
   subset
-status: In Progress
+status: Done
 assignee: []
 created_date: '2026-07-25 02:31'
-updated_date: '2026-07-26 05:14'
+updated_date: '2026-07-26 06:04'
 labels: []
 milestone: m-7
 dependencies:
@@ -54,6 +54,8 @@ AC #2 (static-only queries fold to a constant emitter) is NOT yet satisfied: nee
 Stretch 4 landed (commit ac3786a): builtin catalogue implemented strictly from measured pins (8-agent workflow fan-out measured DuckDB 1.5.5; spec at docs/superpowers/specs/2026-07-26-stretch4-builtin-pins.md). New IR insts across all six files + fuzz gen: supper/slower, strim.{both,lead,trail}, ssubstr (virtual-window codepoint arithmetic), iabs/fabs/fround, BinOp::Frem. Frontend catalogue: upper/lower/ltrim/rtrim/abs/round(1-arg)/concat/coalesce/nullif + TRIM/SUBSTRING dedicated AST forms; || is ALWAYS concat (even 1 || 2) with implicit VARCHAR casts; CONCAT skips NULLs (all-NULL -> ''); COALESCE per-row lazy via CASE desugار; NULLIF compares at promoted type, keeps first arg's type. TWO DIVERGENCES IN PREVIOUSLY-LANDED CODE were found by pinning and FIXED in-branch with pin tests (PM: flagging per the disagreement protocol — these were fixes toward the oracle, not semantics patches to pass tests): (1) integer % by zero returns NULL in DuckDB, we trapped -> lowering now CASE-guards the divisor, MIN % -1 still traps like DuckDB; (2) DOUBLE comparisons: DuckDB order is NaN=NaN / NaN above everything / zeros equal, we had IEEE partial -> shared exec::duck_fcmp used by interp AND fold. Known deliberate divergence (strict-xfail + needs a ticket): Rust std lacks Unicode SIMPLE case maps, so upper('ß') gives 'ß' vs DuckDB 'ẞ' and lower('İ') differs; ASCII exact. Deliberate ceilings: round(x, digits) unsupported; decimal literals stay f64 (stretch-1 ceiling). 114 cargo tests + 21 differential pytest + 1 strict xfail; gate green. Adversarial verify fan-out (6 probe agents vs duckdb) running; findings will be triaged fix-vs-xfail before stretch 5 (differential suite backend id + corpus replay in gate).
 
 Stretch 6 landed (commit 0782bf1): AC #2 satisfied — static-only queries become constant emitters, evaluated once at build time by DuckDB itself (Python boundary; no IR is built, so trivially no probe/filter ops). The fallback fires on Unsupported/Parse prepare errors and self-validates: dynamic queries reference the row table, unknown to DuckDB, so evaluation fails and the original clean error surfaces. Bonus: aggregation/ORDER BY/dialect-beyond-sqlparser all work on the static-only path. Final corpus: 53 match / 625 clean-unsupported / 0 FAIL of 678, wired into the gate via pytest. MILESTONE COMPLETE pending review — hard stop before M-cranelift (TASK-44). Two items for PM: (1) deviation note — AC #3's 'backend id specialized' wording: the DataFusion-oracle differential harness (test_diff_*) can't host the specializer (different oracle, e.g. `/` semantics differ by design), so the specializer has its own duck_check differential suite (32 cases) instead; (2) ticket request per the disagreement protocol — Unicode SIMPLE case mapping (upper('ß')/'İ'/'ᾀ' class): Rust std only exposes full maps; strict-xfail in place; candidate fixes are a small static table of the ~100 divergent codepoints or a unicode-data crate dependency.
+
+Post-review addendum (commit 4f75e3d): both PM items resolved. (1) Case-mapping ticket WITHDRAWN — AmirHossein decided no dependency; implemented as a measured, generated exception table (src/specializer/exec/casemap.rs, 139 entries) from scripts/gen_casemap.py. Two exception classes, both captured by measurement: simple-vs-full mapping divergence, and three-way Unicode version skew (Python/Rust/utf8proc all ship different vintages — 56 entries Python's tables cannot see, caught by the generator's phase-2 census of the actual compiled engine). The full-codepoint census test (every Unicode scalar value through both engines) is the standing authority; generator is idempotent. The strict xfail is now two passing tests: gate 605 pytest + 12 xfail. (2) DataFusion: per review — legacy serving line's oracle only, likely retired later (small chance: a supported dialect); the specializer ignores it entirely.
 <!-- SECTION:NOTES:END -->
 
 ## Final Summary

backlog/tasks/task-44 - Specializer-M-cranelift-codegen-backend-behind-the-same-interface.md

@@ -1,9 +1,10 @@
 ---
 id: TASK-44
 title: 'Specializer M-cranelift: codegen backend behind the same interface'
-status: To Do
+status: In Progress
 assignee: []
 created_date: '2026-07-25 02:31'
+updated_date: '2026-07-26 06:05'
 labels: []
 milestone: m-7
 dependencies:
@@ -27,3 +28,13 @@ Cranelift-jit backend for the imperative IR (design doc §7; cranelift-jit 0.126
 - [ ] #3 p50/p99 ns/call reported at n in {1, 8, 64, 1024} against the interpreter control and the existing native + codegen engines
 - [ ] #4 mise gate-specializer green
 <!-- AC:END -->
+
+## Implementation Plan
+
+<!-- SECTION:PLAN:BEGIN -->
+Stretch plan (recorded 2026-07-26, design doc §7 + §10):
+1. ABI + scalar spine: add cranelift-jit 0.126 (+module/codegen/frontend); exec/cranelift.rs. One JIT'd fn per program, called per row: extern "C" fn(ctx: *mut RowCtx) -> i64 (0 = emit, 1 = skip, 2+k = trap k; trap messages in a side table). RowCtx carries column base pointers, row index, output sink, arena. Coverage-first op strategy: inline CLIF only where trivially safe (const, float arith, cmp via duck_fcmp helper or inline, select, conversions); EVERYTHING nontrivial (checked int arith, strings, probes, loads/stores) via extern "C" helpers shared with the interpreter's semantics — correctness and full coverage first, inlining hot ops is a later measured optimization. ponytail: helper-call backend that agrees beats an inline backend that diverges.
+2. CFG: IR blocks/params map 1:1 to CLIF blocks/params (the IR was shaped for this); Brif/Jump/Emit/Skip/Trap terms. compile_cranelift(p, statics) -> Result<CompiledFn, Unsupported>; DuckDBInferFn tries cranelift, falls back to interpreter (AC #2), exposes which backend ran for tests.
+3. Differential: gen.rs random-IR fuzz interpreter-vs-cranelift (same seeds, byte-identical outputs incl. NaN/-0.0/arena strings); corpus replay + duck_check suite through the cranelift backend; gate green.
+4. Bench per §10: baseline the boundary with a no-op f first, then p50/p99 ns/call at n in {1,8,64,1024}, interpreter as control, next to the existing native + codegen engines; committed as a script, numbers into the task.
+<!-- SECTION:PLAN:END -->

scripts/bench_specializer.py

@@ -0,0 +1,150 @@
+"""Specializer ns/call bench — the design doc §10 measurement discipline.
+
+Reports p50/p99 ns per call at n in {1, 8, 64, 1024} for:
+  boundary   — a no-op passthrough (SELECT a FROM __THIS__) through the
+               generic pydantic path: the floor every engine pays today
+  cranelift  — the specializer's JIT backend (default)
+  interp     — the same programs on the interpreter backend (the control;
+               SPECIALIZER_FORCE_INTERP=1 in a subprocess)
+  native     — the existing DataFusion-semantics InferFn
+  codegen    — the existing Python codegen engine
+
+Run: `uv run python scripts/bench_specializer.py [--json out.json]`
+The interp control re-execs this script in a subprocess so the env knob is
+set before the module builds its functions.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import subprocess
+import sys
+import time
+
+import pyarrow as pa
+from pydantic import create_model
+
+NS = [1, 8, 64, 1024]
+CASES = {
+    "noop": "SELECT a FROM __THIS__",
+    "arith+where": "SELECT a * 2 + 1 AS x, b / 2 AS h FROM __THIS__ WHERE a % 3 <> 0",
+    "strings": "SELECT upper(s) || '-' || a AS t, substr(s, 2, 3) AS m FROM __THIS__",
+    "join": "SELECT a, name FROM __THIS__ JOIN dim ON a = dim.id",
+}
+
+
+def rows_of(n):
+    return [{"a": i % 60, "b": (i % 7) / 3.0, "s": f"item{i}"} for i in range(n)]
+
+
+def quantiles(samples_ns):
+    s = sorted(samples_ns)
+    return s[len(s) // 2], s[min(len(s) - 1, int(len(s) * 0.99))]
+
+
+def bench_fn(f, n):
+    iters = max(30, min(3000, 3_000_000 // max(n, 1)))
+    f()  # warm
+    out = []
+    for _ in range(iters):
+        t0 = time.perf_counter_ns()
+        f()
+        out.append(time.perf_counter_ns() - t0)
+    return quantiles(out)
+
+
+def build_specializer(sql, model, dim):
+    from sql_transform._interpreter import DuckDBInferFn
+
+    return DuckDBInferFn(
+        sql, row_tables={"__THIS__": model}, static_tables={"dim": dim}
+    )
+
+
+def build_native(sql, model, dim):
+    from sql_transform._interpreter import InferFn
+
+    return InferFn(sql, row_tables={"__THIS__": model}, static_tables={"dim": dim})
+
+
+def build_codegen(sql, model, dim):
+    from sql_transform._codegen import CodegenFn
+
+    return CodegenFn(sql, row_tables={"__THIS__": model}, static_tables={"dim": dim})
+
+
+def main():
+    model = create_model("Row", a=(int, ...), b=(float, ...), s=(str, ...))
+    dim = pa.table({"id": list(range(60)), "name": [f"n{i}" for i in range(60)]})
+    engine = os.environ.get("BENCH_ENGINE", "cranelift")
+
+    builders = {"cranelift": build_specializer, "interp": build_specializer}
+    if engine in ("native", "codegen"):
+        builders = {"native": build_native, "codegen": build_codegen}
+
+    results = {}
+    for case, sql in CASES.items():
+        try:
+            fn = builders[engine](sql, model, dim)
+        except Exception as e:  # noqa: BLE001 -- engines differ in coverage
+            results[case] = {"error": str(e)[:120]}
+            continue
+        if engine in ("cranelift", "interp"):
+            want = "interpreter" if engine == "interp" else "cranelift"
+            assert fn.backend == want, f"{case}: backend is {fn.backend}, wanted {want}"
+        per_n = {}
+        for n in NS:
+            objs = [model(**r) for r in rows_of(n)]
+            p50, p99 = bench_fn(lambda f=fn, o=objs: f.infer({"__THIS__": o}), n)
+            per_n[n] = {"p50_ns": p50, "p99_ns": p99}
+        results[case] = per_n
+    print(json.dumps({engine: results}))
+
+
+def orchestrate():
+    """Run every engine (interp in a subprocess for the env knob), merge."""
+    merged = {}
+    for engine in ("cranelift", "interp", "native", "codegen"):
+        env = os.environ.copy()
+        env["BENCH_ENGINE"] = engine
+        if engine == "interp":
+            env["SPECIALIZER_FORCE_INTERP"] = "1"
+        out = subprocess.run(  # noqa: S603 -- fixed argv
+            [sys.executable, __file__, "--engine-run"],
+            env=env,
+            capture_output=True,
+            text=True,
+            check=True,
+        )
+        merged.update(json.loads(out.stdout.strip().splitlines()[-1]))
+
+    # Human table: p50 ns/call (p99 in parens), one row per case+engine.
+    hdr = f"{'case':<12} {'engine':<10}" + "".join(f"{f'n={n}':>16}" for n in NS)
+    print(hdr)
+    print("-" * len(hdr))
+    for case in CASES:
+        for engine, per_case in merged.items():
+            r = per_case.get(case)
+            if r is None:
+                continue
+            if "error" in r:
+                print(f"{case:<12} {engine:<10}  unsupported: {r['error'][:60]}")
+                continue
+            cells = "".join(
+                f"{r[str(n)]['p50_ns'] if str(n) in r else r[n]['p50_ns']:>13,}ns"
+                for n in NS
+            )
+            print(f"{case:<12} {engine:<10}{cells}")
+    if "--json" in sys.argv:
+        path = sys.argv[sys.argv.index("--json") + 1]
+        with open(path, "w", encoding="utf-8") as f:
+            json.dump(merged, f, indent=2)
+        print(f"wrote {path}")
+
+
+if __name__ == "__main__":
+    if "--engine-run" in sys.argv:
+        main()
+    else:
+        orchestrate()

src/duckdb/mod.rs

@@ -14,8 +14,10 @@ use pyo3::types::PyDict;
 
 use crate::error::InterpError;
 use crate::schema;
+use crate::specializer::exec::cranelift::{self, CraneliftFn};
 use crate::specializer::exec::interp::{compile, InterpFn};
 use crate::specializer::exec::{Batch, ColData, KeyBits, OutCol, ScalarVal, StaticData};
+use crate::specializer::exec::{RunState, Trap};
 use crate::specializer::ir::{Col, ColTy, StaticTy, Ty};
 use crate::specializer::plan::StaticTable;
 use crate::specializer::{prepare, StaticSpec};
@@ -169,9 +171,38 @@ fn eval_static_only(
     Ok((rows, fields))
 }
 
+/// The execution backend: cranelift when it compiles, the interpreter as
+/// the always-available fallback (AC #2 — an uncovered op must not fail
+/// prepare). Both agree byte-for-byte by the 500-seed differential.
+enum Backend {
+    Cranelift(CraneliftFn),
+    Interp(InterpFn),
+}
+
+impl Backend {
+    fn name(&self) -> &'static str {
+        match self {
+            Backend::Cranelift(_) => "cranelift",
+            Backend::Interp(_) => "interpreter",
+        }
+    }
+    fn new_state(&self) -> RunState {
+        match self {
+            Backend::Cranelift(f) => f.new_state(),
+            Backend::Interp(f) => f.new_state(),
+        }
+    }
+    fn run(&self, input: &Batch, st: &mut RunState) -> Result<(), Trap> {
+        match self {
+            Backend::Cranelift(f) => f.run(input, st),
+            Backend::Interp(f) => f.run(input, st),
+        }
+    }
+}
+
 enum Engine {
     Compiled {
-        fun: InterpFn,
+        fun: Backend,
         in_cols: Vec<Col>,
         out_cols: Vec<Col>,
     },
@@ -293,7 +324,34 @@ impl DuckDBInferFn {
             data.push(materialize_map(py, table, spec, keys, values)?);
         }
 
-        let fun = compile(&prepared.program, data).map_err(|e| build_err(e.to_string()))?;
+        // SPECIALIZER_FORCE_INTERP pins the interpreter — the bench control
+        // and a debugging escape hatch.
+        let force_interp = std::env::var_os("SPECIALIZER_FORCE_INTERP").is_some();
+        let fun = match (force_interp, data) {
+            (true, data) => Backend::Interp(
+                compile(&prepared.program, data).map_err(|e| build_err(e.to_string()))?,
+            ),
+            (false, data) => match cranelift::compile(&prepared.program, data) {
+                Ok(f) => Backend::Cranelift(f),
+                // The failed attempt consumed the static data; rematerialize
+                // on this cold path and fall back to the interpreter.
+                Err(_) => {
+                    let mut data = Vec::with_capacity(prepared.statics.len());
+                    for (spec, sty) in prepared.statics.iter().zip(&prepared.program.statics) {
+                        let StaticTy::Map { keys, values } = sty else {
+                            return Err(build_err("internal: v0 lowering emits only map statics"));
+                        };
+                        let table = static_tables
+                            .get(&spec.table)
+                            .expect("spec names come from the catalog");
+                        data.push(materialize_map(py, table, spec, keys, values)?);
+                    }
+                    Backend::Interp(
+                        compile(&prepared.program, data).map_err(|e| build_err(e.to_string()))?,
+                    )
+                }
+            },
+        };
         let output_model = match output_model {
             // Supplied models are trusted as-is in v0 (no shape validation).
             Some(m) => m,
@@ -310,6 +368,15 @@ impl DuckDBInferFn {
         })
     }
 
+    /// Which engine executes: "cranelift", "interpreter", or "constant".
+    #[getter]
+    fn backend(&self) -> &'static str {
+        match &self.engine {
+            Engine::Compiled { fun, .. } => fun.name(),
+            Engine::Constant { .. } => "constant",
+        }
+    }
+
     #[pyo3(signature = (tables=None, **kwargs))]
     fn infer(
         &self,