refactor: extract the DuckDB engine into the confit package; monorepo

The serving engine gets a name and a package of its own. Confit: SQL plus
static tables frozen at fit time are cooked once into a native function, then
service is just bringing a portion up to heat.

Layout — a uv + cargo workspace of two packages:

  packages/confit/          the serving engine, standalone and publishable
    src/{specializer,duckdb,error,types,schema}.rs
    confit/                 python package -> confit._engine
    tests/                  the 16 engine suites + the mined corpus
  packages/sql-transform/   authoring layer, depends on confit
    src/{datafusion,lookup,value}.rs
    sql_transform/          python package -> sql_transform._interpreter

Only the DuckDB engine moved. The DataFusion engine and the codegen backend
stay in sql-transform, which now links confit for the shared, semantics-free
vocabulary (error/types/schema) and imports DuckDBInferFn from it.

Coupling was already small: the specializer touched exactly four lines of
crate-root code, and no confit test referenced sql_transform, so the split is
a move rather than a disentangling.

Notes:
  * confit's lib target is named _engine (maturin's convention here), so
    sql-transform renames the crate: `extern crate _engine as confit;`.
  * generator scripts had hardcoded src/ output paths that would have written
    into a directory that no longer exists — repointed.
  * the _interpreter.pyi stub for DuckDBInferFn was stale (it claimed infer()
    raised NotImplementedError); replaced with an accurate confit/_engine.pyi
    covering shape/output/backend/boundary, infer_rows and infer_arrow.
  * both native guards now watch the sources they actually depend on;
    sql-transform's also watches confit's, since it links it.
  * carries DRAFT-20 (the serving-pipelines-in-SQL design notes), parked
    earlier in the same session.

Gate unchanged: cargo 168, pytest 835 passed + 13 xfailed, pre-commit clean,
uv sync --locked clean. Verified confit imports and serves standalone without
sql_transform entering sys.modules.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: 2 people

.gitignore

@@ -24,7 +24,10 @@ wheels/
 # DuckDB source clone — reference corpus for the SQL specializer, not a dependency
 /duckdb/
 
-# Compiled PyO3 extension module (built in-place by `maturin develop`)
-sql_transform/_interpreter*.so
-sql_transform/_interpreter*.pyd
-sql_transform/_interpreter*.pdb
+# Compiled PyO3 extension modules (built in-place by `maturin develop`)
+**/sql_transform/_interpreter*.so
+**/sql_transform/_interpreter*.pyd
+**/sql_transform/_interpreter*.pdb
+**/confit/_engine*.so
+**/confit/_engine*.pyd
+**/confit/_engine*.pdb

Cargo.lock

@@ -1,22 +1,10 @@
-[package]
-name = "sql-transform-interpreter"
+[workspace]
+members = ["packages/confit", "packages/sql-transform"]
+resolver = "2"
+
+[workspace.package]
 version = "0.1.0"
 edition = "2021"
 
-[lib]
-name = "_interpreter"
-crate-type = ["cdylib"]
-
-# extension-module is deliberately NOT a default feature: it suppresses the
-# libpython link, which is right for the wheel but breaks `cargo test`.
-# maturin enables it via [tool.maturin] features in pyproject.toml.
-[dependencies]
-cranelift-codegen = "0.126"
-cranelift-frontend = "0.126"
-cranelift-jit = "0.126"
-cranelift-module = "0.126"
+[workspace.dependencies]
 pyo3 = { version = "0.29", features = ["abi3-py314"] }
-regex = "1"
-unicode-segmentation = "1"
-sqlparser = "0.62"
-target-lexicon = "0.13.5"

Cargo.toml

@@ -3,22 +3,41 @@
 Define ML feature transforms as SQL, fit once, then run them at batch or
 low-latency single-row speed.
 
+## Packages
+
+This repository is a workspace of two packages:
+
+| package | what it is |
+|---|---|
+| [`packages/confit`](packages/confit) | **Confit** — the serving engine. SQL plus static tables frozen at fit time are partially evaluated, once, into a native function. Serves bit-exact with DuckDB or refuses at build time. Usable on its own. |
+| [`packages/sql-transform`](packages/sql-transform) | The authoring layer: `SQLTransform`, the DataFusion-differentiated engine, and the codegen backend. Depends on Confit for specialized serving. |
+
 ## Installation
 
 ```bash
-pip install sql-transform
+pip install sql-transform      # authoring + serving
+pip install confit             # the serving engine alone
 ```
 
 ### Development
 
 ```bash
 git clone https://github.com/ahrzb/sql-transforms.git
 cd sql-transforms
-mise run install        # uv sync — installs deps and builds the Rust extension
+mise run install        # uv sync — installs both packages and builds their Rust extensions
 ```
 
-The inference engine is a Rust/PyO3 module built by [maturin](https://www.maturin.rs/).
-After changing Rust code, rebuild it with `uv run maturin develop`.
+Each package ships a Rust/PyO3 extension built by
+[maturin](https://www.maturin.rs/) — `confit._engine` and
+`sql_transform._interpreter`. After changing Rust code, rebuild with
+`uv run maturin develop` from inside that package's directory; the test suite
+also rebuilds automatically when a `.rs` file is newer than the built module.
+
+Run the whole gate from the repository root:
+
+```bash
+uv run pytest -q && cargo test --release
+```
 
 ## Quick Start
 

README.md

@@ -0,0 +1,131 @@
+---
+id: DRAFT-20
+title: Serving pipelines in SQL — marginalized aggregates, fitted artifacts, and the leakage question
+status: Draft
+type: spike
+created: 2026-07-28
+---
+
+## Where this came from
+
+Design dialogue with AmirHossein, 2026-07-28. Stated goal: **express the entire
+serving pipeline in SQL.** Not implemented, not scheduled — parked so the
+conclusions survive.
+
+## The architecture we converged on
+
+**Aggregates over `__THIS__` are the static half of a binding-time analysis.**
+The user writes one text; a training-side preprocess marginalizes every
+aggregate over `__THIS__` into a params table and rewrites it to a join.
+`PARTITION BY g` becomes the join key. AmirHossein's plan; pipelines appear as
+UDAFs at the training end.
+
+```sql
+-- one text
+SELECT (age - avg(age) OVER (PARTITION BY country)) / stddev(age) AS age_z FROM __THIS__
+-- serving form after rewrite (already compilable today)
+SELECT (age - p.mean) / p.scale FROM __THIS__ LEFT JOIN params p USING (country)
+```
+
+Consequences:
+
+- **Most sklearn transformers become unnecessary.** StandardScaler, MinMax,
+  SimpleImputer, target encoding, one-hot domains are all `avg`/`stddev`/
+  `median`/`DISTINCT`/`GROUP BY`. No importer, no `State<T>` surface — the
+  state struct is what the *rewriter emits*, not a user-facing concept.
+- **A fitted transformer is a static table that gets joined.** Per-group
+  families (per-country PCA) are the general case, not a special case — the
+  group is the join key, and multi-column keys already work.
+- **Params must be pivoted WIDE**, not tidy/long: a long params table would
+  need `SUM(...) GROUP BY` to form a dot product, and serving rejects
+  aggregation by design.
+- **The engine's no-aggregation rule stops being a limitation** and becomes the
+  type system: the rewrite's output is aggregate-free by construction, so
+  anything the specializer refuses is a bug in the rewrite.
+- **The payoff is training/serving skew elimination by construction**, not
+  latency — one text, mechanically split.
+
+## What actually needs building (small)
+
+Everything reduces to **two instructions plus a params-materialization library**:
+
+```
+matvec         → linear, logistic, PCA, any projection (+ link fn)
+tree_ensemble  → GBDT / RF / extra trees
+everything else → desugar over existing lanes; ZERO engine change
+```
+
+Suggested order: (1) params-as-static-tables + desugar catalogue — no engine
+change, covers most real pipelines; (2) `matvec` — unlocks linear/logistic,
+the most-served model class, plus PCA; (3) `tree_ensemble`, sklearn-only,
+when something real needs it.
+
+Sizing (my estimate, from the wave work): a bit-exact tree ensemble for ONE
+model family ≈ 1.5–2 weeks. The tree walk itself is a day. The cost is the
+six-file instruction tax, the importer (per format), and the semantics pins —
+float32-vs-f64 threshold comparison, `<` vs `<=` ties, per-node missing
+direction, link/objective, base_score. Cost scales with *formats supported*,
+not tree count. XGBoost +1wk, LightGBM (categorical bitset splits) +1–2wk;
+that way lies a model-format zoo.
+
+## Bit-exactness, inverted
+
+- Scalers / imputers / encoders / trees: **bit-exact is reachable**. Tree walks
+  are comparisons plus a fixed-order sum.
+- Anything matmul-shaped (PCA, linear): **not bit-exact with sklearn** — numpy
+  dispatches to BLAS, whose summation order varies by build, threading, and
+  size. Needs a documented own-order + tolerance tier, loudly opt-in. It would
+  be the first approximate answer this engine ever serves.
+
+Oracle split: feature SQL keeps DuckDB (existing machinery, unchanged, and
+functions defined AS their expansion stay verifiable — pin whether DuckDB
+`CREATE MACRO` takes struct args). Model scoring's oracle becomes
+sklearn/XGBoost. Closing that seam would mean shipping a DuckDB extension with
+the same functions — roughly a doubling of the work.
+
+## THE OPEN QUESTION — leakage (AmirHossein: "very tricky, brainstorm at some point")
+
+**Cross-fitting breaks the one-text-one-rewrite symmetry**, which is the
+property everything else rests on:
+
+```
+serving:  LEFT JOIN params         USING (country)
+training: LEFT JOIN params_by_fold USING (country, fold)   -- fold ∉ serving
+```
+
+The marginalizing rewrite would no longer be the same transformation in both
+phases. Sub-questions for that session:
+
+1. How does the preprocessor learn **which columns are targets**? `avg(age)` is
+   safe; `avg(target)` leaks. SQL carries no such metadata today.
+2. Refuse, or cross-fit automatically? The design makes the leaky version the
+   easiest thing to write.
+3. Smoothed target encoding (m-estimate / prior) means the marginalized
+   aggregate has **hyperparameters** — it isn't a pure `avg`.
+4. High-cardinality group keys: a group of size 1 is the row's own target.
+
+## Other open items
+
+- Unknown group at serving is unavoidable (a country unseen in training). The
+  join type is the policy — LEFT → null lanes, INNER → row drops, COALESCE to a
+  global row → fallback. Must be an explicit per-table choice.
+- `PARTITION BY g` requires `g` to be a serving input field — a build-time
+  proof that should name the missing column.
+- `avg(x)` silently means training-time, never request-batch. Safe (serving
+  sees one row) but a real trap; wants an explain output.
+- Params-table size ceiling for high-cardinality keys (statics are frozen in
+  memory).
+- Tfidf vocabularies fall on the blob side of the line, same as matrices/trees.
+
+## The line
+
+State-as-struct for fixed-arity scalar params. Opaque frozen blob plus a
+dedicated instruction for anything whose size scales with model complexity —
+matrices, ensembles, vocabularies. A GBDT as struct lanes would be ~150k
+compile-time lanes; its structure *is* control flow, so it belongs in the
+compiled code, not a data field.
+
+End state: the deployable artifact is SQL text + Arrow params tables, no Python
+at serving. Given the WASM spike ran near-native under Go (wazero) and Java
+(chicory, with its opt-in compiler), the same artifact serves from a JVM or Go
+service with nothing reimplemented.

backlog/drafts/draft-20 - Serving-pipelines-in-SQL-marginalized-aggregates-fitted-artifacts-and-the-leakage-question.md

@@ -150,7 +150,8 @@ def engine_run(engines):
 def refuse_debug_build():
     """Abort unless the imported native extension is a release build.
 
-    tests/_native_guard.py rebuilds the cwd-local .pyd via plain `maturin
+    packages/sql-transform/tests/_native_guard.py rebuilds the cwd-local .pyd
+    via plain `maturin
     develop` (debug) whenever src/*.rs is newer; that shadows the venv's
     release wheel and silently inflates engine rows ~5x (measured 2026-07-26).
     """

benchmarks/bench_serving.py

@@ -69,7 +69,7 @@ def rows_table(mod, rows: list[dict]) -> pa.Table:
 
 def build_spec_fn(mod, statics: dict[str, pa.Table], output: str = "model"):
     """The specializer serving fn (env knobs select backend/boundary)."""
-    from sql_transform._interpreter import DuckDBInferFn
+    from confit import DuckDBInferFn
 
     return DuckDBInferFn(
         mod.SQL,

benchmarks/serving_scenarios/__init__.py

@@ -2,7 +2,7 @@
 
 This is the user-facing contract of the SQL specializer (`DuckDBInferFn`):
 what it refuses to serve, **why**, and what you see when you hit a limit.
-Its executable twin is [`tests/test_known_limitations.py`](../tests/test_known_limitations.py) —
+Its executable twin is [`packages/confit/tests/test_known_limitations.py`](../tests/test_known_limitations.py) —
 every limitation below is asserted there, so lifting one breaks a test and
 forces this document to change with it.
 
@@ -105,7 +105,7 @@ wrong answer or require semantics we can't reproduce exactly:
 | `^` operator | It IS pow in DuckDB, but sqlparser's precedence differs from DuckDB's (`2*x^y` would parse as `(2*x)^y`). Mapping it computes the wrong tree silently. Use `pow()`. |
 | prefix `~`, `#`, `NOT GLOB` | Same class: precedence/parse divergences that would silently mis-associate. `xor()` covers bit-xor; `NOT (x GLOB p)` works. |
 | Regex reject list: `\B`, `\Q…\E`, `(?<name>…)`, duplicate group names, bounds > 1000, stacked quantifiers (`a*+`), `\u` escapes, negated Perl classes inside `[...]` | The RE2↔rust-regex differential battery (98 entries) proved these are the constructs where the engines disagree or DuckDB itself is broken (`\B` crashes DuckDB at runtime on non-ASCII). Everything else is byte-identical. |
-| Fuzzer-found regex rejects (TASK-54): `\1`–`\9` backrefs outside classes, the full stacked-quantifier grammar (`{2}*`, `?*`, `a???` — one lazy `?` is the only legal follower), nested repetition products > 1000, whitespace inside `{m, n}` bounds, class set-op lookalikes (`--`/`&&`/`~~`), non-POSIX `[` inside classes, Perl-class range endpoints (`[a-\d]`), capturing `(x){0}`, anchor-only multi-anchor patterns (DuckDB is SELF-inconsistent on these — its row path disagrees with its own constant fold), `$` anchors in non-final position (`'$hello'` — DuckDB's row path literal-optimizes the leading `$`+literal into a PREFIX match, matching "hello world", while its own constant fold matches normally; found by the standing fuzzer on seed 20260728), and counted repetitions over RE2's PROGRAM-SIZE budget (`(\p{L}){1,500}` is "pattern too large" in DuckDB while rust-regex serves it — rejected via a one-sided weight estimate that always fires before DuckDB's real budget; same seed, pins `pins-waveB/fuzzer-20260728.json`) | The standing differential fuzzer (`tests/test_duckdb_regexp_fuzz.py`, in the normal gate) found these 12 classes in its first 3k-case deep run — each one a silent-wrong-answer risk in rust-regex — then re-swept to ZERO divergences over 40k cases across 8 seeds. Pins: `pins-waveB/fuzzer-task54.json`. |
+| Fuzzer-found regex rejects (TASK-54): `\1`–`\9` backrefs outside classes, the full stacked-quantifier grammar (`{2}*`, `?*`, `a???` — one lazy `?` is the only legal follower), nested repetition products > 1000, whitespace inside `{m, n}` bounds, class set-op lookalikes (`--`/`&&`/`~~`), non-POSIX `[` inside classes, Perl-class range endpoints (`[a-\d]`), capturing `(x){0}`, anchor-only multi-anchor patterns (DuckDB is SELF-inconsistent on these — its row path disagrees with its own constant fold), `$` anchors in non-final position (`'$hello'` — DuckDB's row path literal-optimizes the leading `$`+literal into a PREFIX match, matching "hello world", while its own constant fold matches normally; found by the standing fuzzer on seed 20260728), and counted repetitions over RE2's PROGRAM-SIZE budget (`(\p{L}){1,500}` is "pattern too large" in DuckDB while rust-regex serves it — rejected via a one-sided weight estimate that always fires before DuckDB's real budget; same seed, pins `pins-waveB/fuzzer-20260728.json`) | The standing differential fuzzer (`packages/confit/tests/test_duckdb_regexp_fuzz.py`, in the normal gate) found these 12 classes in its first 3k-case deep run — each one a silent-wrong-answer risk in rust-regex — then re-swept to ZERO divergences over 40k cases across 8 seeds. Pins: `pins-waveB/fuzzer-task54.json`. |
 | `SIMILAR TO ... ESCAPE` | Not implemented in DuckDB itself. |
 | `* EXCLUDE (t.key)` on a USING join | DuckDB UNMERGES the coalesced column (it reappears at the right table's position) — measured, not modeled. Unqualified EXCLUDE works. |
 | `BETWEEN`/`IN` mixing non-numeric string literals with numbers | DuckDB converts at EXECUTION time (an empty input succeeds!); a bind-time conversion was measured to be over-eager. Numeric literals convert fine. |
@@ -131,7 +131,7 @@ These are served, but with a consciously chosen surface — know them:
   corpus only ever compares successful results, so texts never affect
   parity.
 - **Two known oracle divergences** (excluded from the corpus by name in
-  `tests/test_corpus_replay.py::_KNOWN_DIVERGENT_SOURCES`): DuckDB
+  `packages/confit/tests/test_corpus_replay.py::_KNOWN_DIVERGENT_SOURCES`): DuckDB
   behaviors that depend on column STATISTICS (e.g. ILIKE's NUL handling
   selects a different kernel depending on *sibling rows*). A row-at-a-time
   engine cannot reproduce statistics-dependent semantics even in
@@ -172,9 +172,9 @@ Three mechanisms, all in the normal test gate:
 1. **The corpus replay** (678 statements mined from DuckDB's test suite):
    every statement must match bit-for-bit, reject cleanly, or be a named
    divergence — a wrong answer anywhere fails the gate.
-2. **The executable twin** (`tests/test_known_limitations.py`): every
+2. **The executable twin** (`packages/confit/tests/test_known_limitations.py`): every
    limitation in this document is asserted; lifting one breaks a test.
-3. **The standing differential fuzzer** (`tests/test_duckdb_regexp_fuzz.py`):
+3. **The standing differential fuzzer** (`packages/confit/tests/test_duckdb_regexp_fuzz.py`):
    randomized DuckDB-vs-engine sweeps of the regex surface on every run
    (seed/size overridable for deep runs) — new divergences fail with the
    reproducing seed and SQL, and their fix lands as a reject-list entry