Skip to content

Commit 08574de

Browse files
ahrzbclaude-agent-ahrzb[bot]
authored andcommitted
refactor!: reset sql-transform to the confit-served path
Start over on the SQLTransform line. What survives is the transform that was already built on confit -- SpecializedTransform, renamed to SQLTransform, which is now the whole package. Removed: * the DataFusion-semantics native engine (the sql-transform Rust crate: datafusion/, lookup.rs, value.rs, InferFn) -- the package is pure Python now and the cargo workspace has one member * the Python codegen backend (sql_transform/_codegen) * the batch transform() path, _batch.py, transformer UDFs * the differential harness and every test built on it (45 files, ~10k lines) * the legacy backlog: tasks 1-34, all 7 decisions, the archived conformance and multi-runtime tasks, and every draft except DRAFT-20 Kept: confit untouched, and the fit-and-serve path -- _compose, _rewrite, _schema, _sql, _state, _transformer_ref. Composition (`t"... {other}(col) ..."`) could not survive: it read the other transform's _state_tables/_rewritten_sql and re-fitted its definition through the deleted batch engine. Repairing it would have meant reimplementing the feature on the line being reset, so the surface is kept and raises NotImplementedError by name, with tests asserting the refusal. Also repointed what would otherwise dangle: the bench scripts' engine registries and debug-build guard (now confit's), _state.py's docstring, and the two native-guard docstrings that cited the now-deleted TASK-33. Gate: pytest 311 passed + 1 xfailed (confit 250+1 unchanged; sql-transform 597 -> 60), cargo 168, pre-commit clean.
1 parent f1ae4d6 commit 08574de

111 files changed

Lines changed: 191 additions & 12835 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

Cargo.lock

Lines changed: 0 additions & 9 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[workspace]
2-
members = ["packages/confit", "packages/sql-transform"]
2+
members = ["packages/confit"]
33
resolver = "2"
44

55
[workspace.package]

README.md

Lines changed: 51 additions & 126 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# SQL Transforms
22

3-
Define ML feature transforms as SQL, fit once, then run them at batch or
4-
low-latency single-row speed.
3+
Define ML feature transforms as SQL, fit once, then serve them row-at-a-time
4+
with sub-microsecond latency.
55

66
## Packages
77

@@ -10,7 +10,11 @@ This repository is a workspace of two packages:
1010
| package | what it is |
1111
|---|---|
1212
| [`packages/confit`](packages/confit) | **Confit** — the serving engine. SQL plus static tables frozen at fit time are partially evaluated, once, into a native function. Serves bit-exact with DuckDB or refuses at build time. Usable on its own. |
13-
| [`packages/sql-transform`](packages/sql-transform) | The authoring layer: `SQLTransform`, the DataFusion-differentiated engine, and the codegen backend. Depends on Confit for specialized serving. |
13+
| [`packages/sql-transform`](packages/sql-transform) | The authoring layer: `SQLTransform` — fit window-aggregate state from training data, then serve through Confit. |
14+
15+
> **sql-transform was reset.** The DataFusion-differentiated native engine, the
16+
> Python codegen backend, and the batch `transform()` path were removed; the
17+
> package is being rebuilt on Confit. Confit itself is unaffected.
1418
1519
## Installation
1620

@@ -24,14 +28,14 @@ pip install confit # the serving engine alone
2428
```bash
2529
git clone https://github.com/ahrzb/sql-transforms.git
2630
cd sql-transforms
27-
mise run install # uv sync — installs both packages and builds their Rust extensions
31+
mise run install # uv sync — installs both packages, builds Confit's extension
2832
```
2933

30-
Each package ships a Rust/PyO3 extension built by
31-
[maturin](https://www.maturin.rs/)`confit._engine` and
32-
`sql_transform._interpreter`. After changing Rust code, rebuild with
33-
`uv run maturin develop` from inside that package's directory; the test suite
34-
also rebuilds automatically when a `.rs` file is newer than the built module.
34+
Confit ships a Rust/PyO3 extension (`confit._engine`) built by
35+
[maturin](https://www.maturin.rs/); sql-transform is pure Python. After changing
36+
Rust code, rebuild with `uv run maturin develop` from `packages/confit`; the test
37+
suite also rebuilds automatically when a `.rs` file is newer than the built
38+
module.
3539

3640
Run the whole gate from the repository root:
3741

@@ -42,8 +46,8 @@ uv run pytest -q && cargo test --release
4246
## Quick Start
4347

4448
```python
45-
from sql_transform import SQLTransform
4649
import pyarrow as pa
50+
from sql_transform import SQLTransform
4751

4852
data = pa.table({
4953
"feature1": [1.0, 2.0, 3.0, 4.0, 5.0],
@@ -58,121 +62,69 @@ SELECT
5862
FROM __THIS__
5963
"""
6064

61-
transformer = SQLTransform(sql)
62-
transformer.fit(data)
65+
t = SQLTransform(sql)
66+
t.fit(data)
6367

64-
# Batch transform through DataFusion (pyarrow in / pyarrow out).
65-
result = transformer.transform(data)
66-
print(result)
67-
68-
# Low-latency inference through the native engine (dict or Pydantic model in,
69-
# typed model out). infer() for one row, infer_batch() for many.
70-
one = transformer.infer({"feature1": 2.0, "feature2": 20})
68+
# Serving: dict or Pydantic model in, typed model out.
69+
one = t.infer({"feature1": 2.0, "feature2": 20})
7170
print(one.feature1_norm)
72-
many = transformer.infer_batch([{"feature1": 2.0, "feature2": 20}])
71+
many = t.infer_batch([{"feature1": 2.0, "feature2": 20}])
7372
```
7473

75-
Per-group statistics use `OVER (PARTITION BY ...)` — the group means/counts/etc.
76-
are frozen at `fit` and looked up per row at inference:
74+
Per-group statistics use `OVER (PARTITION BY ...)` — the group means/counts are
75+
frozen at `fit` and looked up per row at inference:
7776

7877
```python
7978
sql = "SELECT target / MEAN(target) OVER (PARTITION BY city) AS enc FROM __THIS__"
8079
```
8180

82-
## What it supports
83-
84-
- **Window aggregates**, computed once at `fit` and frozen: whole-table `OVER ()`
85-
and per-group `OVER (PARTITION BY ...)` (e.g. `MEAN`, `SUM`, `COUNT`, `STDDEV`).
86-
- **Expressions** (batch and inference): arithmetic, comparisons, `CAST`,
87-
`UPPER`/`LOWER`/`TRIM`/`SUBSTR`/`CONCAT`, `ABS`, `ROUND`, `COALESCE`, `NULLIF`,
88-
with SQL NULL-propagation semantics.
89-
- **Joins**: `INNER`/`CROSS`, plus a static-table lookup join (a row joined to a
90-
preloaded `pyarrow.Table` by key — no per-row Python callback).
91-
- **Typed I/O**: Pydantic models for the input row and output, validated when the
92-
transformer is built and again at call time. Output is a typed model; the input
93-
schema is auto-synthesized or user-supplied.
94-
- **Transformer references**: interpolate a fitted sklearn transformer directly
95-
into a t-string and apply it to columns (see below).
96-
97-
See [docs/SQL_SUPPORT.md](docs/SQL_SUPPORT.md) for the feature-by-feature tracker.
98-
9981
## Architecture
10082

101-
Two phases, two engines, one rewritten query:
83+
Two phases, one rewritten query:
10284

10385
```
10486
SQL over __THIS__
10587
10688
107-
fit(train) ── DataFusion runs the SQL, freezes each window-aggregate (e.g.
108-
│ MEAN(age)) into a typed __STATE__, and rewrites the SQL to
109-
│ reference __STATE__ + the raw row __THIS__ instead of
89+
fit(train) ── DataFusion runs the SQL, freezes each window aggregate (e.g.
90+
│ MEAN(age)) into a typed __STATE__ table, and rewrites the SQL
91+
to reference __STATE__ + the raw row __THIS__ instead of
11092
│ recomputing aggregates.
11193
11294
│ rewritten SQL + frozen state
113-
├───────────────────────────────┬───────────────────────────────┐
114-
▼ ▼
115-
transform(batch) infer(row) / infer_batch(rows)
116-
DataFusion, vectorized native InferFn interpreter, row-at-a-time,
117-
columnar over the batch no SQL engine at call time
118-
```
119-
120-
Both paths run the **same** rewritten SQL against the **same** frozen state, so
121-
they return identical values on the normal numeric path. `fit` pays for a real
122-
query engine once; inference pays only for a lean interpreter walking a plan. This
123-
separation of **fit** (compute statistics) and **transform/infer** (apply them)
124-
is the standard ML pattern — fit on training data, apply to training and serving.
125-
126-
## Referencing a fitted sklearn transformer
127-
128-
Interpolate a fitted transformer into a t-string and apply it to columns:
129-
130-
```python
131-
import pandas as pd
132-
import pyarrow as pa
133-
from sklearn.preprocessing import StandardScaler
134-
from sql_transform import SQLTransform
135-
136-
train_df = pd.DataFrame(
137-
{"age": [10.0, 20.0, 30.0, 40.0], "income": [1.0, 2.0, 3.0, 4.0]}
138-
)
139-
table = pa.Table.from_pandas(train_df)
140-
141-
sc = StandardScaler().fit(train_df) # fit on a DataFrame -> records feature_names_in_
142-
t = SQLTransform(t"SELECT {sc}(age, income) AS scaled FROM __THIS__").fit(table)
143-
```
144-
145-
**Output is a single Arrow struct column**, not one column per feature:
146-
147-
```python
148-
t.transform(table).schema # scaled: struct<age: double, income: double>
149-
t.transform(table).flatten().schema.names # ['scaled.age', 'scaled.income']
95+
96+
Confit ── partially evaluates the pair into a native function:
97+
│ binding-time analysis collapses every static lookup into a
98+
│ prepare-time probe, so nothing general remains at call time.
99+
100+
infer(row) / infer_batch(rows)
150101
```
151102

152-
Call `.flatten()` to get flat columns for an sklearn handoff.
103+
`fit` pays for a real query engine once; serving pays only for straight-line
104+
native code over the frozen state. Confit's contract carries through: the fitted
105+
SQL either serves bit-exact with DuckDB, or `fit()` raises and names the
106+
construct it will not serve — see
107+
[Confit's known limitations](docs/known-limitations.md).
153108

154-
**Column binding** depends on how the transformer was fitted:
109+
## What it supports
155110

156-
| fitted with | `feature_names_in_` | binding |
157-
|---|---|---|
158-
| `fit(DataFrame)` | recorded | by **name** — call order is free, and is validated against the names |
159-
| `fit(ndarray)` | absent | by **position**, in call order — only the count is checked |
111+
- **Window aggregates**, computed once at `fit` and frozen: whole-table `OVER ()`
112+
and per-group `OVER (PARTITION BY ...)` (`MEAN`, `SUM`, `COUNT`, `STDDEV`, …).
113+
- **Everything Confit serves** at inference — the expression surface, joins to
114+
static tables, and the row-shape contract are documented in
115+
[`packages/confit`](packages/confit) and
116+
[docs/known-limitations.md](docs/known-limitations.md).
117+
- **Typed I/O**: Pydantic models for the input row and output, validated when the
118+
transform is fitted and again at call time.
160119

161-
With positional binding, `{sc}(income, age)` against a transformer fitted as
162-
`[age, income]` silently swaps the features. Fit on a DataFrame when you can.
120+
Not currently supported, pending the rebuild: batch `transform()`, sklearn
121+
transformer references, and composing one `SQLTransform` into another.
163122

164-
Aggregating over a transformer's output (`AVG({sc}(age)) OVER ()`) is not
165-
supported — it is inherently two-stage and needs a subquery. Aggregate over an
166-
input column, or use a `SQLTransform` reference, which inlines to a scalar and
167-
composes with aggregates freely:
123+
## Reports
168124

169-
```python
170-
norm = SQLTransform("SELECT age / MEAN(age) OVER () AS a FROM __THIS__").fit(table)
171-
t2 = SQLTransform(
172-
t"SELECT AVG({norm.transform}(age)) OVER () AS m FROM __THIS__"
173-
).fit(table)
174-
t2.transform(table).column("m").to_pylist() # [1.0, 1.0, 1.0, 1.0]
175-
```
125+
- [The architecture of Confit](docs/reports/confit-architecture.md)
126+
- [Pins-first: building a bit-exact engine twin](docs/reports/pins-first-methodology.md)
127+
- [Performance: the serving regime, measured](docs/reports/performance-report.md)
176128

177129
## Development
178130

@@ -182,30 +134,3 @@ mise run fmt # ruff check + format
182134
mise run check # fmt + test
183135
mise tasks # list all tasks
184136
```
185-
186-
## Project docs
187-
188-
- **Milestones** (`backlog milestone list`) — the sequenced phases; tasks (TASK-N) are the actionable units (see Backlog.md below).
189-
- The **reasoning archive***why* things were deferred, design briefs, and ADRs — now
190-
lives in `backlog/docs`, `backlog/decisions`, and `backlog/drafts` (see Backlog.md below).
191-
- **Backlog.md** (`backlog/`) — live task board (`backlog board` / `backlog task
192-
list`) grouped by **milestones** (`backlog milestone list`), architecture
193-
**decisions** (`backlog/decisions/`), and reference **docs** (`backlog/docs/`):
194-
[Vision](<backlog/docs/doc-3 - Vision.md>) (what the project is and where it's
195-
headed), the [DataFusion function catalogue](<backlog/docs/doc-1 - DataFusion-function-catalogue.md>)
196-
(interpreter parity target, auto-generated), and the
197-
[sklearn transformer plan](<backlog/docs/doc-2 - sklearn-transformer-implementation-plan.md>)
198-
(tiers + native-machinery status), and the
199-
[Multi-language inference runtimes design brief](<backlog/docs/doc-4 - multi-language-inference-runtimes-design-brief.md>)
200-
(multi-language serving — out of scope / parked).
201-
202-
## Contributing
203-
204-
1. Pick an item from the [Backlog.md board](backlog/) (`backlog task list` / `backlog board`).
205-
2. Open an issue.
206-
3. Implement with tests.
207-
4. Submit a PR.
208-
209-
## License
210-
211-
MIT

backlog/archive/tasks/task-10 - Conformance-harness-S3-nullability-tolerant-comparator.md

Lines changed: 0 additions & 25 deletions
This file was deleted.

backlog/archive/tasks/task-11 - Conformance-harness-S4-wire-the-steal-the-spec-DoD-hook.md

Lines changed: 0 additions & 24 deletions
This file was deleted.

backlog/archive/tasks/task-12 - Conformance-harness-S2-allowlist-extractor-plus-dual-engine-runner.md

Lines changed: 0 additions & 26 deletions
This file was deleted.

backlog/archive/tasks/task-18 - B1-SPIKE-full-producer-coverage-sweep-Substrait.md

Lines changed: 0 additions & 23 deletions
This file was deleted.

backlog/archive/tasks/task-19 - B2-DECIDE-define-the-frozen-inference-IR.md

Lines changed: 0 additions & 23 deletions
This file was deleted.

0 commit comments

Comments
 (0)