|
| 1 | +# Transformers as UDAFs, v0: just run the pipeline |
| 2 | + |
| 3 | +**Date:** 2026-07-29 |
| 4 | +**Status:** loop 5 of SQLProjection; designed in dialogue with AmirHossein. |
| 5 | +**Deliberately deferred** (AmirHossein's scoping): desugaring transformers |
| 6 | +into SQL windows/applies (that is the *optimization* pass, soon), and the |
| 7 | +`sklearn.*` curated-semantics namespace (needs an index of sklearn |
| 8 | +functions). This loop runs the actual sklearn objects. |
| 9 | + |
| 10 | +## Surface |
| 11 | + |
| 12 | +A window call whose function name is **not in DuckDB's function catalog** |
| 13 | +resolves to a transformer: first from an explicit registry, then from the |
| 14 | +caller's Python scope (the `FROM df` replacement-scan idiom): |
| 15 | + |
| 16 | +```python |
| 17 | +embed = Pipeline([("sc", StandardScaler()), ("pca", PCA(2))]) |
| 18 | +p = SQLProjection( |
| 19 | + "SELECT embed(* EXCLUDE (label, country)) OVER (PARTITION BY country) AS e," |
| 20 | + " age + 1 AS a1 FROM __THIS__", |
| 21 | + this_model=Row, # transformers={"embed": ...} also accepted |
| 22 | +).fit(train) |
| 23 | +out = p.transform(table) # NEW: batch apply (pa.Table -> pa.Table) |
| 24 | +``` |
| 25 | + |
| 26 | +- Lookup order: `transformers=` dict beats caller scope (locals then globals, |
| 27 | + frames walked past our own library code). The oracle-catalog guard means a |
| 28 | + scope variable can never shadow a real SQL function. Unknown name + no |
| 29 | + match → named refusal with a hint. Captured objects are snapshotted at |
| 30 | + construction and exposed as `p.transformers`. |
| 31 | +- A transformer object is accepted iff it has `fit` and `transform` |
| 32 | + (duck-typed; sklearn `clone` used when available, else `copy.deepcopy`). |
| 33 | +- Arguments: one bundle — a plain column, a `struct_pack(...)`, or `*` / |
| 34 | + `COLUMNS(...)` with the loop-4 modifiers (schema mode required for |
| 35 | + star forms; the bundle rule from the design dialogue: one call, one named |
| 36 | + struct, model order). |
| 37 | +- `PARTITION BY` = per-group fitting (a clone per group, NULL keys are a |
| 38 | + group). `OVER ()` = one global fit. ORDER BY / frames on transformer |
| 39 | + windows: named refusals. |
| 40 | +- v0 restrictions, each a named refusal: transformer calls only in the |
| 41 | + final level of a chain; only as a top-level select item (no nesting the |
| 42 | + result in an expression); no transformer inside FILTER/keys/aggregates. |
| 43 | + |
| 44 | +## The plan: a second step kind |
| 45 | + |
| 46 | +`FitStep` grows `kind: "sql" | "fit"` (sql default) plus, for fit steps, |
| 47 | +`transformer` (registry name), `features` (bundle field names, model order), |
| 48 | +and `keys` (partition key column names in the level table). The level table |
| 49 | +materializes the bundle as `__cf_f{n}` columns and keys as `__cf_k{m}` — |
| 50 | +same mechanism as windows and keys today. The executor grows one branch: |
| 51 | + |
| 52 | +``` |
| 53 | +__CF_LEVEL_0__ sql reads (__THIS__,) SELECT <items>, <f-cols>, <k-cols> FROM __THIS__ |
| 54 | +__CF_PARAMS_1__ fit reads (__CF_LEVEL_0__,) transformer=embed keys=(country,) features=(age, fare) |
| 55 | +``` |
| 56 | + |
| 57 | +A fit step's "params table" is not SQL-shaped in v0 (no wide extraction — |
| 58 | +that is the deferred optimization): `fit()` stores fitted clones on the |
| 59 | +projection, keyed by partition-key tuple, in `p.fitted[step.name]`. The |
| 60 | +plan's DAG order and `reads` stay exactly as loop 3 defined them. |
| 61 | + |
| 62 | +## Serving/apply: batch `transform(table)` |
| 63 | + |
| 64 | +`serving_sql` cannot express an opaque sklearn apply, so v0 adds a **batch** |
| 65 | +apply path (row-at-a-time `infer` through Confit stays NotImplemented): |
| 66 | + |
| 67 | +- In `serving_sql`, each transformer item is replaced by helper columns: |
| 68 | + its bundle features (`__cf_tf{i}_f{n}`) and its partition keys |
| 69 | + (`__cf_tf{i}_k{m}`), all row-wise expressions the SQL layer can compute. |
| 70 | +- `transform(table)` runs `serving_sql` through DuckDB (threads=1, params |
| 71 | + registered), then for each transformer item: group rows by the key tuple, |
| 72 | + look up the fitted clone (unseen group → all-NULL output, the exact-join |
| 73 | + policy), call `.transform` on the feature block, splice the result in at |
| 74 | + the item's position under its alias, drop helpers. |
| 75 | +- Transformer output: sklearn returns an (n, k) array → a |
| 76 | + `FixedSizeList(float64, k)` column in v0 (named struct fields are part of |
| 77 | + the deferred optimization; k is discovered at fit, which is why the output |
| 78 | + schema is fit-time knowledge — documented). |
| 79 | + |
| 80 | +## The gate |
| 81 | + |
| 82 | +The round-trip invariant splits by column kind, both halves oracle-anchored: |
| 83 | + |
| 84 | +- SQL columns: DuckDB differential, unchanged, bit-exact. |
| 85 | +- Transformer columns: `p.fit(train).transform(train)` must equal an |
| 86 | + **independent reference path** — pandas groupby over the same keys, |
| 87 | + `clone(est).fit(X_g).transform(X_g)` per group — allclose with tight |
| 88 | + tolerance (sklearn's own numerics are BLAS-order dependent; bit-exactness |
| 89 | + was never on the table for this stratum, per the DRAFT-20 sizing). |
| 90 | +- Refusal tests for every new named refusal; corpus untouched this loop |
| 91 | + (transformers are not SQL statements; a transformer corpus can come with |
| 92 | + the desugar pass). |
| 93 | + |
| 94 | +## Out of scope, explicitly |
| 95 | + |
| 96 | +Desugar-to-SQL (scaler family etc.), wide params extraction, `sklearn.*` |
| 97 | +namespace and its semantics contracts, matvec/tree_ensemble instructions, |
| 98 | +Confit wiring of applies, transformer calls in non-final levels, struct |
| 99 | +access into transformer results, the DRAFT-20 leakage question. |
0 commit comments