diff --git a/backlog/drafts/draft-10 - Feature-output-tfidf-array-multi-hot-opaque-needs-explode.md b/backlog/drafts/draft-10 - Feature-output-tfidf-array-multi-hot-opaque-needs-explode.md index 9b752e6..10aea4c 100644 --- a/backlog/drafts/draft-10 - Feature-output-tfidf-array-multi-hot-opaque-needs-explode.md +++ b/backlog/drafts/draft-10 - Feature-output-tfidf-array-multi-hot-opaque-needs-explode.md @@ -4,7 +4,7 @@ title: 'Feature output: tfidf / array multi-hot (opaque, needs explode)' status: Draft assignee: [] created_date: '2026-07-18 15:52' -updated_date: '2026-07-23 04:46' +updated_date: '2026-07-23 14:31' labels: - feature-output milestone: m-1 @@ -20,7 +20,27 @@ ordinal: 17000 ## Description -tfidf and array multi-hot need variable-expansion (explode), so they are the OPAQUE ones -- map onto the shipped opaque-transform mechanism (decision-3), corpus-gated. Distinct from the sparse-column / scalar-one-hot tasks (TASK-14/15) which are fixed-fanout and composable. Depends on opaque-transform (Part 1/2 shipped). +WHAT A USER HITS +You have a free-text column — a listing description, a support ticket body, a product title — and you want it as features: + + tfidf = TfidfVectorizer().fit(train["description"]) + SELECT {tfidf}(description) AS text_feats FROM __THIS__ + +One input row produces a wide sparse vector over the learned vocabulary (thousands of terms). Same for array multi-hot: a `tags` column holding ["garage", "pool", "corner_lot"] expands to one indicator per known tag. + +This is the shape the current engine cannot express. Every other transformer we compile is FIXED FAN-OUT — one input value maps to a known number of output columns decided at fit time, and the expression stays per-row. tfidf needs variable expansion (an explode), so it does not fit the scalar/window-agg surface the fast path is built on. + +Practical consequence for the user: text features are the one preprocessing step that will NOT fuse into the single native serving expression. They run through the opaque callout, which means a Python interpreter in the loop and no Go/Java/WASM serving for a pipeline containing them. + +WHAT THIS TICKET DOES +Route tfidf and array multi-hot onto the shipped opaque-transform mechanism (decision-3), corpus-gated. Deliberately NOT trying to make them native — accepting the opaque path is the correct trade for a fundamentally variable-width operation. + +Distinct from DRAFT-9 (sparse COO column) and TASK-15 (scalar one-hot), which are fixed-fanout and composable — those stay on the fast path. This is the escape hatch for the ones that genuinely cannot. + +WHY IT IS A DRAFT +Very low priority (AmirHossein, 2026-07-23) and depends on DRAFT-9: tfidf output is sparse, so it needs the sparse-column materializer path to exist before it is buildable. Re-promote only if text features get actual demand. + +Context: doc-10 (feature-output model), doc-9 (rich type system and UNNEST). ## Acceptance Criteria diff --git a/backlog/drafts/draft-11 - SQL-named-argument-authoring-surface-for-transformer-calls-needs-Rust-ScalarUDF-or-upstream-bindings.md b/backlog/drafts/draft-11 - SQL-named-argument-authoring-surface-for-transformer-calls-needs-Rust-ScalarUDF-or-upstream-bindings.md new file mode 100644 index 0000000..b3de6ac --- /dev/null +++ b/backlog/drafts/draft-11 - SQL-named-argument-authoring-surface-for-transformer-calls-needs-Rust-ScalarUDF-or-upstream-bindings.md @@ -0,0 +1,65 @@ +--- +id: DRAFT-11 +title: >- + SQL named-argument authoring surface for transformer calls (needs Rust + ScalarUDF or upstream bindings) +status: Draft +assignee: [] +created_date: '2026-07-23 14:11' +updated_date: '2026-07-23 14:33' +labels: + - transformer-refs + - authoring-surface + - native + - ceiling +dependencies: [] +references: + - 'src/lib.rs:103' + - 'src/expr.rs:414' +documentation: + - 'doc-8 (Composition — {transform}(col) references)' + - doc-7 (Transformer execution model) +priority: low +type: feature +--- + +## Description + + +WHAT A USER HITS +You have a fitted transformer and you want to be explicit about which column feeds which input — the way you would in any function call: + + SELECT {scaler}(age => applicant_age, income => gross_income) AS scaled FROM __THIS__ + +You cannot. Today binding is POSITIONAL and name-matched through metadata, so the user has to know the order the transformer was fit in and pass columns in that order: + + SELECT {scaler}(applicant_age, gross_income) AS scaled FROM __THIS__ + +Get the order wrong and you have silently swapped two features — the scaler happily standardizes income using age's mean. Nothing errors. For a transformer fit on 10 columns, positional binding is a genuine footgun, and named arguments are the obvious ergonomic fix (project goal 1 is ergonomic authoring). + +WHY IT IS NOT A SMALL FIX (measured on datafusion 54.0.0, by Wren, 2026-07-23) +The blocker is a Python-BINDING gap, not a SQL-syntax gap: + + SELECT abs(x => age) FROM t -> "Function 'abs' does not support named arguments" + SELECT abs(x := age) FROM t -> same + +So the PARSER accepts both named-arg spellings and gets all the way to planning; the function then declines. Named-argument support is a per-function, Rust-side ScalarUDF capability. The Python bindings expose no way to opt in — datafusion.udf() has no named-arg parameter, and ScalarUDF exposes only [from_pycapsule, name, udf]. + +CONSEQUENCE for our model: what we call "input names" today are Arrow STRUCT FIELD NAMES carried by the named_struct we synthesize — metadata on the type, not SQL argument names. Both engines align on that metadata (native reads feature_names_in_ at src/lib.rs:103 and reorders at src/expr.rs:414, hard-erroring when it is absent). + +Building a real named-argument surface therefore needs EITHER a Rust-side ScalarUDF registered via the PyCapsule path, OR upstream python-datafusion bindings that expose the capability. It is not a Python-side change. + +WHY IT IS A DRAFT +No demand today — nobody has asked to author named args — and it needs design plus Rust/upstream work. Recorded primarily so nobody re-derives the finding. Becomes relevant if the ergonomic authoring goal ever wants named binding. + +NOT TO BE CONFUSED WITH the ndarray-fit positional-binding problem, which was solved Python-only inside TASK-3: synthesize the names from the call site onto a copy.copy() of the transformer (doc-8 clone contract preserved, fitted state shared, no deep copy, no src/*.rs edit). + +Context: doc-8 (composition — {transform}(col) references), doc-7 (transformer execution model). + + +## Acceptance Criteria + +- [ ] #1 Design decision recorded: Rust-side ScalarUDF via PyCapsule vs waiting on upstream python-datafusion bindings +- [ ] #2 {t}(name => col) authoring form works end-to-end with transform == infer parity (DataFusion oracle, decision-1) +- [ ] #3 Interaction with the existing feature_names_in_ / named_struct field-name mechanism is defined — named args must not silently diverge from struct field-name ordering + diff --git a/backlog/drafts/draft-12 - native-no-dispatch-for-struct...-and-make_array...-construction.md b/backlog/drafts/draft-12 - native-no-dispatch-for-struct...-and-make_array...-construction.md new file mode 100644 index 0000000..08401ae --- /dev/null +++ b/backlog/drafts/draft-12 - native-no-dispatch-for-struct...-and-make_array...-construction.md @@ -0,0 +1,71 @@ +--- +id: DRAFT-12 +title: 'native: no dispatch for struct(...) and make_array(...) construction' +status: Draft +assignee: [] +created_date: '2026-07-23 14:30' +updated_date: '2026-07-23 14:31' +labels: + - native + - parity + - sql-surface + - containers +dependencies: [] +references: + - src/expr_build.rs + - tests/test_diff_types.py +documentation: + - doc-9 (Rich type system and UNNEST — status and deferred edges) + - doc-1 (DataFusion function catalogue — parity oracle) +priority: medium +type: feature +--- + +## Description + + +WHAT A USER HITS +You copy a snippet from the DataFusion docs (or write what feels natural) to pack fields together: + + SELECT struct(bedrooms AS beds, baths AS baths) AS layout FROM __THIS__ + -- or + SELECT make_array(lat, lon) AS coords FROM __THIS__ + +You call fit() and transform() on your training frame. Both work — that path runs on DataFusion. +Then you deploy and call infer() for single-row serving. It fails: native doesn't recognize the function at all. + +So the feature works in your notebook and breaks in production. The user did nothing wrong — they used valid DataFusion SQL that the library accepted at fit time. + +The confusing part: for LISTS there is a spelling that works and one that doesn't, with nothing to tell them apart — + + SELECT [lat, lon] AS coords -- works on both engines today + SELECT make_array(lat, lon) -- works on transform(), fails on infer() + +For STRUCTS there is no working alternative spelling at all on native. + +ROOT CAUSE +native's convert_function (src/expr_build.rs) has no case for these container-construction forms, so it doesn't recognize them as functions. Codegen supports them and matches the DataFusion oracle; native does not: + + SELECT struct(a, b) AS s -> DataFusion/codegen: {c0: 1, c1: 2} (positional field names). Native: no dispatch. + SELECT struct(a AS x, b AS y) -> DataFusion/codegen: {x: 1, y: 2} (parses as exp.PropertyEQ). Native: no dispatch. + SELECT make_array(a, b) AS l -> DataFusion/codegen: [1, 2]. Native: no dispatch. + +NOT affected: the bracket literal `[a, b]` DOES reach native's list path and passes today (tests/test_diff_types.py::test_list_construct). The gap is specifically the FUNCTION-call spellings, not list/struct construction as a concept. + +SEVERITY vs DRAFT-13 +This is a missing capability that fails loudly, not a wrong answer — bad, but self-announcing. DRAFT-13 (mixed-numeric list widening) is the silent-wrong-value one and is rated higher. + +WHY IT MATTERS: native is the DEFAULT serving engine (decision-7), so a surface that works on the opt-in engine but not the default is backwards. Medium rather than High because the bracket literal gives users a working alternative for lists, and struct construction has no demonstrated demand yet. + +Surfaced by Ritchie's TASK-29 container work (2026-07-23), pinned by 3 strict xfail_on_native markers in tests/test_diff_types.py. Filed per the standing native-bug process (xfail-strict + ticket, never fix inline). + +DRAFT pending AmirHossein's review of scope/priority. + + +## Acceptance Criteria + +- [ ] #1 native convert_function (src/expr_build.rs) dispatches struct(...) positional form, naming fields c0/c1/... exactly as DataFusion does +- [ ] #2 native dispatches struct(a AS x, ...) named form (exp.PropertyEQ) with explicit field names +- [ ] #3 native dispatches make_array(...) to the same construction path the bracket literal [a, b] already uses +- [ ] #4 The 3 xfail_on_native markers in tests/test_diff_types.py (test_struct_construct_positional, test_struct_construct_named, test_make_array_construct) are removed and the tests pass on both engines against the DataFusion oracle + diff --git a/backlog/drafts/draft-13 - BUG-native-mixed-int-float-list-elements-are-not-widened-silent-value-divergence.md b/backlog/drafts/draft-13 - BUG-native-mixed-int-float-list-elements-are-not-widened-silent-value-divergence.md new file mode 100644 index 0000000..3aa09fd --- /dev/null +++ b/backlog/drafts/draft-13 - BUG-native-mixed-int-float-list-elements-are-not-widened-silent-value-divergence.md @@ -0,0 +1,66 @@ +--- +id: DRAFT-13 +title: >- + BUG native: mixed int/float list elements are not widened (silent value + divergence) +status: Draft +assignee: [] +created_date: '2026-07-23 14:30' +updated_date: '2026-07-23 14:30' +labels: + - native + - parity + - bug + - containers +dependencies: [] +references: + - src/types.rs + - tests/test_diff_types.py +documentation: + - doc-9 (Rich type system and UNNEST — status and deferred edges) + - doc-1 (DataFusion function catalogue — parity oracle) +priority: high +type: bug +--- + +## Description + + +WHAT A USER HITS +You bundle two features into one list column — a count and a ratio: + + SELECT [bedrooms, price_per_sqft] AS dims FROM __THIS__ + +`bedrooms` is an int column, `price_per_sqft` is a float column. + +At training time you call transform() over the DataFrame and get: [3.0, 12.5] +At serving time you call infer() on a single row and get: [3, 12.5] + +No error. No warning. The int element silently stays an int on the serving path and becomes a float on the training path. Anything downstream that is dtype-sensitive — a numpy cast, a model expecting float64, a serialized feature vector compared against a stored schema — now sees different data in training than in production. This is the failure mode the whole differential harness exists to prevent: you only find it when the model misbehaves in prod, and nothing in the stack points at the list literal. + +Realistic triggers: any list feature mixing an integer column with a float column — [count, rate], [year_built, lot_ratio], [n_visits, avg_spend]. Users won't think of these as "mixed-type" — they're just two numbers. + +ROOT CAUSE (measured, not inferred) +native's unify_list_element_types (src/types.rs) is EXACT-EQUALITY-ONLY, so it refuses to widen mixed numeric elements to a common type. DataFusion (the oracle) and codegen both widen to list. + + SELECT [x, y] AS l FROM t -- x int = 1, y float = 2.5 + DataFusion (oracle) + codegen: [1.0, 2.5] (widened to list) + native: [1, 2.5] (un-widened) + +WHY THIS IS THE SERIOUS ONE +Unlike the struct/make_array dispatch gap (DRAFT-12), this is NOT a missing capability that errors out — native accepts the query and returns a DIFFERENT VALUE. Silent cross-engine divergence on the DEFAULT serving engine (decision-7). Proposed High on that basis. + +SAME BUG CLASS as the one Ritchie just fixed on the codegen side in infer_type's ListExpr arm (unifying element bases via _common_base, like COALESCE). The native fix mirrors it: compute a common numeric base instead of demanding exact equality. + +Surfaced by Ritchie's TASK-29 container work (2026-07-23), pinned by a strict xfail_on_native in tests/test_diff_types.py::test_list_construct_mixed_numeric_widens. Filed per the standing native-bug process — Ritchie correctly flagged rather than fixing inline. DataFusion is the oracle (decision-1). + +DRAFT pending AmirHossein's review of scope/priority. + + +## Acceptance Criteria + +- [ ] #1 native unify_list_element_types (src/types.rs) computes a common numeric base for mixed elements instead of exact equality, mirroring codegen's _common_base/COALESCE logic +- [ ] #2 SELECT [x, y] with int + float elements yields [1.0, 2.5] on native, matching the DataFusion oracle +- [ ] #3 The xfail_on_native marker on tests/test_diff_types.py::test_list_construct_mixed_numeric_widens is removed and the test passes on both engines +- [ ] #4 Root-cause sweep: confirm no other native type-unification site is exact-equality-only where DataFusion widens (the same class of silent divergence elsewhere) + diff --git a/backlog/drafts/draft-7 - Native-swap-native-per-transformer-Tier-0.md b/backlog/drafts/draft-7 - Native-swap-native-per-transformer-Tier-0.md index a6c3c14..0a48e3b 100644 --- a/backlog/drafts/draft-7 - Native-swap-native-per-transformer-Tier-0.md +++ b/backlog/drafts/draft-7 - Native-swap-native-per-transformer-Tier-0.md @@ -4,7 +4,7 @@ title: 'Native-swap: native per-transformer (Tier 0)' status: Draft assignee: [] created_date: '2026-07-18 13:44' -updated_date: '2026-07-23 01:01' +updated_date: '2026-07-23 14:31' labels: - sklearn - native-swap @@ -20,7 +20,33 @@ ordinal: 5000 ## Description -Swap each fallback-backed transformer to a native engine impl, diffed against the fallback oracle, in tier order. Most Tier 0/1 map onto shipped machinery (window aggs, PARTITION BY, LookupJoin, struct/UNNEST). Full plan: doc-2 (sklearn transformer plan). +WHAT A USER HITS +You build a normal preprocessing pipeline: + + scaler = StandardScaler().fit(train[["age", "income"]]) + t = SQLTransform("SELECT {scaler}(age, income) AS scaled FROM __THIS__").fit(train) + t.infer({"age": 41, "income": 82000}) # single-row serving + +Today this works, but every single call crosses into Python and runs sklearn's own .transform() — the opaque path. So the per-row serving latency is dominated by the Python/FFI boundary, which is exactly the cost the project exists to remove (see the boundary-bound finding). At n=1 you pay a full sklearn callout to compute (x - mean) / scale. + +The user-visible symptom is not wrongness, it's speed: serving is far slower than it should be for arithmetic this trivial, and the transformer can't be served from the non-Python runtimes at all (Go/Java/WASM) because it needs a Python interpreter in the loop. + +WHAT THIS TICKET DOES +Swap each Tier 0 transformer from the opaque sklearn callout to a NATIVE implementation expressed in the engine's own SQL/expression surface, so it fuses into the single per-row expression instead of calling out. + +The insight (doc-7): these transformers are already expressible as window-aggregate SQL we compile today. + StandardScaler == (x - AVG(x) OVER ()) / STDDEV(x) OVER () + SimpleImputer == COALESCE(x, ) + OrdinalEncoder == a code-map lookup + OneHotEncoder == one CAST(x = 'cat_i' AS INT) per learned category +So most of this is macro definitions over shipped machinery, not new engine code. + +Each swap is diffed against the sklearn fallback as the oracle — the point is that switching a user from opaque to native must not change a single output value. + +WHY IT IS A DRAFT +Needs design work before it is a dispatchable task (AmirHossein, 2026-07-23). Open questions: the state-shape contract per transformer (scalar vs list vs code-map vs per-group table), where the macro definitions live, how unknown-category handling is expressed, and whether OneHotEncoder's multi-output fan-out belongs here or in the feature-output work. Re-scope and re-promote to a TASK before dispatching. + +Full plan context: doc-2 (sklearn transformer implementation plan), doc-7 (transformer execution model). ## Acceptance Criteria diff --git a/backlog/drafts/draft-8 - ColumnTransformer-assembly-surface.md b/backlog/drafts/draft-8 - ColumnTransformer-assembly-surface.md index d9402b7..d4691cc 100644 --- a/backlog/drafts/draft-8 - ColumnTransformer-assembly-surface.md +++ b/backlog/drafts/draft-8 - ColumnTransformer-assembly-surface.md @@ -4,7 +4,7 @@ title: ColumnTransformer assembly surface status: Draft assignee: [] created_date: '2026-07-18 13:44' -updated_date: '2026-07-23 01:02' +updated_date: '2026-07-23 14:31' labels: - sklearn - fallback @@ -21,7 +21,24 @@ ordinal: 6000 ## Description -Column routing + horizontal concat; the end-to-end assembly-parity oracle: bit-identical width + column order + values vs stock ColumnTransformer.transform(). Must include a multi-output transformer. Detail in doc-2 (sklearn strategy) / doc-10 (feature-output). +WHAT A USER HITS +A real preprocessing job is never one transformer — it is different treatment per column group: + + numeric = ["age", "income", "lot_area"] -> impute median, then scale + categorical = ["neighborhood", "house_style"] -> one-hot + ordinal = ["quality"] -> Ex/Gd/TA -> 5/4/3 + +In sklearn you express that with a ColumnTransformer and get one matrix out, with a known column order. Today in this library there is no equivalent assembly surface: the user writes per-column SQL and is left to stitch the outputs together themselves, and the column ORDER of the assembled result is whatever falls out — which is the one thing a model cannot tolerate. Feed the model columns in a different order than it was trained on and you get silently wrong predictions, no error. + +The multi-output case is where it really bites: one-hot on a 25-category column contributes 25 columns whose width depends on what was seen at fit time. Users need the assembled width and order to be pinned by the fitted artifact, not recomputed per batch. + +WHAT THIS TICKET DOES +Column routing + horizontal concat, with the parity bar being bit-identical output vs a stock sklearn ColumnTransformer.transform() — same width, same column order, same values. Must include at least one multi-output transformer so variable-width concat and feature-name expansion are actually exercised. Feature names/provenance must survive routing and concat in order, so users can map a column index back to what produced it. + +WHY IT IS A DRAFT +Needs design work before it is dispatchable (AmirHossein, 2026-07-23), and it sits on top of the native-transformer draft (DRAFT-7). Open questions: what the user-facing authoring surface for column routing even looks like in SQL (this is the goal-1 ergonomics question, not just an implementation), how remainder/passthrough columns behave, and how this relates to the type-directed assembler (TASK-16) which solves the adjacent dense+sparse problem. Re-scope before promoting. + +Context: doc-2 (sklearn transformer implementation plan), doc-10 (feature-output model). ## Acceptance Criteria diff --git a/backlog/drafts/draft-9 - Feature-output-sparse-COO-struct-column-CSR-materializer-width-invariant-assert.md b/backlog/drafts/draft-9 - Feature-output-sparse-COO-struct-column-CSR-materializer-width-invariant-assert.md index 1fb242b..f0724a6 100644 --- a/backlog/drafts/draft-9 - Feature-output-sparse-COO-struct-column-CSR-materializer-width-invariant-assert.md +++ b/backlog/drafts/draft-9 - Feature-output-sparse-COO-struct-column-CSR-materializer-width-invariant-assert.md @@ -6,7 +6,7 @@ title: >- status: Draft assignee: [] created_date: '2026-07-18 15:52' -updated_date: '2026-07-23 01:01' +updated_date: '2026-07-23 14:31' labels: - feature-output milestone: m-1 @@ -20,7 +20,24 @@ ordinal: 14000 ## Description -Per-row sparse feature = Arrow struct, values: list> (COO). 1:1 with rows, so it mixes with dense scalar columns in ONE SELECT (sparseness lives in the cell, not row cardinality). Materialize to scipy CSR: concat rows' arrays -> data/indices, per-row lengths -> indptr. Dimension N + unseen-key handling come from the FITTED transform (self-contained artifact), NOT the cell/type. N pins shape=(n,N); a width-invariant assert catches silent batch-width drift (a model-misalignment bug). +WHAT A USER HITS +You one-hot encode a high-cardinality column — say `neighborhood` with 800 distinct values, or a product SKU with 50k: + + SELECT {ohe}(neighborhood) AS nbhd FROM __THIS__ + +Dense output means every row carries 800 (or 50,000) float64 columns, nearly all zeros. That is the difference between a feature matrix that fits in memory and one that does not, and between a fast model fit and an unusable one. sklearn's own OneHotEncoder returns a scipy sparse matrix by default for exactly this reason — users coming from sklearn will expect sparse and be surprised when a 50k-category encode tries to materialize dense. + +Second, subtler failure this ticket guards against: BATCH-WIDTH DRIFT. If the encoded width is derived from the data in each batch rather than pinned by the fitted artifact, a serving batch that happens not to contain some category produces a NARROWER matrix. The model then receives misaligned columns — feature 300 means something different than it did at training. No error, just wrong predictions. That is why the width-invariant assert is an acceptance criterion and not a nice-to-have. + +WHAT THIS TICKET DOES +Represent a per-row sparse feature as an Arrow struct, values: list> (COO). Because it stays 1:1 with rows, it mixes with ordinary dense scalar columns in ONE SELECT — the sparseness lives in the cell, not in the row cardinality, so nothing has to explode. Materialize to scipy CSR by concatenating the per-row arrays into data/indices and using per-row lengths as indptr. + +Dimension N and unknown-category handling come from the FITTED transform (a self-contained artifact), NOT from the cell or the type. N pins shape=(n, N), and a width-invariant assert fails loudly if a batch ever disagrees. + +WHY IT IS A DRAFT +Super-low priority AND needs design work (AmirHossein, 2026-07-23). No demand today. Blocks TASK-16 (type-directed assembler) and DRAFT-10 (tfidf), both of which need the sparse cell type to exist first — so if either of those ever becomes wanted, this gets scoped first. + +Context: doc-10 (feature-output model — records, dense, sparse). ## Acceptance Criteria diff --git a/backlog/tasks/task-15 - Feature-output-scalar-one-hot-as-join-to-domain.md b/backlog/tasks/task-15 - Feature-output-scalar-one-hot-as-join-to-domain.md index 1c25928..3eb8a8b 100644 --- a/backlog/tasks/task-15 - Feature-output-scalar-one-hot-as-join-to-domain.md +++ b/backlog/tasks/task-15 - Feature-output-scalar-one-hot-as-join-to-domain.md @@ -4,7 +4,7 @@ title: 'Feature output: scalar one-hot as join-to-domain' status: To Do assignee: [] created_date: '2026-07-18 15:52' -updated_date: '2026-07-23 00:52' +updated_date: '2026-07-23 14:32' labels: - feature-output milestone: m-1 @@ -18,7 +18,28 @@ ordinal: 15000 ## Description -Scalar one-hot via a join to the fitted domain table (composable, no explode). Rides the existing static_tables/lookup mechanism. Part of the feature-output model. +WHAT A USER HITS +You want to one-hot a categorical column — the single most common preprocessing step there is: + + SELECT {ohe}(house_style) AS style FROM __THIS__ + +Today this goes through the opaque sklearn callout: a Python boundary crossing per serving call, and a step that cannot run on the Go/Java/WASM runtimes. For an operation that is conceptually just "is this value equal to each known category," that is a lot of machinery. + +The user-visible payoff of this ticket is that a one-hot stops being a black box and becomes part of the fused per-row expression — so single-row serving gets fast, and the pipeline stays portable. + +WHAT THIS TICKET DOES +Compile a SCALAR one-hot into a JOIN against the fitted category domain table, riding the existing static_tables / lookup mechanism the engine already has (the same machinery TargetEncoder-style PARTITION BY work uses). + +Why a join and not an explode: this is the FIXED-FANOUT case. The category list is frozen at fit time, so the output width is known and the operation stays 1:1 with rows — no row-cardinality change, nothing to explode, and it composes with everything else in the same SELECT. That is what makes it cheap and why it is separable from the hard cases. + +Unknown categories at serving (a `house_style` never seen during fit) resolve through the join as a miss, which is the natural place to express sklearn's handle_unknown behavior. + +SCOPE BOUNDARY +This is deliberately the EASY half of one-hot. The variable-expansion cases — tfidf, array multi-hot — need an explode and are parked separately (DRAFT-10). The sparse representation for high-cardinality columns is also separate (DRAFT-9). This ticket is the composable, fast-path slice that needs neither. + +Independent: nothing blocks it, and it does not depend on the drafted spine. + +Context: doc-10 (feature-output model — records, dense, sparse). ## Acceptance Criteria diff --git a/backlog/tasks/task-16 - Feature-output-type-directed-assembler-dense-sparse-in-one-SELECT.md b/backlog/tasks/task-16 - Feature-output-type-directed-assembler-dense-sparse-in-one-SELECT.md index c574b15..6b96fad 100644 --- a/backlog/tasks/task-16 - Feature-output-type-directed-assembler-dense-sparse-in-one-SELECT.md +++ b/backlog/tasks/task-16 - Feature-output-type-directed-assembler-dense-sparse-in-one-SELECT.md @@ -4,7 +4,7 @@ title: 'Feature output: type-directed assembler (dense + sparse in one SELECT)' status: To Do assignee: [] created_date: '2026-07-18 15:52' -updated_date: '2026-07-23 01:02' +updated_date: '2026-07-23 14:32' labels: - feature-output milestone: m-1 @@ -19,7 +19,26 @@ ordinal: 16000 ## Description -One SELECT compiles to dense(+)sparse via type-directed decompose+assemble: split the projection by output type -> route -> hstack -> materialize. sklearn ColumnTransformer is the INTERNAL assembly target (not a user API) -- users write SQL, we compile the ColumnTransformer. Depends on the sparse-column (TASK-14) + dense-matrix (TASK-13) tasks. +WHAT A USER HITS +A realistic feature set mixes shapes. Some columns are plain numbers, one is a high-cardinality one-hot: + + SELECT + (age - AVG(age) OVER ()) / STDDEV(age) OVER () AS age_z, -- dense scalar + lot_area / AVG(lot_area) OVER () AS lot_norm, -- dense scalar + {ohe}(neighborhood) AS nbhd -- 800-wide, sparse + FROM __THIS__ + +The user wrote ONE query and expects ONE feature matrix. But the pieces have genuinely different physical representations: two float64 scalars and an 800-wide mostly-zero block. Forcing everything dense blows up memory; forcing everything sparse wastes it on the scalars. Today the user has to know which is which and assemble by hand, which means they also own the column-ordering problem — and getting that wrong silently misfeeds the model. + +WHAT THIS TICKET DOES +Make one SELECT compile to a correctly-assembled mixed dense+sparse feature set, automatically, by TYPE-DIRECTED decompose and assemble: split the projection by output type, route each part to the right representation, hstack, materialize. + +Key design point: sklearn's ColumnTransformer is used as the INTERNAL assembly target — the thing we compile TO for hstack/densify — not as a user-facing API. The user writes SQL; we compile the ColumnTransformer for them. That inversion is the whole point (project goal 1): the SQL is the authoring surface, sklearn is an implementation detail. + +BLOCKED +Depends on DRAFT-9 (sparse COO struct column) — there is no sparse representation to route to until that exists, and DRAFT-9 is currently parked as an unscoped draft. The dense half (TASK-13) has already landed. So this cannot finish until DRAFT-9 is scoped, promoted, and done. + +Context: doc-10 (feature-output model — records, dense, sparse). ## Acceptance Criteria diff --git a/backlog/tasks/task-2 - Opaque-transform-Part-1-review-follow-ups.md b/backlog/tasks/task-2 - Opaque-transform-Part-1-review-follow-ups.md index 7629224..6057cb2 100644 --- a/backlog/tasks/task-2 - Opaque-transform-Part-1-review-follow-ups.md +++ b/backlog/tasks/task-2 - Opaque-transform-Part-1-review-follow-ups.md @@ -1,10 +1,11 @@ --- id: TASK-2 title: Opaque transform Part-1 review follow-ups -status: To Do -assignee: [] +status: Done +assignee: + - Wren created_date: '2026-07-18 13:44' -updated_date: '2026-07-23 00:52' +updated_date: '2026-07-23 13:09' labels: - rust - parity @@ -25,7 +26,23 @@ Follow-ups from the Part-1 (engine capability) review. Split rationale + deferre ## Acceptance Criteria -- [ ] #1 Out-of-projection transformer calls: build-time guard/consistent resolve so Rust and DF agree -- [ ] #2 Add a single-field 1-in/1-out transformer differential parity case (oracle y[:,i] assumes 2-D) -- [ ] #3 Enforce or reconcile out_schema vs the transform natural output dtype +- [x] #1 Out-of-projection transformer calls: build-time guard/consistent resolve so Rust and DF agree +- [x] #2 Add a single-field 1-in/1-out transformer differential parity case (oracle y[:,i] assumes 2-D) +- [x] #3 Enforce or reconcile out_schema vs the transform natural output dtype + +## Comments + + +author: Iris (PM) +created: 2026-07-23 04:49 +--- +Dispatched to Wren (2026-07-23), AmirHossein's explicit go. Opaque-transform Part-1 engine-correctness hardening (3 ACs). Reminded Wren to use the superpowers skills (brainstorm → TDD with regression-catching tests → verification-before-completion, own worktree). Parity bugs → xfail-strict + ticket request, no inline fixes. +--- + +author: Iris (PM) +created: 2026-07-23 13:09 +--- +Verified against the diff (0cda202 + 2c9be9b), not the report. AC#1: _sql.py::require_in_projection raises a build-time ValueError; the reachable clause set was measured (QUALIFY/DISTINCT ON/CLUSTER BY/SORT BY/WINDOW survive parse_and_validate) rather than assumed — engines demonstrably disagreed before (fit accepted, transform raised, infer silently ignored). AC#2: test_single_field_one_in_one_out_parity, mutation-checked (a squeeze() breaks only this test, all 8 existing 2-in/2-out cases stay green). AC#3: _transformer_udf.py::check_out_schema_natural ENFORCES rather than reconciles — correct per decision-1, since reconciling means silent coercion. Scope extension to _compose.py ACCEPTED: the ref path had the identical hole, so guarding only the callout path would have left a sibling caller broken — root-cause fix, not creep. Wren reports 527 passed / 9 skipped / 4 xfailed; I verified the diff and tests, did not re-run the suite. No native parity bug found, so no ticket request. +--- + diff --git a/backlog/tasks/task-29 - codegen-implement-deferred-SQL-surface-containers-UNNEST-unary-minus.md b/backlog/tasks/task-29 - codegen-implement-deferred-SQL-surface-containers-UNNEST-unary-minus.md index 594ed6c..8ce24bc 100644 --- a/backlog/tasks/task-29 - codegen-implement-deferred-SQL-surface-containers-UNNEST-unary-minus.md +++ b/backlog/tasks/task-29 - codegen-implement-deferred-SQL-surface-containers-UNNEST-unary-minus.md @@ -5,7 +5,7 @@ status: In Progress assignee: - Ritchie created_date: '2026-07-18 20:14' -updated_date: '2026-07-23 00:53' +updated_date: '2026-07-23 14:32' labels: - codegen - feature @@ -20,7 +20,31 @@ ordinal: 29000 ## Description -The codegen engine defers SQL surface it doesn't implement yet -- shows as 16 skips on the codegen backend in the differential suite (native + DataFusion oracle cover and pass all of it). NOT bugs: codegen raises UnsupportedInCodegen and skips LOUDLY. Deferred surface (from the 16 skips): containers -- struct/list column projection, struct field access, struct/list construction (named_struct / array), struct/list comparison, UNNEST (~13 of 16); unary minus on a non-literal (-a); || string-concat operator. Enumerated in the codegen spec 'Deferred' section as intended fast-follows. BLOCKED-ON-FRAMING: gated on the codegen-engine framing decision (maintained/default vs opt-in) -- the pluggable-backend design paused mid-brainstorm + the two-engine ratification (decision-7) that's parked / not-short-term. If codegen becomes default -> real feature-completeness, priority rises; if opt-in -> stays low. Not a bug, not blocking anything today; tests/test_codegen_coverage.py asserts codegen skips ONLY this exact set, guarding against silent drift. +WHAT A USER HITS +You opted into the codegen engine (native is the default per decision-7, codegen is the opt-in alternative). Your SQL uses container types or a couple of ordinary operators: + + SELECT named_struct('lat', latitude, 'lon', longitude) AS coords FROM __THIS__ + SELECT first_name || ' ' || last_name AS full_name FROM __THIS__ + SELECT -balance AS debit FROM __THIS__ + +On native these all work. On codegen you get UnsupportedInCodegen. So switching engines changes which SQL is legal — the same query that ran yesterday stops running when you flip the flag. + +The important part: this FAILS LOUDLY. Codegen raises rather than silently computing something different. These are not bugs and never produced a wrong answer — they are honest "not implemented yet" refusals, and tests/test_codegen_coverage.py pins the exact set so nothing drifts in silently. + +WHAT THIS TICKET DOES +Close the deferred surface so the opt-in engine stops being a downgrade. The gap was 16 skips in the differential suite (native + the DataFusion oracle cover and pass all of it), inventoried by QA on 2026-07-19: + + Container surface (~13 of 16) — struct/list column projection, struct field access, struct/list construction (named_struct / array), struct/list comparison, UNNEST + Operators (2) — unary minus on a non-literal (-a), and the || string-concat operator + +Each item landing shrinks the skip set and updates test_codegen_coverage.py, which currently pins the exact list. + +PRIORITY CONTEXT +Low, and deliberately so. decision-7 ruled native default / codegen opt-in, which settled the precondition (AC#3) — this is a fast-follow for people who opt into codegen, not milestone work. If codegen were ever promoted toward default, this becomes real feature-completeness and the priority rises. + +Ordering note from PM: the two operator defers are cheap scalar work and were never truly framing-gated (same category as CASE, which shipped on codegen regardless), so they went first; the container surface is the real body of the ticket. + +Context: doc-9 (rich type system and UNNEST), doc-8 (composition — deferred slices). ## Acceptance Criteria diff --git a/backlog/tasks/task-3 - Transformer-refs-Part-2-authoring-surface-review-follow-ups.md b/backlog/tasks/task-3 - Transformer-refs-Part-2-authoring-surface-review-follow-ups.md index 1937be5..35c26ba 100644 --- a/backlog/tasks/task-3 - Transformer-refs-Part-2-authoring-surface-review-follow-ups.md +++ b/backlog/tasks/task-3 - Transformer-refs-Part-2-authoring-surface-review-follow-ups.md @@ -1,10 +1,11 @@ --- id: TASK-3 title: Transformer-refs (Part-2 authoring surface) review follow-ups -status: To Do -assignee: [] +status: In Progress +assignee: + - Wren created_date: '2026-07-18 13:44' -updated_date: '2026-07-23 00:52' +updated_date: '2026-07-23 14:32' labels: - python - transformer-refs @@ -20,7 +21,28 @@ ordinal: 3000 ## Description -Follow-ups from the whole-branch review (ready-to-merge, no Critical/Important). Split rationale + deferred direction: decision-3 (opaque-transform split, Part 2). +WHAT A USER HITS +These are the papercuts on the shipped {transformer}(col) authoring surface — each one is a real thing a user runs into: + +1. SILENT DOUBLE WORK. A user fits a transform with a single transformer ref and it runs sklearn's .transform() TWICE at fit time — once to derive the output schema, once for real. On an expensive transformer over a large training frame, they pay double for no reason and nothing tells them why fit is slow. + +2. CRYPTIC ERRORS ON PREDICTABLE MISTAKES. Two mistakes users will absolutely make: + SELECT AVG({scaler}(age)) OVER () FROM __THIS__ -- aggregating over a transformer's output + SELECT {scaler}(age) FROM __THIS__ -- where scaler was never .fit() + Today these surface as whatever the engine happens to throw deep in the stack, not as "you can't aggregate over a transformer output" / "this transformer isn't fitted." + +3. THE feature_names_in_ FOOTGUN. This is the nastiest one. A transformer-ref needs feature_names_in_ to bind columns. But sklearn only sets it when the transformer was fit with NAMED columns — i.e. a DataFrame. Fit a OneHotEncoder on a numpy array and it silently has no feature_names_in_, so the ref fails at a confusing place. The user's mistake happened much earlier (in their sklearn code, not ours) and nothing connects the two. Options: document it loudly, or accept an explicit names argument so the footgun stops existing. + +4. STRUCT OUTPUT IS A SURPRISE AT THE SKLEARN HANDOFF. Transformer-ref output is ONE Arrow struct column, not N flat columns. A user who wants to hand the result to a model has to flatten it, and nothing in the README shows that step — so their first attempt to feed our output into .fit() fails on shape. + +WHAT THIS TICKET DOES +Fix 1 by reusing the _derive_schemas probe and skipping _materialize when nothing consumes the output. Fix 2 with friendly pre-check errors. Fix 3 by documenting the contract and considering an explicit names arg (API change — brainstorm with PM before building). Fix 4 with a README note/example showing the flatten step; this is near-term DX that the feature-output dense/assembler work (TASK-16) supersedes long-term. + +Plus the negative/contract tests that pin all of it, and a regression test for transformer + PARTITION BY input-col. + +Origin: follow-ups from the whole-branch review (which was ready-to-merge, no Critical/Important findings). Split rationale: decision-3 (opaque-transform split, Part 2). + +Context: doc-8 (composition — {transform}(col) references), doc-7 (transformer execution model). ## Acceptance Criteria @@ -32,3 +54,19 @@ Follow-ups from the whole-branch review (ready-to-merge, no Critical/Important). - [ ] #5 Document the feature_names_in_ contract: transformer-ref needs it; OneHotEncoder sets it only when fit with named columns (a DataFrame) -- else hand-assign obj.feature_names_in_ = names. Consider accepting an explicit names arg to remove the footgun. - [ ] #6 README note/example: transformer-ref output is a single Arrow struct column; show the flatten step for the sklearn handoff (near-term DX; the feature-output model's dense output / assembler task TASK-16 supersedes it long-term). + +## Comments + + +author: Iris (PM) +created: 2026-07-23 04:54 +--- +PRE-AUTHORIZED (2026-07-23): AmirHossein green-lit dispatching TASK-3 to Wren after TASK-2 completes. PM dispatches on TASK-2 done (verify diff, close 2, then hand Wren TASK-3 with the superpowers reminder). No further approval needed. +--- + +author: Iris (PM) +created: 2026-07-23 13:09 +--- +Dispatched to Wren (2026-07-23) per the standing pre-authorization, on TASK-2 completion. 6 ACs — mostly DX/guardrails on the shipped authoring surface. Note AC#6 is a README/docs item and AC#4 is self-described low value; Wren to flag if any AC looks not-worth-it rather than padding it out. +--- + diff --git "a/backlog/tasks/task-34 - codegen-transformer-support-resolve_transformers-\342\206\222-Expr-Transform-native-only-today.md" "b/backlog/tasks/task-34 - codegen-transformer-support-resolve_transformers-\342\206\222-Expr-Transform-native-only-today.md" index 1d68004..d8ba3fb 100644 --- "a/backlog/tasks/task-34 - codegen-transformer-support-resolve_transformers-\342\206\222-Expr-Transform-native-only-today.md" +++ "b/backlog/tasks/task-34 - codegen-transformer-support-resolve_transformers-\342\206\222-Expr-Transform-native-only-today.md" @@ -7,7 +7,7 @@ status: To Do assignee: - Ritchie created_date: '2026-07-19 16:08' -updated_date: '2026-07-23 04:41' +updated_date: '2026-07-23 14:32' labels: - codegen - transformer @@ -28,7 +28,26 @@ ordinal: 34000 ## Description -Codegen has NO transformer support — transformer refs are a native-only feature. SQLTransform.infer/infer_batch runs on the native InferFn, and the resolve_transformers rewrite (SQL call → Expr::Transform) lives only in the native path; codegen raises UnsupportedInCodegen for it (verified by TASK-31: the CASE-branch transformer test asserts codegen DEFERS loudly rather than computing). This is a whole feature class codegen lacks, distinct from TASK-29's deferred expr/container surface (which explicitly excludes transformers). Placeholder ticket so the gap is tracked and this 'do we have a ticket?' question stops recurring — NOT active work. BLOCKED-ON-FRAMING: only worth building if codegen becomes a maintained/default serving path (decision-7, the two-engine framing question, still open/parked). Bigger lift than TASK-29 — it's the transformer-resolution machinery, not a few expr shapes. Until decision-7 lands, native carries transformers and codegen defers them loudly (correct, not a bug). +WHAT A USER HITS +You wrote a transform that uses a fitted sklearn transformer — the library's headline feature: + + SELECT {scaler}(age, income) AS scaled FROM __THIS__ + +It works. Then you opt into the codegen engine for serving and it refuses: UnsupportedInCodegen. Not a subset of the syntax — the ENTIRE transformer-ref feature is unavailable. A user who builds their pipeline around transformer refs simply cannot use codegen at all. + +Like TASK-29's gaps, this fails loudly rather than silently miscomputing (verified by TASK-31: the CASE-branch transformer test asserts codegen DEFERS rather than computing a wrong answer). Correct behavior, not a bug — but a whole feature class the opt-in engine lacks. + +WHAT THIS TICKET DOES +Give codegen its own transformer resolution. Today the resolve_transformers rewrite (SQL call -> Expr::Transform) lives only in the native path — SQLTransform.infer/infer_batch runs on the native InferFn, and codegen has no equivalent. This would build that machinery on the codegen side, with transform == infer parity against the DataFusion/native oracle, and flip TASK-31's "defers loudly" assertion into a real parity case. + +Distinct from TASK-29, which covers deferred EXPRESSION and CONTAINER surface and explicitly excludes transformers. This is the bigger lift of the two — transformer-resolution machinery, not a handful of expression shapes. + +PRIORITY CONTEXT +Low. decision-7 ruled native default / codegen opt-in, which satisfied the precondition (AC#1) — so this is decided-low rather than blocked-on-an-open-question. It is a tracked placeholder so the recurring "do we have a ticket for this?" question stops. Only becomes real work if codegen is ever promoted toward being a maintained/default serving path. + +Depends on TASK-29. Pre-authorized by AmirHossein to dispatch to Ritchie automatically once TASK-29 lands. + +Context: doc-7 (transformer execution model), doc-8 (composition — {transform}(col) references). ## Acceptance Criteria