backlog: user-facing ticket descriptions + drafts for pinned native container gaps

backlog/drafts/draft-10 - Feature-output-tfidf-array-multi-hot-opaque-needs-explode.md

@@ -4,7 +4,7 @@ title: 'Feature output: tfidf / array multi-hot (opaque, needs explode)'
 status: Draft
 assignee: []
 created_date: '2026-07-18 15:52'
-updated_date: '2026-07-23 04:46'
+updated_date: '2026-07-23 14:31'
 labels:
   - feature-output
 milestone: m-1
@@ -20,7 +20,27 @@ ordinal: 17000
 ## Description
 
 <!-- SECTION:DESCRIPTION:BEGIN -->
-tfidf and array multi-hot need variable-expansion (explode), so they are the OPAQUE ones -- map onto the shipped opaque-transform mechanism (decision-3), corpus-gated. Distinct from the sparse-column / scalar-one-hot tasks (TASK-14/15) which are fixed-fanout and composable. Depends on opaque-transform (Part 1/2 shipped).
+WHAT A USER HITS
+You have a free-text column — a listing description, a support ticket body, a product title — and you want it as features:
+
+    tfidf = TfidfVectorizer().fit(train["description"])
+    SELECT {tfidf}(description) AS text_feats FROM __THIS__
+
+One input row produces a wide sparse vector over the learned vocabulary (thousands of terms). Same for array multi-hot: a `tags` column holding ["garage", "pool", "corner_lot"] expands to one indicator per known tag.
+
+This is the shape the current engine cannot express. Every other transformer we compile is FIXED FAN-OUT — one input value maps to a known number of output columns decided at fit time, and the expression stays per-row. tfidf needs variable expansion (an explode), so it does not fit the scalar/window-agg surface the fast path is built on.
+
+Practical consequence for the user: text features are the one preprocessing step that will NOT fuse into the single native serving expression. They run through the opaque callout, which means a Python interpreter in the loop and no Go/Java/WASM serving for a pipeline containing them.
+
+WHAT THIS TICKET DOES
+Route tfidf and array multi-hot onto the shipped opaque-transform mechanism (decision-3), corpus-gated. Deliberately NOT trying to make them native — accepting the opaque path is the correct trade for a fundamentally variable-width operation.
+
+Distinct from DRAFT-9 (sparse COO column) and TASK-15 (scalar one-hot), which are fixed-fanout and composable — those stay on the fast path. This is the escape hatch for the ones that genuinely cannot.
+
+WHY IT IS A DRAFT
+Very low priority (AmirHossein, 2026-07-23) and depends on DRAFT-9: tfidf output is sparse, so it needs the sparse-column materializer path to exist before it is buildable. Re-promote only if text features get actual demand.
+
+Context: doc-10 (feature-output model), doc-9 (rich type system and UNNEST).
 <!-- SECTION:DESCRIPTION:END -->
 
 ## Acceptance Criteria

backlog/drafts/draft-11 - SQL-named-argument-authoring-surface-for-transformer-calls-needs-Rust-ScalarUDF-or-upstream-bindings.md

@@ -0,0 +1,65 @@
+---
+id: DRAFT-11
+title: >-
+  SQL named-argument authoring surface for transformer calls (needs Rust
+  ScalarUDF or upstream bindings)
+status: Draft
+assignee: []
+created_date: '2026-07-23 14:11'
+updated_date: '2026-07-23 14:33'
+labels:
+  - transformer-refs
+  - authoring-surface
+  - native
+  - ceiling
+dependencies: []
+references:
+  - 'src/lib.rs:103'
+  - 'src/expr.rs:414'
+documentation:
+  - 'doc-8 (Composition — {transform}(col) references)'
+  - doc-7 (Transformer execution model)
+priority: low
+type: feature
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+WHAT A USER HITS
+You have a fitted transformer and you want to be explicit about which column feeds which input — the way you would in any function call:
+
+    SELECT {scaler}(age => applicant_age, income => gross_income) AS scaled FROM __THIS__
+
+You cannot. Today binding is POSITIONAL and name-matched through metadata, so the user has to know the order the transformer was fit in and pass columns in that order:
+
+    SELECT {scaler}(applicant_age, gross_income) AS scaled FROM __THIS__
+
+Get the order wrong and you have silently swapped two features — the scaler happily standardizes income using age's mean. Nothing errors. For a transformer fit on 10 columns, positional binding is a genuine footgun, and named arguments are the obvious ergonomic fix (project goal 1 is ergonomic authoring).
+
+WHY IT IS NOT A SMALL FIX (measured on datafusion 54.0.0, by Wren, 2026-07-23)
+The blocker is a Python-BINDING gap, not a SQL-syntax gap:
+
+    SELECT abs(x => age) FROM t    -> "Function 'abs' does not support named arguments"
+    SELECT abs(x := age) FROM t    -> same
+
+So the PARSER accepts both named-arg spellings and gets all the way to planning; the function then declines. Named-argument support is a per-function, Rust-side ScalarUDF capability. The Python bindings expose no way to opt in — datafusion.udf() has no named-arg parameter, and ScalarUDF exposes only [from_pycapsule, name, udf].
+
+CONSEQUENCE for our model: what we call "input names" today are Arrow STRUCT FIELD NAMES carried by the named_struct we synthesize — metadata on the type, not SQL argument names. Both engines align on that metadata (native reads feature_names_in_ at src/lib.rs:103 and reorders at src/expr.rs:414, hard-erroring when it is absent).
+
+Building a real named-argument surface therefore needs EITHER a Rust-side ScalarUDF registered via the PyCapsule path, OR upstream python-datafusion bindings that expose the capability. It is not a Python-side change.
+
+WHY IT IS A DRAFT
+No demand today — nobody has asked to author named args — and it needs design plus Rust/upstream work. Recorded primarily so nobody re-derives the finding. Becomes relevant if the ergonomic authoring goal ever wants named binding.
+
+NOT TO BE CONFUSED WITH the ndarray-fit positional-binding problem, which was solved Python-only inside TASK-3: synthesize the names from the call site onto a copy.copy() of the transformer (doc-8 clone contract preserved, fitted state shared, no deep copy, no src/*.rs edit).
+
+Context: doc-8 (composition — {transform}(col) references), doc-7 (transformer execution model).
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [ ] #1 Design decision recorded: Rust-side ScalarUDF via PyCapsule vs waiting on upstream python-datafusion bindings
+- [ ] #2 {t}(name => col) authoring form works end-to-end with transform == infer parity (DataFusion oracle, decision-1)
+- [ ] #3 Interaction with the existing feature_names_in_ / named_struct field-name mechanism is defined — named args must not silently diverge from struct field-name ordering
+<!-- AC:END -->

backlog/drafts/draft-12 - native-no-dispatch-for-struct...-and-make_array...-construction.md

@@ -0,0 +1,71 @@
+---
+id: DRAFT-12
+title: 'native: no dispatch for struct(...) and make_array(...) construction'
+status: Draft
+assignee: []
+created_date: '2026-07-23 14:30'
+updated_date: '2026-07-23 14:31'
+labels:
+  - native
+  - parity
+  - sql-surface
+  - containers
+dependencies: []
+references:
+  - src/expr_build.rs
+  - tests/test_diff_types.py
+documentation:
+  - doc-9 (Rich type system and UNNEST — status and deferred edges)
+  - doc-1 (DataFusion function catalogue — parity oracle)
+priority: medium
+type: feature
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+WHAT A USER HITS
+You copy a snippet from the DataFusion docs (or write what feels natural) to pack fields together:
+
+    SELECT struct(bedrooms AS beds, baths AS baths) AS layout FROM __THIS__
+    -- or
+    SELECT make_array(lat, lon) AS coords FROM __THIS__
+
+You call fit() and transform() on your training frame. Both work — that path runs on DataFusion.
+Then you deploy and call infer() for single-row serving. It fails: native doesn't recognize the function at all.
+
+So the feature works in your notebook and breaks in production. The user did nothing wrong — they used valid DataFusion SQL that the library accepted at fit time.
+
+The confusing part: for LISTS there is a spelling that works and one that doesn't, with nothing to tell them apart —
+
+    SELECT [lat, lon] AS coords    -- works on both engines today
+    SELECT make_array(lat, lon)    -- works on transform(), fails on infer()
+
+For STRUCTS there is no working alternative spelling at all on native.
+
+ROOT CAUSE
+native's convert_function (src/expr_build.rs) has no case for these container-construction forms, so it doesn't recognize them as functions. Codegen supports them and matches the DataFusion oracle; native does not:
+
+    SELECT struct(a, b) AS s        -> DataFusion/codegen: {c0: 1, c1: 2} (positional field names). Native: no dispatch.
+    SELECT struct(a AS x, b AS y)   -> DataFusion/codegen: {x: 1, y: 2} (parses as exp.PropertyEQ). Native: no dispatch.
+    SELECT make_array(a, b) AS l    -> DataFusion/codegen: [1, 2]. Native: no dispatch.
+
+NOT affected: the bracket literal `[a, b]` DOES reach native's list path and passes today (tests/test_diff_types.py::test_list_construct). The gap is specifically the FUNCTION-call spellings, not list/struct construction as a concept.
+
+SEVERITY vs DRAFT-13
+This is a missing capability that fails loudly, not a wrong answer — bad, but self-announcing. DRAFT-13 (mixed-numeric list widening) is the silent-wrong-value one and is rated higher.
+
+WHY IT MATTERS: native is the DEFAULT serving engine (decision-7), so a surface that works on the opt-in engine but not the default is backwards. Medium rather than High because the bracket literal gives users a working alternative for lists, and struct construction has no demonstrated demand yet.
+
+Surfaced by Ritchie's TASK-29 container work (2026-07-23), pinned by 3 strict xfail_on_native markers in tests/test_diff_types.py. Filed per the standing native-bug process (xfail-strict + ticket, never fix inline).
+
+DRAFT pending AmirHossein's review of scope/priority.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [ ] #1 native convert_function (src/expr_build.rs) dispatches struct(...) positional form, naming fields c0/c1/... exactly as DataFusion does
+- [ ] #2 native dispatches struct(a AS x, ...) named form (exp.PropertyEQ) with explicit field names
+- [ ] #3 native dispatches make_array(...) to the same construction path the bracket literal [a, b] already uses
+- [ ] #4 The 3 xfail_on_native markers in tests/test_diff_types.py (test_struct_construct_positional, test_struct_construct_named, test_make_array_construct) are removed and the tests pass on both engines against the DataFusion oracle
+<!-- AC:END -->

backlog/drafts/draft-13 - BUG-native-mixed-int-float-list-elements-are-not-widened-silent-value-divergence.md

@@ -0,0 +1,66 @@
+---
+id: DRAFT-13
+title: >-
+  BUG native: mixed int/float list elements are not widened (silent value
+  divergence)
+status: Draft
+assignee: []
+created_date: '2026-07-23 14:30'
+updated_date: '2026-07-23 14:30'
+labels:
+  - native
+  - parity
+  - bug
+  - containers
+dependencies: []
+references:
+  - src/types.rs
+  - tests/test_diff_types.py
+documentation:
+  - doc-9 (Rich type system and UNNEST — status and deferred edges)
+  - doc-1 (DataFusion function catalogue — parity oracle)
+priority: high
+type: bug
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+WHAT A USER HITS
+You bundle two features into one list column — a count and a ratio:
+
+    SELECT [bedrooms, price_per_sqft] AS dims FROM __THIS__
+
+`bedrooms` is an int column, `price_per_sqft` is a float column.
+
+At training time you call transform() over the DataFrame and get:   [3.0, 12.5]
+At serving time you call infer() on a single row and get:           [3, 12.5]
+
+No error. No warning. The int element silently stays an int on the serving path and becomes a float on the training path. Anything downstream that is dtype-sensitive — a numpy cast, a model expecting float64, a serialized feature vector compared against a stored schema — now sees different data in training than in production. This is the failure mode the whole differential harness exists to prevent: you only find it when the model misbehaves in prod, and nothing in the stack points at the list literal.
+
+Realistic triggers: any list feature mixing an integer column with a float column — [count, rate], [year_built, lot_ratio], [n_visits, avg_spend]. Users won't think of these as "mixed-type" — they're just two numbers.
+
+ROOT CAUSE (measured, not inferred)
+native's unify_list_element_types (src/types.rs) is EXACT-EQUALITY-ONLY, so it refuses to widen mixed numeric elements to a common type. DataFusion (the oracle) and codegen both widen to list<double>.
+
+    SELECT [x, y] AS l FROM t   -- x int = 1, y float = 2.5
+      DataFusion (oracle) + codegen:  [1.0, 2.5]   (widened to list<double>)
+      native:                         [1, 2.5]     (un-widened)
+
+WHY THIS IS THE SERIOUS ONE
+Unlike the struct/make_array dispatch gap (DRAFT-12), this is NOT a missing capability that errors out — native accepts the query and returns a DIFFERENT VALUE. Silent cross-engine divergence on the DEFAULT serving engine (decision-7). Proposed High on that basis.
+
+SAME BUG CLASS as the one Ritchie just fixed on the codegen side in infer_type's ListExpr arm (unifying element bases via _common_base, like COALESCE). The native fix mirrors it: compute a common numeric base instead of demanding exact equality.
+
+Surfaced by Ritchie's TASK-29 container work (2026-07-23), pinned by a strict xfail_on_native in tests/test_diff_types.py::test_list_construct_mixed_numeric_widens. Filed per the standing native-bug process — Ritchie correctly flagged rather than fixing inline. DataFusion is the oracle (decision-1).
+
+DRAFT pending AmirHossein's review of scope/priority.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [ ] #1 native unify_list_element_types (src/types.rs) computes a common numeric base for mixed elements instead of exact equality, mirroring codegen's _common_base/COALESCE logic
+- [ ] #2 SELECT [x, y] with int + float elements yields [1.0, 2.5] on native, matching the DataFusion oracle
+- [ ] #3 The xfail_on_native marker on tests/test_diff_types.py::test_list_construct_mixed_numeric_widens is removed and the test passes on both engines
+- [ ] #4 Root-cause sweep: confirm no other native type-unification site is exact-equality-only where DataFusion widens (the same class of silent divergence elsewhere)
+<!-- AC:END -->

backlog/drafts/draft-7 - Native-swap-native-per-transformer-Tier-0.md

@@ -4,7 +4,7 @@ title: 'Native-swap: native per-transformer (Tier 0)'
 status: Draft
 assignee: []
 created_date: '2026-07-18 13:44'
-updated_date: '2026-07-23 01:01'
+updated_date: '2026-07-23 14:31'
 labels:
   - sklearn
   - native-swap
@@ -20,7 +20,33 @@ ordinal: 5000
 ## Description
 
 <!-- SECTION:DESCRIPTION:BEGIN -->
-Swap each fallback-backed transformer to a native engine impl, diffed against the fallback oracle, in tier order. Most Tier 0/1 map onto shipped machinery (window aggs, PARTITION BY, LookupJoin, struct/UNNEST). Full plan: doc-2 (sklearn transformer plan).
+WHAT A USER HITS
+You build a normal preprocessing pipeline:
+
+    scaler = StandardScaler().fit(train[["age", "income"]])
+    t = SQLTransform("SELECT {scaler}(age, income) AS scaled FROM __THIS__").fit(train)
+    t.infer({"age": 41, "income": 82000})   # single-row serving
+
+Today this works, but every single call crosses into Python and runs sklearn's own .transform() — the opaque path. So the per-row serving latency is dominated by the Python/FFI boundary, which is exactly the cost the project exists to remove (see the boundary-bound finding). At n=1 you pay a full sklearn callout to compute (x - mean) / scale.
+
+The user-visible symptom is not wrongness, it's speed: serving is far slower than it should be for arithmetic this trivial, and the transformer can't be served from the non-Python runtimes at all (Go/Java/WASM) because it needs a Python interpreter in the loop.
+
+WHAT THIS TICKET DOES
+Swap each Tier 0 transformer from the opaque sklearn callout to a NATIVE implementation expressed in the engine's own SQL/expression surface, so it fuses into the single per-row expression instead of calling out.
+
+The insight (doc-7): these transformers are already expressible as window-aggregate SQL we compile today.
+    StandardScaler  ==  (x - AVG(x) OVER ()) / STDDEV(x) OVER ()
+    SimpleImputer   ==  COALESCE(x, <frozen fit-state>)
+    OrdinalEncoder  ==  a code-map lookup
+    OneHotEncoder   ==  one CAST(x = 'cat_i' AS INT) per learned category
+So most of this is macro definitions over shipped machinery, not new engine code.
+
+Each swap is diffed against the sklearn fallback as the oracle — the point is that switching a user from opaque to native must not change a single output value.
+
+WHY IT IS A DRAFT
+Needs design work before it is a dispatchable task (AmirHossein, 2026-07-23). Open questions: the state-shape contract per transformer (scalar vs list vs code-map vs per-group table), where the macro definitions live, how unknown-category handling is expressed, and whether OneHotEncoder's multi-output fan-out belongs here or in the feature-output work. Re-scope and re-promote to a TASK before dispatching.
+
+Full plan context: doc-2 (sklearn transformer implementation plan), doc-7 (transformer execution model).
 <!-- SECTION:DESCRIPTION:END -->
 
 ## Acceptance Criteria

backlog/drafts/draft-8 - ColumnTransformer-assembly-surface.md

@@ -4,7 +4,7 @@ title: ColumnTransformer assembly surface
 status: Draft
 assignee: []
 created_date: '2026-07-18 13:44'
-updated_date: '2026-07-23 01:02'
+updated_date: '2026-07-23 14:31'
 labels:
   - sklearn
   - fallback
@@ -21,7 +21,24 @@ ordinal: 6000
 ## Description
 
 <!-- SECTION:DESCRIPTION:BEGIN -->
-Column routing + horizontal concat; the end-to-end assembly-parity oracle: bit-identical width + column order + values vs stock ColumnTransformer.transform(). Must include a multi-output transformer. Detail in doc-2 (sklearn strategy) / doc-10 (feature-output).
+WHAT A USER HITS
+A real preprocessing job is never one transformer — it is different treatment per column group:
+
+    numeric   = ["age", "income", "lot_area"]        -> impute median, then scale
+    categorical = ["neighborhood", "house_style"]    -> one-hot
+    ordinal   = ["quality"]                          -> Ex/Gd/TA -> 5/4/3
+
+In sklearn you express that with a ColumnTransformer and get one matrix out, with a known column order. Today in this library there is no equivalent assembly surface: the user writes per-column SQL and is left to stitch the outputs together themselves, and the column ORDER of the assembled result is whatever falls out — which is the one thing a model cannot tolerate. Feed the model columns in a different order than it was trained on and you get silently wrong predictions, no error.
+
+The multi-output case is where it really bites: one-hot on a 25-category column contributes 25 columns whose width depends on what was seen at fit time. Users need the assembled width and order to be pinned by the fitted artifact, not recomputed per batch.
+
+WHAT THIS TICKET DOES
+Column routing + horizontal concat, with the parity bar being bit-identical output vs a stock sklearn ColumnTransformer.transform() — same width, same column order, same values. Must include at least one multi-output transformer so variable-width concat and feature-name expansion are actually exercised. Feature names/provenance must survive routing and concat in order, so users can map a column index back to what produced it.
+
+WHY IT IS A DRAFT
+Needs design work before it is dispatchable (AmirHossein, 2026-07-23), and it sits on top of the native-transformer draft (DRAFT-7). Open questions: what the user-facing authoring surface for column routing even looks like in SQL (this is the goal-1 ergonomics question, not just an implementation), how remainder/passthrough columns behave, and how this relates to the type-directed assembler (TASK-16) which solves the adjacent dense+sparse problem. Re-scope before promoting.
+
+Context: doc-2 (sklearn transformer implementation plan), doc-10 (feature-output model).
 <!-- SECTION:DESCRIPTION:END -->
 
 ## Acceptance Criteria