Board updates (rolling) — merge whenever

backlog/drafts/draft-14 - Per-group-transformer-fitting-transformerx-OVER-PARTITION-BY-g-fits-one-transformer-per-group.md

@@ -0,0 +1,94 @@
+---
+id: DRAFT-14
+title: >-
+  Per-group transformer fitting: transformer(x) OVER (PARTITION BY g) fits one
+  transformer per group
+status: Draft
+assignee: []
+created_date: '2026-07-23 15:38'
+labels:
+  - authoring-surface
+  - transformer-refs
+  - sklearn
+  - feature
+dependencies: []
+documentation:
+  - 'doc-7 (Transformer execution model — UDF/UDAF, macros, composition)'
+  - doc-2 (sklearn transformer implementation plan)
+  - 'doc-8 (Composition — {transform}(col) references)'
+priority: medium
+type: feature
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+ORIGIN: AmirHossein's proposal (2026-07-23), routed via Wren out of TASK-3 brainstorming. Measurements below are Wren's, taken on the current engine — not inferred.
+
+WHAT A USER HITS
+You are modeling something that spans groups with genuinely different scales — houses across countries, sales across regions, sensor readings across devices:
+
+    SELECT {scaler}(price) AS price_z FROM __THIS__
+
+This fits ONE scaler over all rows pooled. So every house in a cheap country reads as "below average" against the global mean, and the within-country signal — the thing you actually wanted — is flattened away. The user wants one scaler PER COUNTRY.
+
+Measured on today's engine: with country a = {10, 20} and b = {30, 50}, an unfit ref's state holds avg_age = [27.5] — the global mean — not the per-country 15 and 40.
+
+IMPORTANT: that is the CORRECT current semantic, not a bug. It matches sklearn, where a Pipeline step is fitted once on all training data. So this ticket is a genuinely new feature.
+
+Today the only workaround is to split the frame by group yourself, fit N scalers, and stitch the results — which happens outside the library and throws away the single-artifact serving story that is the whole point.
+
+PROPOSED SURFACE
+
+    standardscaler(x) OVER (country)          -- AmirHossein's shorthand
+    {scaler}(x) OVER (PARTITION BY country)   -- standard SQL spelling
+
+THE SPLIT — two very different sizes of work, and the ticket should stay honest about that:
+
+(a) SQL-EXPRESSIBLE transformers — StandardScaler, MinMaxScaler, MaxAbsScaler, mean-SimpleImputer.
+    ALREADY WORKS END-TO-END TODAY. This is pure syntax sugar; no engine work. Measured:
+
+        SELECT (x - AVG(x) OVER (PARTITION BY country)) /
+               STDDEV(x) OVER (PARTITION BY country) AS z FROM __THIS__
+
+        state __STATE_BY_country__ = {'country': ['b','a'],
+                                      'avg_x':    [40.0, 15.0],
+                                      'stddev_x': [14.142, 7.071]}   <- ONE ROW PER GROUP
+        batch == infer == [-0.7071, 0.7071, -0.7071, 0.7071]
+        unseen country at infer -> None (NULL), consistent with the existing unseen-partition semantic
+
+    So for this class the work is a MACRO / DESUGAR layer — exactly doc-7's "transformers are macros over the
+    window-agg/scalar SQL surface." Per-group state is the already-shipped PARTITION BY mechanism, and the
+    cold-start story is inherited for free.
+
+(b) OPAQUE sklearn objects — an arbitrary fitted transformer via {sc}(...).
+    GENUINELY NEW WORK. Needs N fitted clones keyed by group, per-group fitted state threaded through the
+    artifact and lookup path, and an explicit unseen-group policy at inference.
+
+OPEN DESIGN QUESTIONS (why this is a draft, not a task)
+1. UNSEEN-GROUP POLICY for (b): NULL, fall back to a globally-fitted transformer, or hard error? doc-2 calls
+   unknown-category handling "a designed-in requirement, not a flag," so this wants a ruled decision rather
+   than a default. Note (a) already answers it implicitly as NULL via the existing unseen-partition semantic —
+   worth deciding whether (b) must match.
+2. SYNTAX: accept the OVER (country) shorthand, or require the standard OVER (PARTITION BY country)?
+   Wren leans standard, on the grounds that (a) then desugars literally into already-valid SQL that the engine
+   machinery already speaks. A shorthand means new parsing for no new capability.
+3. SCOPE: is (a) alone worth shipping first? It is nearly free and delivers the common case (scalers/imputers),
+   while (b) is the real engine lift. Sequencing them as separate tickets is probably right.
+4. Artifact size for (b): N fitted clones means the serialized artifact grows with group cardinality. Is there
+   a ceiling, and what happens at high cardinality?
+
+NOT TO BE CONFUSED WITH DRAFT-11 (SQL named arguments). That one is about how ARGUMENTS BIND to a transformer's
+inputs. This one is about WHAT DATA a transformer is fitted on. Flagged explicitly so they do not get merged.
+
+Context: doc-7 (transformer execution model), doc-2 (sklearn transformer plan), doc-8 (composition).
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [ ] #1 DECISION: unseen-group policy at inference ruled explicitly (NULL vs global-fallback vs error), and whether the opaque path must match the SQL-expressible path's existing NULL semantic
+- [ ] #2 DECISION: shorthand OVER (g) accepted, or standard OVER (PARTITION BY g) required
+- [ ] #3 SQL-expressible transformers (StandardScaler/MinMaxScaler/MaxAbsScaler/mean-SimpleImputer) desugar to the existing PARTITION BY window-agg form, with transform == infer parity and one state row per group
+- [ ] #4 Opaque sklearn refs fit one clone per group, with per-group state carried in the artifact and resolved at inference per the ruled unseen-group policy
+- [ ] #5 Parity bar: per-group results match fitting the equivalent sklearn transformer separately on each group's rows
+<!-- AC:END -->

backlog/tasks/task-15 - Feature-output-scalar-one-hot-as-join-to-domain.md → backlog/drafts/draft-15 - Feature-output-scalar-one-hot-as-join-to-domain.md

@@ -1,16 +1,15 @@
----
-id: TASK-15
+---
+id: DRAFT-15
 title: 'Feature output: scalar one-hot as join-to-domain'
-status: To Do
+status: Draft
 assignee: []
 created_date: '2026-07-18 15:52'
-updated_date: '2026-07-23 14:32'
+updated_date: '2026-07-23 16:28'
 labels:
   - feature-output
-milestone: m-1
 dependencies: []
 documentation:
-  - 'doc-10 (Feature-output model — records, dense, sparse)'
+  - 'doc-10 (Feature-output model — records, dense, sparse)'
 priority: medium
 ordinal: 15000
 ---
@@ -19,30 +18,41 @@ ordinal: 15000
 
 <!-- SECTION:DESCRIPTION:BEGIN -->
 WHAT A USER HITS
-You want to one-hot a categorical column — the single most common preprocessing step there is:
+You want to one-hot a categorical column — the single most common preprocessing step there is:
 
     SELECT {ohe}(house_style) AS style FROM __THIS__
 
 Today this goes through the opaque sklearn callout: a Python boundary crossing per serving call, and a step that cannot run on the Go/Java/WASM runtimes. For an operation that is conceptually just "is this value equal to each known category," that is a lot of machinery.
 
-The user-visible payoff of this ticket is that a one-hot stops being a black box and becomes part of the fused per-row expression — so single-row serving gets fast, and the pipeline stays portable.
+The user-visible payoff of this ticket is that a one-hot stops being a black box and becomes part of the fused per-row expression — so single-row serving gets fast, and the pipeline stays portable.
 
 WHAT THIS TICKET DOES
 Compile a SCALAR one-hot into a JOIN against the fitted category domain table, riding the existing static_tables / lookup mechanism the engine already has (the same machinery TargetEncoder-style PARTITION BY work uses).
 
-Why a join and not an explode: this is the FIXED-FANOUT case. The category list is frozen at fit time, so the output width is known and the operation stays 1:1 with rows — no row-cardinality change, nothing to explode, and it composes with everything else in the same SELECT. That is what makes it cheap and why it is separable from the hard cases.
+Why a join and not an explode: this is the FIXED-FANOUT case. The category list is frozen at fit time, so the output width is known and the operation stays 1:1 with rows — no row-cardinality change, nothing to explode, and it composes with everything else in the same SELECT. That is what makes it cheap and why it is separable from the hard cases.
 
 Unknown categories at serving (a `house_style` never seen during fit) resolve through the join as a miss, which is the natural place to express sklearn's handle_unknown behavior.
 
 SCOPE BOUNDARY
-This is deliberately the EASY half of one-hot. The variable-expansion cases — tfidf, array multi-hot — need an explode and are parked separately (DRAFT-10). The sparse representation for high-cardinality columns is also separate (DRAFT-9). This ticket is the composable, fast-path slice that needs neither.
+This is deliberately the EASY half of one-hot. The variable-expansion cases — tfidf, array multi-hot — need an explode and are parked separately (DRAFT-10). The sparse representation for high-cardinality columns is also separate (DRAFT-9). This ticket is the composable, fast-path slice that needs neither.
 
 Independent: nothing blocks it, and it does not depend on the drafted spine.
 
-Context: doc-10 (feature-output model — records, dense, sparse).
+Context: doc-10 (feature-output model — records, dense, sparse).
 <!-- SECTION:DESCRIPTION:END -->
 
 ## Acceptance Criteria
 <!-- AC:BEGIN -->
 - [ ] #1 scalar one-hot compiles to a join-to-domain, no explode
 <!-- AC:END -->
+
+## Comments
+
+<!-- COMMENTS:BEGIN -->
+author: Iris (PM)
+created: 2026-07-23 16:28
+---
+Moved to Draft (2026-07-23, AmirHossein). This is an OPTIMIZED SKLEARN TRANSFORM implementation — compiling one-hot down to a join-to-domain instead of the opaque callout. That class of work is getting its own milestone, which does NOT exist yet by explicit decision. Parked here until that milestone is created and this is scoped into it. Sibling drafts in the same class: DRAFT-7 (native per-transformer swap).
+---
+<!-- COMMENTS:END -->
+

backlog/drafts/draft-17 - BUG-differential-helper-used-exact-float-equality-—-older-tests-in-that-style-are-latent-flakes.md

@@ -0,0 +1,53 @@
+---
+id: DRAFT-17
+title: >-
+  BUG differential helper used exact float equality — older tests in that style
+  are latent flakes
+status: Draft
+assignee: []
+created_date: '2026-07-24 01:59'
+labels:
+  - test-infra
+  - parity
+  - bug
+  - flake
+dependencies: []
+references:
+  - 'PR #16'
+  - tests/test_transformer_ref.py
+priority: high
+type: bug
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+WHAT GOES WRONG
+The `_both_engines` differential helper compared engine outputs with EXACT float equality. The batch path and the row-at-a-time path are not bit-identical — an ordinary non-collinear fixture differs by ~2e-16 between them. Existing tests passed only because their fixtures happened to land on values where the two paths agreed exactly.
+
+So the test suite has been asserting "these engines agree" using a comparison that can fail on a fixture change no one would think twice about — adding a row, nudging a value, changing a column. The failure would look like a real parity regression and cost someone a debugging session before they realized the comparison itself was wrong.
+
+WHY THIS IS RATED HIGH DESPITE BEING "JUST TESTS"
+Our entire correctness story is differential testing against the DataFusion oracle (decision-1). A comparison primitive that passes on fixture luck weakens every assertion built on it, in both directions:
+- FALSE ALARM: an innocuous fixture edit trips a 2e-16 difference and reads as a parity bug.
+- FALSE CONFIDENCE, the worse one: a helper that is wrong about how to compare is a helper nobody should trust to have been catching real divergence.
+
+Found by Wren during TASK-3 (2026-07-23). He fixed the instance in PR #16 because he was already adding tests to that file, but the fix was NOT swept across the codebase — any older test written in the same exact-equality style is still a latent flake.
+
+SCOPE
+This is the sweep TASK-3 deliberately did not do. Find every exact float comparison in the differential/parity test helpers and tests, and make them use an appropriate tolerance. The point is not to fix one file; it is to establish that no parity assertion anywhere is passing by fixture luck.
+
+Worth deciding as part of this: what the project's standard float comparison for parity assertions IS, so the next person does not reinvent it. A single shared helper is better than a tolerance argument sprinkled at call sites.
+
+IN SCOPE FOR THE CURRENT MILESTONE: this is squarely "bulletproof opaque transforms that actually work" — it is about whether our correctness net has holes, not about making anything faster.
+
+DRAFT pending AmirHossein's review of scope/priority.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [ ] #1 Every exact float comparison in differential/parity test helpers and tests is identified (a sweep, not a single-file fix)
+- [ ] #2 A single shared float-comparison helper with a documented tolerance is used for parity assertions, rather than per-call-site tolerances
+- [ ] #3 Verified the sweep is real: a deliberately perturbed fixture (values changed but semantics identical) does not flip any parity test
+- [ ] #4 Any test found to have been passing on fixture luck is called out, since it means that assertion was not actually proving what it claimed
+<!-- AC:END -->