feat: schema-aware resolution, plus DRAFT-21 (step semantics, parked)

SQLProjection(sql, this_model=Row) declares the __THIS__ schema (pydantic
field names, definition order; the model is authoritative — fit
canonicalizes the table to it). One mechanism carries the whole loop:
the base substitution environment becomes explicit, and everything falls
out — unknown columns refuse at construction with bind-style messages;
every star expands to named items so * EXCLUDE/REPLACE/RENAME compose at
any level; COLUMNS('regex') expands with the pattern matched by DuckDB's
own regex engine, never Python's re; lateral aliases resolve by DuckDB's
column-wins rule (still refused inside window arguments, whose fit-side
text runs against the source relation); struct access composes through
plain projected columns. this_model moves from fit to the constructor
(v0 break); schema-free construction keeps every loop-1..3 behavior.

Schema mode always canonicalizes (no identity fast-path): expansion is
the feature, and the serving text stays in vanilla form.

DRAFT-21 parks the agreed direction for order-keyed windows off the
training support: step-function semantics via AS-OF joins (cume_dist
becomes the empirical CDF against training), sequenced after this loop.

Gate: 1500-case fuzz bit-exact with half of all cases running schema
mode; corpus gains 5 schema families; root pytest 494+1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: ahrzb

backlog/drafts/draft-21 - Step-semantics-for-order-keyed-windows-off-the-training-support.md

@@ -0,0 +1,90 @@
+---
+id: DRAFT-21
+title: Step semantics for order-keyed windows off the training support
+status: Draft
+type: spike
+created: 2026-07-29
+---
+
+## Where this came from
+
+Design dialogue with AmirHossein, 2026-07-29, right after loop 3
+(projection chains + the fit plan) merged. The question: **what should
+`rank()` — and every order-keyed window — mean when the serving row's value
+was never in the training set?** Parked with a converged direction so the
+conclusions survive; sequenced after the schema-aware-resolution loop.
+
+## The problem
+
+Order-keyed windows (`rank`, `dense_rank`, `percent_rank`, `cume_dist`,
+running aggregates under RANGE/GROUPS frames, `last_value`/`nth_value`)
+marginalize with the order values in the join key set. Serving joins by
+exact `IS NOT DISTINCT FROM` match — so an unseen order value (almost
+*every* value, when the key is continuous like `salary`) misses the LEFT
+JOIN and yields NULL. Truthful, but it makes the whole order-keyed family
+nearly useless on real serving traffic.
+
+## The converged answer: the step-function reading
+
+Every order-discriminated window we admit is, per partition, a **step
+function of the order value** — that is exactly *why* it was marginalizable.
+The params table is that function sampled at the training points. The fitted
+transform should mean: **evaluate the training-set step function at the
+serving row's coordinate** (row excluded).
+
+```
+rank() OVER (PARTITION BY dep ORDER BY salary)
+-- fitted step for dep='sales':  4800 → 1 (two peers), 5000 → 3
+-- serving salary=4750 (unseen):
+--   exact-join semantics (today):  NULL
+--   step semantics:                1 + #{training rows < 4750} = 1
+```
+
+Why this is the right default, not an accommodation:
+
+1. **It extends, never contradicts.** At observed coordinates the step
+   function equals the exact join, so the training-set round-trip invariant
+   is untouched. Exact-join is the degenerate restriction of step semantics.
+2. **Partition keys stay exact.** Categorical keys have no "between";
+   NULL for an unseen group remains the honest answer. Step semantics is for
+   *order* keys only — partition keys are categorical, order keys ordinal.
+3. **`cume_dist() OVER (ORDER BY x)` becomes the empirical CDF against
+   training** — sklearn's QuantileTransformer, falling out of one SQL
+   function. Strong evidence the step reading is the ML-correct semantics.
+4. **Still oracle-testable.** Off-support behavior has a definitional form
+   DuckDB can execute per probe (`SELECT 1 + count(*) FROM train WHERE
+   dep = ? AND salary < ?`), so the gate extends rather than weakens.
+
+## Mechanics (sketch)
+
+AS-OF join instead of equality on the order-key part of the join predicate.
+DuckDB has `ASOF JOIN` natively for the fit-side gate; on the Confit side it
+is a binary search over a sorted params column — fits the static-lookup
+architecture. Per family:
+
+- running `sum`/`count`/`min`/`max` (RANGE ≤ frames): as-of pick of the
+  cumulative value at the greatest observed `o' ≤ v`.
+- `rank`: `1 + cum_count(o < v)` — one extra cumulative-count column in the
+  params table; exact hits read the stored rank.
+- `percent_rank`/`cume_dist`: derived from rank/cum_count and `n`.
+
+## Pins needed before believing anything
+
+- Tie boundaries: `<` vs `≤` per function, and DuckDB ASOF's strictness.
+- NULL order values at serving: NULL is its own peer class — stays an exact
+  match, never an as-of probe.
+- `last_value` off-support is genuinely debatable: "the last peer of a value
+  nobody has" is arguably the serving row itself — a different kind of
+  answer than a frozen param. Decide per-function; refusing step semantics
+  for `last_value`/`nth_value` and keeping them exact-only is a fine v0.
+- Bounded RANGE frames (`RANGE BETWEEN 2 PRECEDING AND CURRENT ROW`): the
+  step function has 2·n breakpoints, not n — the as-of table must sample
+  frame *boundaries*, not just observed points. May stay exact-only at
+  first.
+
+## Decision state
+
+Direction agreed ("ah that's nice I like it"): adopt step semantics as THE
+meaning of order-keyed windows, in its own loop with its own pins and an
+off-support gate, sequenced after schema-aware resolution. Until that loop,
+NULL-off-support is the documented behavior.

docs/superpowers/specs/2026-07-29-schema-aware-resolution-design.md

@@ -0,0 +1,56 @@
+# Schema-aware resolution
+
+**Date:** 2026-07-29
+**Status:** loop 4 of SQLProjection marginalization; extends loops 1–3.
+
+## What changes
+
+`marginalize(sql, columns=None)` optionally takes the `__THIS__` column
+names; `SQLProjection(sql, this_model=SomeModel)` passes its pydantic field
+names (in definition order — the model is the declared, authoritative
+schema). `fit(table)` then canonicalizes with `table.select(columns)`: model
+order wins, extra table columns drop, missing ones fail with a named error.
+`this_model` moves from `fit` to the constructor (v0 break) — errors surface
+at construction and the plan is final before any data exists.
+
+Schema-free mode is unchanged: every loop-1–3 behavior and refusal stands.
+
+## The mechanism: the base environment becomes explicit
+
+Schema knowledge is one change, not many: the base level's substitution
+environment stops being a star-fallback and becomes an explicit ordered list
+of `(name, __cf_t.name)` entries — base columns are just substitution
+expressions like any projected column. Everything else falls out:
+
+- **Unknown columns refuse at construction** with a bind-style message.
+- **Struct access composes through plain columns**: a substitution target
+  that is itself a pure column reference extends (`s.f` → `__cf_t.s.f`) —
+  including through a CTE that projected the column; only access through a
+  *computed* expression stays refused.
+- **Every star expands to named items**, so `* EXCLUDE/REPLACE/RENAME`
+  compose at any level (previously final-over-base only), and star-through-
+  CTE needs no special cases.
+- **`COLUMNS('regex')` expands** against the known names — the pattern is
+  matched *by DuckDB itself* (`regexp_matches` at marginalize time), never by
+  Python's `re`, so the dialect of the regex is the oracle's. Lambda and
+  EXCLUDE forms of COLUMNS stay refused by name. Select-item position only.
+- **Lateral aliases resolve exactly by DuckDB's rule**: a source column wins
+  over an earlier item's alias; otherwise the alias's rewritten expression is
+  inlined; otherwise unknown-column. The schema-free ambiguity refusal
+  remains only in schema-free mode. Inside window arguments/keys/filters a
+  lateral alias stays refused (the window's fit-side text runs against the
+  source relation, which has no such column) — a named error with the same
+  disambiguation hint.
+
+## Gate
+
+The round-trip invariant is unchanged and the fuzz gains a schema mode: half
+of all generated cases construct with a `this_model` built from the table, so
+every resolution path runs explicit. Corpus: new curated families (COLUMNS,
+star modifiers through CTEs, resolved lateral aliases, struct-through-CTE)
+move the pinned scoreboard up.
+
+## Out of scope
+
+Step semantics for off-support order keys (DRAFT-21, next); t-strings
+(deprioritized); joins/static tables; `COLUMNS` inside expressions.

packages/sql-transform/README.md

@@ -46,6 +46,13 @@ named windows — plus scalar and `EXISTS` subqueries over `__THIS__`, which
 are provably uncorrelated (there is no syntax to reach the outer row) and run
 verbatim at fit time.
 
+Declaring a schema unlocks more: `SQLProjection(sql, this_model=Row)` (the
+pydantic model is the authoritative `__THIS__` schema) makes unknown columns
+refuse at construction, expands `COLUMNS('regex')` (matched by DuckDB's own
+regex engine) and `* EXCLUDE/REPLACE/RENAME` at any level, and resolves
+lateral aliases by DuckDB's column-wins rule. Schema-free construction keeps
+working, with the ambiguous cases refused.
+
 The rule deciding it all: a window is marginalizable iff its value is a
 function of row-visible values. What depends on **physical row position** —
 `row_number`, `ntile`, `lag`/`lead`, bounded ROWS frames, `EXCLUDE` — is

packages/sql-transform/sql_transform/_corpus_test.py

@@ -171,6 +171,17 @@
 ]
 
 
+# --- curated, schema-aware (loop 4): replayed with a declared this_model -----
+
+CURATED_SCHEMA = [
+    "SELECT COLUMNS('.*name') FROM __THIS__",
+    "SELECT * EXCLUDE (enroll_date) REPLACE (salary + 1 AS salary) FROM __THIS__",
+    "SELECT * RENAME (depname AS dep) FROM __THIS__",
+    "SELECT salary + 1 AS s2, s2 * 2 AS s4 FROM __THIS__",
+    "WITH a AS (SELECT * EXCLUDE (empno) FROM __THIS__) SELECT salary - avg(salary) OVER (PARTITION BY depname) AS d FROM a",
+]
+
+
 def _outcome(sql: str) -> tuple[str, str]:
     try:
         marginalize(sql)
@@ -208,9 +219,21 @@ def test_curated_refuses(sql):
     assert kind == "refused", f"{kind}: {detail}"
 
 
+@pytest.mark.parametrize("sql", CURATED_SCHEMA, ids=lambda s: s[:56])
+def test_curated_schema_marginalizes(sql):
+    from sql_transform._projection_test import gate
+
+    try:
+        marginalize(sql, list(EMPSALARY.column_names))
+    except MarginalizeError as e:
+        raise AssertionError(f"refused: {e}") from e
+    gate(sql, EMPSALARY, schema=True)
+
+
 def test_progression_totals():
     """The metric, in one place. Edit these pins when a loop widens support."""
     assert len(MINED) == 22
     assert MINED_SCOREBOARD["marginalized"] + MINED_SCOREBOARD["refused"] == 22
     assert len(CURATED_MARGINALIZED) == 39
     assert len(CURATED_REFUSED) == 17
+    assert len(CURATED_SCHEMA) == 5