Specializer SQL support: SELECT * star expansion — corpus 53 → 172 (TASK-46)

backlog/tasks/task-45 - Specializer-M-boundary-generated-row-marshaller-Python-API.md

@@ -1,10 +1,10 @@
 ---
 id: TASK-45
 title: 'Specializer M-boundary: generated row marshaller + Python API'
-status: In Progress
+status: Done
 assignee: []
 created_date: '2026-07-25 02:32'
-updated_date: '2026-07-26 09:10'
+updated_date: '2026-07-26 11:40'
 labels: []
 milestone: m-7
 dependencies:
@@ -46,3 +46,9 @@ Stretch plan (recorded 2026-07-26, design doc §3 flag 1 + §10). Measured targe
 All four stretches landed on claude/specializer-m-boundary. The marshaller (src/duckdb/mod.rs) does at prepare time everything knowable at prepare time: interned attribute-name PyStrings in fixed field order, input buffers + RunState owned and cleared-not-dropped per call, dict rows via get_item and model rows via getattr on the interned names, output rows by direct pydantic-v2 slot fill. Two assumptions died by measurement: (1) pydantic's literal model_construct API is pure-Python and SLOWER than model_validate (1432 vs 882 ns/row on 2.13) — the shipped path is object.__new__ + object.__setattr__ of __dict__/__pydantic_fields_set__/__pydantic_extra__/__pydantic_private__ at 491 ns, semantically equal (eq, fields_set, assignment all verified); (2) the design doc's AoS row structs buy nothing at L1-resident n — the marshaller fills the existing SoA Batch directly (deliberate deviation, noted in the plan). AC #4 forced real work: ColData::Str became one flat buffer + spans (killing per-cell Strings AND a hidden per-load clone in the JIT's h_load_str), substr/trim became pure sub-span arithmetic, case mapping streams into the arena (Arena::case_map), number→text formats via Arena::push_fmt with DuckF64 on a stack buffer, and h_probe emits without its per-call Vec + ScalarVal clones — all shared between backends, pinned by counting-allocator tests over a probe/arith fixture and a string-heavy program on BOTH backends (the remaining per-call allocs are the output objects themselves plus pyo3's input-list Vec, i.e. the AC's "beyond the output objects"). SpecializedTransform (sql_transform/_specialized.py) reuses the whole fit pipeline minus transformer refs (ValueError at ctor), so window aggregates ride the equi-join rewrite onto cranelift — parity with SQLTransform asserted row-for-row; WHERE stays rejected at the authoring surface (parse_and_validate), while raw DuckDBInferFn keeps it. Bench (scripts/bench_specializer.py, +generic engine via SPECIALIZER_GENERIC_BOUNDARY): noop p50 marshaller vs generic = 1.1/2.1µs at n=1, 468/1192µs at n=1024 (1.9-2.5x, AC #2); vs shipping engines at n=1024 the specializer is 3.7-4.2x faster than native and 2.4-3.8x than codegen (AC #3); cranelift-vs-interp is now visible end-to-end (arith 339 vs 391µs at n=1024). SPECIALIZER_GENERIC_BOUNDARY + .boundary getter mirror the FORCE_INTERP pattern; infer_rows() is the direct hot entry SpecializedTransform uses.
 <!-- SECTION:NOTES:END -->
 
+
+## Final Summary
+
+<!-- SECTION:FINAL_SUMMARY:BEGIN -->
+M-boundary delivered and merged (PR #29, rebase-merged 2026-07-26). Generated row marshaller + SpecializedTransform + allocation-free steady state on both backends, with a 17-agent adversarial fleet's 9 confirmed findings (3 root causes) fixed pre-merge: supplied output models keep model_validate semantics, the generic baseline accepts dict rows, reentrant infer falls back instead of erroring. Post-merge the milestone also grew the realistic serving bench (benchmarks/, PR #29): four famous-problem inference paths (titanic 10->24, ames 43->42, ieee-cis fraud 32->41, rossmann 21->44) under an exact three-way parity gate (specializer == DuckDB == handcrafted twin, pytest-enforced). Measured: the specializer beats the handcrafted-Python typed-model server in 16/16 cells (1.1-2.4x), DuckDB-per-call by 1,200-2,700x at n=1, and the previous native/codegen engines build 0/4 scenarios (IS NULL projections, row-x-dim arithmetic unsupported there). Known remaining gap, deliberately reported: a plain-dict handcrafted server is still 1.3-2x faster — typed-output construction is the next perf lever (parked; SQL support prioritized by AmirHossein 2026-07-26).
+<!-- SECTION:FINAL_SUMMARY:END -->

backlog/tasks/task-46 - Specializer-SQL-support-SELECT-star-expansion.md

@@ -0,0 +1,48 @@
+---
+id: TASK-46
+title: 'Specializer SQL support: SELECT * star expansion'
+status: In Progress
+assignee: []
+created_date: '2026-07-26 11:42'
+updated_date: '2026-07-26 11:55'
+labels: []
+milestone: m-7
+dependencies:
+  - TASK-45
+documentation:
+  - docs/superpowers/specs/2026-07-25-sql-specializer-design.md
+type: feature
+ordinal: 40000
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+The single biggest corpus rung: 128 of 625 clean-unsupported corpus cases have `SELECT *` (or `tbl.*`) as their first blocker. Expand the star at bind time in the frontend against DuckDB's measured semantics — column order and naming for the row table alone and under joins (including duplicate-name handling), `tbl.*` qualified forms, and whatever star modifiers the corpus actually uses (EXCLUDE/REPLACE are measured-first: support only what the corpus needs, reject the rest cleanly by name). Pure frontend work — no IR, backend, or boundary changes. Clearing it also de-masks second blockers currently hidden behind star for the builtins wave (TASK-47).
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [x] #1 Star semantics pinned by measurement against DuckDB 1.5.5 (column order, duplicate-name policy, qualified tbl.*, modifier handling) and recorded as duck_check tests
+- [x] #2 Corpus replay: star-first-blocker cases flip to match or to a NAMED deeper blocker; zero FAILs; match count and the new first-blocker tally recorded here
+- [x] #3 Unsupported star forms (if any remain) reject with a clean "unsupported: ..." naming the form
+- [x] #4 mise gate-specializer green
+<!-- AC:END -->
+
+## Implementation Plan
+
+<!-- SECTION:PLAN:BEGIN -->
+Stretch plan (recorded 2026-07-26 after the measurement spike; all facts below are measured on DuckDB 1.5.5).
+Corpus star census: 156 plain `SELECT *`, 26 `tbl.*`, 16 `* EXCLUDE`, 13 `COLUMNS(...)`, 3 `* LIKE`, 2 `* REPLACE`, 2 mixed. Wave scope: plain `*`, `tbl.*`, `EXCLUDE` (~198 mentions); COLUMNS()/REPLACE/LIKE/RENAME/EXCEPT/ILIKE reject by name. All forms already parse under sqlparser 0.62 and funnel to the single rejection at frontend.rs:125 — one expansion site.
+Measured pins: expansion order is FROM order then declared column order per table; `SELECT * FROM a JOIN b ON a.k=b.k` emits DUPLICATE column names ('k','x','k','y') — unrepresentable in a pydantic output model, and any star item covering a joined static table necessarily includes its join-key column, whose projection our engine already cleanly rejects; therefore star items that cover a static table reject cleanly naming the key column (covers the duplicate-name shape too). `__THIS__.*` under a join is fine (row columns only). EXCLUDE: paren and bare forms; case-insensitive; unknown name = binder error mirroring DuckDB ("Column \"nope\" in EXCLUDE list not found"); duplicate list entry = error ("Duplicate entry"); unqualified EXCLUDE removes every same-named column.
+1. Binder::expand_star(qualifier, options) -> Vec<(name, SExpr)> using the existing SKind::Col / static_lane constructors; wire into the projection loop (mixed `*, expr` items fall out of item-position expansion); existing duplicate-output-name check stays as the final guard.
+2. duck_check pins for every measured behavior above incl. the EXCLUDE edges (exclude-all, EXCLUDE K case-folding) and clean-unsupported messages for the rejected forms.
+3. Corpus replay re-tally into this ticket (AC #2): flips + newly-surfaced second blockers.
+4. Gate green.
+<!-- SECTION:PLAN:END -->
+
+## Implementation Notes
+
+<!-- SECTION:NOTES:BEGIN -->
+Landed in one pass: Binder::expand_star (frontend.rs) expands `*` and `tbl.*` at the single projection site using the existing SKind::Col / static_lane constructors — no IR/backend/boundary changes, exactly as scoped. EXCLUDE handles bare + paren forms case-insensitively (sqlparser 0.62 carries entries as ObjectName; single-part idents only, qualified entries reject by name); unknown-column and duplicate-entry EXCLUDE mirror DuckDB's binder errors; exclude-all falls through to the existing empty-SELECT bind error (same shape as DuckDB's "SELECT list is empty after resolving * expressions"). Star items covering a joined static table reject cleanly naming the join-key column (DuckDB includes the key AND emits duplicate output names there — both unrepresentable; `__THIS__.*` under a join works). ILIKE/EXCEPT/REPLACE/RENAME/COLUMNS() reject by name. CORPUS RESULT (AC #2): 53 -> 172 match of 678 (+119, a 3.2x coverage jump), 0 FAILs, 506 clean-unsupported. New first-blocker head: BETWEEN 31, comma join 30, dynamic-table alias 30, table-function FROM 24, then the builtin catalogue (array_slice 23, contains 18, damerau_levenshtein 17, ...) — the TASK-47 target list confirmed. Gate green: cargo 129 + pytest 627 (13 xfail incl. the pre-existing substr const-fold pin from master).
+<!-- SECTION:NOTES:END -->

backlog/tasks/task-47 - Specializer-SQL-support-workload-builtins-and-predicates-wave-1.md

@@ -0,0 +1,31 @@
+---
+id: TASK-47
+title: 'Specializer SQL support: workload builtins & predicates wave 1'
+status: To Do
+assignee: []
+created_date: '2026-07-26 11:42'
+labels: []
+milestone: m-7
+dependencies:
+  - TASK-46
+documentation:
+  - docs/superpowers/specs/2026-07-25-sql-specializer-design.md
+  - docs/superpowers/specs/2026-07-26-stretch4-builtin-pins.md
+type: feature
+ordinal: 41000
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+Close the workload ladder measured by the serving-bench scenarios (benchmarks/serving_scenarios/, each module's compromises list) plus the overlapping corpus predicates. Ranked by how many famous-solution pipelines hit the wall: (1) ln/log/log2/log10/log1p/exp — blocked features in all four scenarios (log-fare, log1p amount, log sales, skew fixes); (2) true floor/ceil/trunc — CAST rounds half-even, so decade bins / cents / week buckets are inexpressible; (3) instr/position/strpos + contains/starts_with/ends_with — title extraction, email-domain and device parsing; (4) IN (...) and BETWEEN predicates — also 72+ corpus first-blocker cases; (5) pow/sqrt (fractional) — Box-Cox and sqrt skew features; (6) sin/cos — cyclical hour/month encodings; (7) least/greatest — clamp ergonomics. Every function lands via the measured-pin discipline (builtin-pins spec): pin DuckDB 1.5.5 semantics with duck_check tests FIRST (edge cases: domain errors, NULL propagation, -0.0/NaN/inf, int/float overloads), then lower, then implement on BOTH backends via shared semantic functions. Float-y functions must match DuckDB bit-exactly or trap cleanly — the differential decides.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [ ] #1 Each shipped function/predicate has measured DuckDB pins recorded as duck_check tests before its implementation landed (domain edges, NULL, special floats)
+- [ ] #2 Interpreter and cranelift agree byte-identically on all new ops (shared semantic fns; 500-seed differential extended to cover them)
+- [ ] #3 The four serving scenarios' compromises lists re-audited: every gap this wave claims to close is exercised by an upgraded scenario feature or a new duck_check
+- [ ] #4 Corpus replay: predicate/function first-blocker cases flip to match or to a named deeper blocker; zero FAILs; new tally recorded here
+- [ ] #5 mise gate-specializer green
+<!-- AC:END -->

src/specializer/frontend.rs

@@ -115,17 +115,11 @@ pub fn frontend(
 
     let mut out_cols = Vec::new();
     let mut exprs = Vec::new();
-    for item in &select.projection {
-        let (name, e) = match item {
-            SelectItem::UnnamedExpr(e) => (default_name(e), fold(binder.expr(e)?)),
-            SelectItem::ExprWithAlias { expr, alias } => {
-                (alias.value.clone(), fold(binder.expr(expr)?))
-            }
-            SelectItem::Wildcard(_) | SelectItem::QualifiedWildcard(..) => {
-                return Err(unsup("SELECT * (star expansion)"))
-            }
-            SelectItem::ExprWithAliases { .. } => return Err(unsup("multi-alias SELECT item")),
-        };
+    let push_item = |out_cols: &mut Vec<Col>,
+                     exprs: &mut Vec<SExpr>,
+                     name: String,
+                     e: SExpr|
+     -> Result<(), PrepareError> {
         if out_cols.iter().any(|c: &Col| c.name == name) {
             // DuckDB allows duplicate output names; our IR requires unique
             // columns. Rare in real queries — punt cleanly for now.
@@ -139,6 +133,40 @@ pub fn frontend(
             },
         });
         exprs.push(e);
+        Ok(())
+    };
+    for item in &select.projection {
+        match item {
+            SelectItem::UnnamedExpr(e) => push_item(
+                &mut out_cols,
+                &mut exprs,
+                default_name(e),
+                fold(binder.expr(e)?),
+            )?,
+            SelectItem::ExprWithAlias { expr, alias } => push_item(
+                &mut out_cols,
+                &mut exprs,
+                alias.value.clone(),
+                fold(binder.expr(expr)?),
+            )?,
+            SelectItem::Wildcard(opts) => {
+                for (name, e) in binder.expand_star(None, opts)? {
+                    push_item(&mut out_cols, &mut exprs, name, e)?;
+                }
+            }
+            SelectItem::QualifiedWildcard(kind, opts) => {
+                let table = match kind {
+                    sqlparser::ast::SelectItemQualifiedWildcardKind::ObjectName(n) => n.to_string(),
+                    sqlparser::ast::SelectItemQualifiedWildcardKind::Expr(_) => {
+                        return Err(unsup("expression.* wildcard"))
+                    }
+                };
+                for (name, e) in binder.expand_star(Some(&table), opts)? {
+                    push_item(&mut out_cols, &mut exprs, name, e)?;
+                }
+            }
+            SelectItem::ExprWithAliases { .. } => return Err(unsup("multi-alias SELECT item")),
+        };
     }
     if exprs.is_empty() {
         return Err(PrepareError::Bind("SELECT list is empty".to_string()));
@@ -450,6 +478,98 @@ struct Binder<'a> {
 }
 
 impl Binder<'_> {
+    /// Expand `*` / `tbl.*` per DuckDB's measured semantics (1.5.5): FROM
+    /// order, declared column order within a table, EXCLUDE filtered
+    /// case-insensitively. A star item covering a joined static table is
+    /// unsupported: DuckDB's expansion includes the join-key column there
+    /// (and emits duplicate output names for the shared key), neither of
+    /// which the engine models — rejected by name, not silently narrowed.
+    fn expand_star(
+        &self,
+        qualifier: Option<&str>,
+        opts: &sqlparser::ast::WildcardAdditionalOptions,
+    ) -> Result<Vec<(String, SExpr)>, PrepareError> {
+        use sqlparser::ast::ExcludeSelectItem;
+        if opts.opt_ilike.is_some() {
+            return Err(unsup("SELECT * ILIKE (COLUMNS filter)"));
+        }
+        if opts.opt_except.is_some() {
+            return Err(unsup("SELECT * EXCEPT"));
+        }
+        if opts.opt_replace.is_some() {
+            return Err(unsup("SELECT * REPLACE"));
+        }
+        if opts.opt_rename.is_some() {
+            return Err(unsup("SELECT * RENAME"));
+        }
+        fn exclude_name(n: &sqlparser::ast::ObjectName) -> Result<&str, PrepareError> {
+            match n.0.as_slice() {
+                [part] => part
+                    .as_ident()
+                    .map(|i| i.value.as_str())
+                    .ok_or_else(|| unsup("EXCLUDE list entry form")),
+                _ => Err(unsup("qualified name in EXCLUDE list")),
+            }
+        }
+        let exclude: Vec<&str> = match &opts.opt_exclude {
+            None => Vec::new(),
+            Some(ExcludeSelectItem::Single(id)) => vec![exclude_name(id)?],
+            Some(ExcludeSelectItem::Multiple(ids)) => ids
+                .iter()
+                .map(exclude_name)
+                .collect::<Result<Vec<_>, _>>()?,
+        };
+        for (i, a) in exclude.iter().enumerate() {
+            if exclude[..i].iter().any(|b| b.eq_ignore_ascii_case(a)) {
+                // DuckDB rejects this at parse; ours surfaces at bind.
+                return Err(PrepareError::Bind(format!(
+                    "duplicate entry \"{a}\" in EXCLUDE list"
+                )));
+            }
+        }
+
+        let mut cols: Vec<(String, SExpr)> = Vec::new();
+        let mut matched = false;
+        if qualifier.is_none_or(|q| q.eq_ignore_ascii_case(&self.this_name)) {
+            matched = true;
+            for (i, c) in self.in_cols.iter().enumerate() {
+                cols.push((
+                    c.name.clone(),
+                    SExpr {
+                        kind: SKind::Col(i as u32),
+                        ty: c.ty.ty,
+                        nullable: c.ty.nullable,
+                    },
+                ));
+            }
+        }
+        for sj in &self.joins {
+            if qualifier.is_none_or(|q| q.eq_ignore_ascii_case(&sj.name)) {
+                let key = &sj.table.cols[sj.key_cols[0] as usize].name;
+                return Err(unsup(format!(
+                    "star expansion over joined table '{}' (includes join key column '{key}')",
+                    sj.name
+                )));
+            }
+        }
+        if !matched {
+            return Err(PrepareError::Bind(format!(
+                "table '{}' in wildcard does not exist in FROM",
+                qualifier.unwrap_or("?")
+            )));
+        }
+
+        for ex in &exclude {
+            if !cols.iter().any(|(n, _)| n.eq_ignore_ascii_case(ex)) {
+                return Err(PrepareError::Bind(format!(
+                    "column \"{ex}\" in EXCLUDE list not found in FROM clause"
+                )));
+            }
+        }
+        cols.retain(|(n, _)| !exclude.iter().any(|ex| ex.eq_ignore_ascii_case(n)));
+        Ok(cols)
+    }
+
     /// Bind an expression that must have a definite type on its own — a bare
     /// NULL literal here is unsupported (no context to type it).
     fn expr(&self, e: &SqlExpr) -> Result<SExpr, PrepareError> {