Specializer SQL support wave 4: join forms — USING, residual ON, semi joins, comma rewrite — corpus 383 → 395 (TASK-50)

backlog/tasks/task-49 - Specializer-SQL-support-builtin-long-tail-similarity-string-subscripts-wave-3.md

@@ -3,10 +3,10 @@ id: TASK-49
 title: >-
   Specializer SQL support: builtin long tail + similarity + string subscripts
   (wave 3)
-status: In Progress
+status: Done
 assignee: []
 created_date: '2026-07-26 16:45'
-updated_date: '2026-07-26 17:44'
+updated_date: '2026-07-26 18:15'
 labels: []
 milestone: m-7
 dependencies:

backlog/tasks/task-50 - Specializer-SQL-support-join-forms-—-USING-residual-ON-semi-joins-comma-rewrite-wave-4.md

@@ -0,0 +1,47 @@
+---
+id: TASK-50
+title: >-
+  Specializer SQL support: join forms — USING, residual ON, semi joins, comma
+  rewrite (wave 4)
+status: In Progress
+assignee: []
+created_date: '2026-07-26 18:15'
+updated_date: '2026-07-26 19:24'
+labels: []
+milestone: m-7
+dependencies:
+  - TASK-49
+documentation:
+  - docs/superpowers/specs/2026-07-25-sql-specializer-design.md
+type: feature
+ordinal: 44000
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+Census-itemized (2026-07-26, post-wave-3: 71 join-blocked cases). STAGE A — no new execution model (~43 cases): JOIN USING desugar incl. chains and repeated statics (8), all-key/semi joins by lifting the no-value-columns restriction (4), star expansion over joined tables incl. key columns reconstructed from the dynamic side (2), comma-join rewrite to INNER equi-probe for dynamic-x-static with equi WHERE conjuncts + residual WHERE (13), equi-ON with residual predicates preserving 0-or-1 multiplicity for INNER and LEFT (12: ON l.id=r.id AND l.val>1 / AND true / AND false / constant equalities), cross join to a PROVABLY 1-row static (4). STAGE B (design gate — present before building): output multiplicity (Emit-continuation in the IR + both backends) for pure non-equi ON and range comma-joins (~20 cases). DESCOPED: dynamic self-joins (~9, needs batch-as-build-side), FULL OUTER (1, build-driven output), NATURAL (2, deeper-blocked by COLUMNS()). Pins fleet FIRST: USING output-column semantics (dedup, qualification), LEFT-join residual-ON vs WHERE placement, ON constant conditions, comma-join equivalence to cross-then-filter.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [x] #1 Join-form pins measured BEFORE implementation (USING star/dedup/qualification; residual ON semantics on INNER and LEFT incl. constants; comma-join = cross-then-filter equivalence; 1-row cross join) — JSON + spec addendum committed
+- [x] #2 Stage A ships: USING, all-key/semi, star-over-join, comma equi rewrite, equi-ON residuals — both backends, differential green
+- [x] #3 Multiplicity (stage B) gets a written design presented at the stop point — NOT built without explicit go
+- [x] #4 Corpus replay: flips recorded, zero FAILs
+- [x] #5 mise gate-specializer green
+<!-- AC:END -->
+
+## Implementation Notes
+
+<!-- SECTION:NOTES:BEGIN -->
+Pins committed (7bca38c) BEFORE implementation. Fleet: 3 agents, 223k tokens. Headlines: comma == INNER probe bit-exactly (rewrite safe, WHERE conjuncts split freely); USING merged col stays at LEFT position with LEFT value (exact under Inner/Left), both quals addressable, chained USING binds merged, case-insensitive even quoted, USING-after-ON must ERROR; LEFT keeps residual-failing key-matches with all-NULL side, AND false => all-NULL side; ERROR-EAGERNESS trap: single-side residuals are scan-pushed (eager traps in duck) while both-sides are pair-lazy => v0 requires single-side residuals conservatively trap-free, both-sides may trap; cross-to-1-row = empty-keys probe with compile-time 1-entry check (0-row annihilation is free); duplicate star output names survive to Arrow => our typed model cannot represent them, SELECT * both-keys shapes stay honestly unsupported (revised flip estimate ~30-37, not 43). STAGE-A DESIGN FROZEN: JoinSpec += residual SExpr (hit-guarded lowering, match = hit AND residual), SKind::JoinHit for key-column reconstruction, empty val_cols allowed, empty keys = cross with materialization check, USING desugar in bind_from with using-col star suppression. NO IR changes.
+
+STAGE A COMPLETE. Corpus 383 -> 395 of 678, zero FAILs (deterministic); gate green (753 + 13 xfail); 15 new join oracle tests all green first run. Join-blocked pool 71 -> 13: only dynamic self-joins (10), NATURAL (2, deeper-blocked by COLUMNS()), FULL OUTER (1) remain — the other 58 either match or reclassified to their TRUE deeper blocker (aggregation, NULL-in-value-column for the nulltable cross cases, non-equi -> the cross path where multi-row statics reject via duplicate-map-key, which is CORRECT: a 1-row static would genuinely serve). Implementation: bind_on equality/residual split with the scan_residual totality/sidedness walk; hit-guarded residual lowering threads probe value lanes through block params + the live stack (strict SSA violation caught by corpus replay, fixed); USING desugar with merged-column suppression; key_lane reconstruction (INNER = key expr, LEFT = CASE JoinHit); all-key semi joins via one-side-empty map statics (grammar + verifier extended, round-trip fuzz green); comma rewrite consuming WHERE equi conjuncts with leftover-WHERE reconstruction; UBIGINT/HUGEINT static payloads reject cleanly. STAGE B WRITTEN DESIGN (not built, per AC #3): (1) Term::EmitTo{then,args} — emit current out-row, continue at block (plain Emit stays as row-end); (2) interp: push row + jump; cranelift: emit becomes an h_emit(cx) helper call (per-row ABI return then means only row-done); (3) lowering wraps the row program in a loop over static entries (block-param induction + a ProbeAt inst reading entry idx), residual evaluated per entry; INNER emits per match, LEFT tracks any-match and emits the NULL-side row after the loop; (4) contract change: output multiplicity becomes data-dependent — batch-level API only, breaks SpecializedTransform 1:1 row mapping; (5) cost O(|static|) per row. Unlocks ~13 non-equi/self-adjacent cases + N-row cross.
+<!-- SECTION:NOTES:END -->
+
+## Final Summary
+
+<!-- SECTION:FINAL_SUMMARY:BEGIN -->
+Wave-4 stage A complete: JOIN USING (merged-column semantics measured and mirrored — left position, left value, both quals addressable, chained binding, silent dedup), residual ON predicates on INNER and LEFT (match = key_hit AND residual, hit-guarded per-pair evaluation exactly matching DuckDB's measured laziness; single-side residuals restricted to trap-free shapes because DuckDB scan-pushes those), all-key/semi joins (one-side-empty map statics with text-format round-trip), star expansion over joined tables with key-column reconstruction (INNER = key expr, LEFT = CASE JoinHit THEN key ELSE NULL), comma-join rewrite to INNER probes with WHERE-conjunct key consumption, and cross join to 1-row statics via empty-key maps (0-row annihilates; multi-row rejects via the duplicate-key check). Corpus 383 -> 395 of 678, zero FAILs; join-blocked pool 71 -> 13 (self-joins, NATURAL, FULL OUTER — all named). Stage B (output multiplicity via Term::EmitTo + per-entry static scan) delivered as a written design, not built, per the design gate.
+<!-- SECTION:FINAL_SUMMARY:END -->

docs/superpowers/specs/2026-07-26-wave4-join-pins.md

@@ -0,0 +1,72 @@
+# Wave-4 join-form pins — DuckDB 1.5.5, measured 2026-07-26
+
+The implementation contract for TASK-50 stage A. Three probe agents,
+native tables, full pin tables in `pins-wave4/*.json`. Nothing inferred
+from documentation.
+
+## Comma joins / cross / star (pins-wave4/comma_cross_star.json)
+
+- `FROM t, u WHERE t.k = u.k` is bit-identical to `INNER JOIN ON` (rows
+  AND star column order); WHERE conjuncts split freely into equi keys +
+  residuals; three-way comma == nested INNER; the comma→probe rewrite is
+  safe. Row ORDER of cross products is an artifact (flips with the
+  projection shape — measured) — multiset comparison is mandatory.
+- Cross join to a 1-row static: output rowcount == driving rowcount; a
+  0-row static annihilates (INNER semantics) — so an empty-keys probe
+  with a compile-time exactly-one-entry check is exact for 1 row and a
+  clean reject beyond. LEFT JOIN ON TRUE to an empty table keeps rows.
+- Star expands FROM-order, declared column order, duplicate names KEPT
+  verbatim into the Arrow schema (`[id, lv, id, rv]`) — our typed output
+  model cannot hold duplicate field names, so duplicate-output-name star
+  shapes stay cleanly unsupported (documented model constraint).
+- `FROM a, a a2`: bare `a` binds the unaliased occurrence (aliasing
+  normally shadows the base name entirely). Schema-qualified same-name
+  statics (`s1.t1, s2.t1`) bind per-schema — out of v0's bare-name
+  static catalog, stays unsupported.
+
+## USING (pins-wave4/using_desugar.json)
+
+- `USING (a)` == `ON t1.a = t2.a` row-wise for INNER and LEFT; NULL keys
+  never match.
+- Star: the merged column appears ONCE, at the LEFT table's declared
+  position (NOT hoisted to the front — refutes the PostgreSQL habit);
+  USING-list order is irrelevant; output spelling = the LEFT table's
+  declared capitalization; matching is case-insensitive EVEN QUOTED.
+- Merged value = COALESCE(left, right) ≡ the LEFT value under
+  INNER/LEFT-only — bit-exact simplification. `t1.a` and `t2.a` both
+  stay addressable; `t2.a` is NULL on a LEFT miss (not coalesced).
+- Chained `USING (a)` binds the MERGED column (≡ left under our kinds);
+  `USING (a)` after a prior ON join that left duplicate visible `a`s is
+  a binder AMBIGUITY error — never silently pick a side.
+- `USING (a, a)` silently dedupes; USING a col missing on one side has
+  side-specific error texts (left checked first); same static joined
+  twice under different aliases is legal.
+
+## Residual ON predicates (pins-wave4/equi_on_residuals.json)
+
+- INNER: ON residual ≡ WHERE (rows and columns) — the rewrite target.
+- LEFT: a key-matching row whose residual fails SURVIVES with an
+  all-NULL right side (incl. the right key column) — ON filters matches,
+  never rows. `AND false` ⇒ every driving row with NULL side; `AND
+  true` no-op; `ON NULL = 2` binds fine (INNER empty / LEFT all-NULL).
+- Match rule for the probe plan: `match = key_hit AND residual` with
+  3VL residual collapse (NULL ⇒ non-match).
+- ERROR EAGERNESS (the trap): DuckDB pushes SINGLE-side residuals into
+  that side's scan — a trapping residual fires on rows/build-entries
+  whose key never matches. BOTH-sides residuals are lazy per candidate
+  pair (both kinds) — identical to our hit-guarded evaluation. v0 rule:
+  single-side residual conjuncts must be conservatively TRAP-FREE
+  (columns, literals, comparisons, IS NULL, logic) or the join rejects
+  by name; both-sides residuals may trap (evaluated per key-matched
+  pair, matching DuckDB's laziness bit-exactly).
+
+## Stage-A scope consequences
+
+- All-key statics: lift the no-value-columns rejection (empty probe dst
+  list); join hit becomes first-class in the plan (SKind::JoinHit) for
+  key-column reconstruction: `r.id` ≡ CASE match THEN dyn-key ELSE NULL.
+- Expected non-flips recorded honestly: `SELECT *` shapes that produce
+  duplicate output names (both-keys star, self comma joins) and
+  schema-qualified statics stay cleanly unsupported.
+- Stage B (output multiplicity — pure non-equi ON, range comma joins,
+  N-row cross) is design-gated: written proposal before any build.