Stage B part 1: join multipliplicity under shape='many' — corpus 529 -> 546 (TASK-59)

backlog/tasks/task-59 - Stage-B-join-multiplicity-under-shapemany-—-dup-key-cross-inequality-and-self-joins.md

@@ -0,0 +1,48 @@
+---
+id: TASK-59
+title: >-
+  Stage B: join multiplicity under shape='many' — dup-key, cross/inequality, and
+  self-joins
+status: In Progress
+assignee: []
+created_date: '2026-07-28 01:55'
+updated_date: '2026-07-28 01:57'
+labels:
+  - specializer
+  - stage-b
+dependencies: []
+type: feature
+ordinal: 53000
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+User go received 2026-07-28 ("feel free to implement the feature gap"), sequenced behind the TASK-58 shape flag: every stage-B construct builds ONLY under shape='many' (0..N rows out per row in) — the serving fence the user asked for.
+
+Constituency (census, pins-waveA/census-all-nonmatches.json): 27 cases = 17 'duplicate map key' (true 1:N equi-joins vs statics with repeated keys, PLUS keyless shapes that reduce to one bucket: comma/cross joins with range residuals, inequality ON predicates) + 10 self-joins (the batch as both probe and build side; mostly EXCLUDE/USING star semantics). Target: corpus 529 -> ~556 of 678 (82%). After this ships, everything left is out-of-scope-by-decision.
+
+Pins-first (fleet): 1:N emission ORDER and determinism, LEFT-miss null-extension under multiplicity, cross/inequality semantics, self-join star expansion + dup-name renames + USING merges, NULL keys among duplicate keys. Design per TASK-50 notes: per-key row lists in the frozen maps, inner emit loop per probe hit, bucket scan + residual predicates for keyless joins, per-call batch-side map build for self-joins. Backend decision (interpreter-first vs cranelift emit loops) goes in the spec, justified.
+
+Hard stop after shipping: columnar-API discussion with the user before any next wave.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [ ] #1 Pins spec for multiplicity semantics (emission order/determinism, LEFT-miss, cross/inequality, self-join star forms, NULL keys among dups) with raw JSONs
+- [ ] #2 All stage-B constructs REJECT under shape='filter'/'map' exactly as today; they build only under shape='many'
+- [ ] #3 1:N equi-joins, cross/inequality joins, and self-joins serve bit-exact (order-insensitive multiset vs DuckDB) under shape='many'
+- [ ] #4 Corpus replay strictly above 529 with zero FAILs (replay builds stage-B cases with shape='many')
+- [ ] #5 Limitations doc + twin updated: stage-B rows move from 'designed, not built' to the shape='many' contract
+- [ ] #6 Full gates green on release build; PR opened (stacked on #44)
+<!-- AC:END -->
+
+## Implementation Plan
+
+<!-- SECTION:PLAN:BEGIN -->
+Emit-machinery study (pre-spec, 2026-07-28):
+
+TODAY: both backends are strictly one-emit-per-row. Interpreter: per row, block walk ends at CTerm::Emit (emitted += 1, output values already written to st.out by instructions) or CTerm::Skip. Cranelift: compiled row_fn returns 0=emitted / 1=skip / 2+=trap — the RETURN CODE is the emit signal, so N-per-row cannot be expressed by return codes.
+
+DESIGN DIRECTION for multiplicity (validate in spec): (1) emission becomes an explicit operation instead of a terminal side effect — either Inst::EmitRow (appends the current out-lane values) with CTerm::Emit demoted to plain end-of-row, or loop-structured blocks using the EXISTING Brif/Jump machinery (loop header + body + back-edge) with an EmitRow inst in the body. Verify/canonicalize must accept back-edges (check: today's block graph is probably a DAG — verify.rs may assume forward-only; if so that is the first thing to extend). (2) New probe ops: ProbeStart {join} -> cursor, ProbeNext {join, cursor} -> (has: i1, cursor', value lanes) over per-key row LISTS in StaticTy::Map (values become Vec-of-rows per key — flat arena + (off, len) per key is the lean layout). (3) Cranelift: emission via a helper call h_emit_row(cx) that appends to st.out (helpers already write through Cx), loop via normal CLIF blocks — feasible without new architecture; the return code stays for trap/end only. If this turns out heavy, interpreter-first for shape='many' with cranelift fallback is the documented fallback plan (bench guard exempts 'many' from backend-identity tests). (4) Self-join: per-call build of the batch-side map before the row loop (new PreparedStatic variant built in run(), or a batch-map handed via RunState); marshaller unaffected. (5) Cross/inequality: a join with ZERO key columns = single bucket = ProbeStart/Next over the whole static (or batch) + residual predicate in the loop body. (6) LEFT: emit null-extension when the loop body emitted nothing for this row (a per-row 'any_emitted' flag register). (7) shape gate: prepare learns the target shape (prepare_opaque param or Prepared metadata + duckdb/mod.rs check): multiplicity constructs produce a NAMED reject unless shape='many'; corpus replay builds stage-B-shaped cases with shape='many' (detect by retrying on the named errors, or always pass shape='many'? NO — corpus must keep proving the default rejects; retry-with-many on the named needles is the honest harness).
+<!-- SECTION:PLAN:END -->

docs/known-limitations.md

@@ -9,7 +9,7 @@ forces this document to change with it.
 **The contract.** For any SQL you hand it, the engine does exactly one of:
 
 1. **Serve it bit-for-bit identical to DuckDB** (verified continuously
-   against DuckDB's own test corpus: 505 of 678 statements as of wave B), or
+   against DuckDB's own test corpus: 546 of 678 statements as of stage B), or
 2. **Refuse loudly at BUILD time** — `DuckDBInferFn(...)` raises a
    `ValueError` naming the construct. Nothing is ever silently wrong or
    silently dropped at inference time.
@@ -30,8 +30,8 @@ rejected, permanently by design:
 |---|---|---|
 | Regex patterns must be constants (`regexp_matches(s, pattern_col)` rejects) | `unsupported: non-constant regex pattern (compiled at prepare in v0)` | Regexes compile at prepare; DuckDB compiles per row. Per-row compilation is the opposite of specialization. |
 | Replacement strings / regex options / extract group indexes must be constants | `non-constant regexp_replace replacement` etc. | Same. |
-| Static (join) tables must be provided at build time, with unique keys | `duplicate map key` | Joins are frozen hash maps baked into the function. Duplicate keys mean 1:N join multiplicity — a designed extension (stage B, TASK-50 notes) that is deliberately not built yet. NULL *values* serve since TASK-55 (they flow through as NULL); NULL *keys* drop the row, matching equi-join semantics. |
-| The dynamic table cannot be joined to itself | `joining the dynamic table to itself` | The batch is the probe side; using it as a build side too needs stage-B machinery. |
+| Static (join) tables must be provided at build time; under the DEFAULT shapes their keys must be unique | `duplicate map key` | Joins are frozen hash maps baked into the function. Duplicate keys mean 1:N join multiplicity, which SERVES under the opt-in `shape='many'` (TASK-59) — along with cross joins, inequality `ON` predicates, and constant `ON` clauses — with multiset parity vs DuckDB (its join output order is a measured hash-join accident; the engine emits probe order outer, build insertion order inner). Under `filter`/`map` the 1:1 contract is unchanged. NULL *values* serve since TASK-55; NULL *keys* never match. |
+| The dynamic table cannot be joined to itself | `joining the dynamic table to itself` | The batch is the probe side; using it as a build side too is the one stage-B piece still in progress (it will also require `shape='many'`). |
 | Exactly one row table drives the query | `the specializer takes exactly one row table`, `must be the dynamic table` | The serving contract is rows-in → rows-out for one entity stream. |
 
 ## 2. Out of scope for row-serving (by decision, not difficulty)
@@ -42,10 +42,11 @@ build time. `"filter"` (the default) is the engine's native 0..1;
 `"map"` statically PROVES exactly-one (`out[i] ↔ in[i]`, the strict
 serving guarantee) by rejecting anything that can drop a row — a WHERE
 clause, an INNER join (key misses drop), a static-tables-only constant
-query; `"many"` (0..N) is reserved for join multiplicity (stage B) and
-will be the only shape under which duplicate-key joins, cross/inequality
-joins, and self-joins ever build — multiplicity can never sneak into a
-serving path by default.
+query; `"many"` (0..N) is the multiplicity opt-in (stage B): duplicate-key
+joins, cross joins, and inequality/constant `ON` joins build ONLY under
+it (one join per query for now, named restriction) — multiplicity can
+never sneak into a serving path by default. Self-joins are still
+rejected under every shape (in progress).
 
 The engine serves **row-at-a-time feature transforms**. Whole-relation
 constructs are out of scope because their output shape is not

docs/proposals/2026-07-28-columnar-path.md

@@ -0,0 +1,124 @@
+# Proposal: own the columnar path
+
+Status: **for discussion** (user + engine). Blocked work: none — this is
+the candidate *next* wave after stage B. Decision owner: AmirHossein.
+
+## Why (measured, not vibes)
+
+TASK-57 closed with a decomposition of where serving time actually goes
+(titanic scenario, 10 input cols, 31 output cols, release build, p50):
+
+| component | cost |
+|---|---|
+| whole boundary floor (trivial 1-col query, incl. 10-col ingest) | ~262 ns/row |
+| output emission | ~37 ns per output column → ~1.15 µs at 31 cols |
+| compute (the compiled program) | ~1.7 µs/row |
+| handcrafted python twin — EVERYTHING | ~2.2 µs/row |
+
+Two conclusions. First, the input side is already near-free — a `__dict__`
+fast path benched **neutral** and was deleted. Second, the remaining
+overhead vs the twin is *per-Python-object work*: every output value is
+boxed into a fresh PyObject, every output row into a dict/model, while the
+twin pointer-copies passthrough fields. No row-at-a-time API can remove
+that floor — it is the API, not the implementation.
+
+Meanwhile the engine is **already columnar inside**: `ColData` lanes in,
+`OutCol` vectors out. The row boundary is a conversion layer we bolt onto
+both ends of a columnar core. The proposal is to stop converting.
+
+## The proposal (option A, recommended)
+
+```python
+fn = DuckDBInferFn(sql, row_tables=..., static_tables=..., shape="map")
+
+out: pa.Table = fn.infer_arrow(batch: pa.Table)   # arrow in, arrow out
+```
+
+- Input: one `pa.Table` (or `RecordBatch`) whose columns map 1:1 onto the
+  engine's lanes (int64/float64/utf8/bool + validity = exactly our four
+  types + null lanes). Ingest = buffer walks, zero per-value PyObjects.
+- Output: arrow arrays built directly from `OutCol` vectors — one
+  allocation per *column*, not per value.
+- The row APIs (`infer`, `infer_rows`, pydantic models) stay untouched.
+  This is an additive fast lane, not a migration.
+
+### Why it composes with everything we just built
+
+- **`shape="map"` is the natural contract here**: `out[i] ↔ in[i]` means
+  the output table aligns positionally with the input table — the exact
+  guarantee vectorized feature pipelines want. `shape="filter"`/`"many"`
+  still work (output row count differs; optionally expose a
+  `__row_index__` output column so callers can re-align — decision below).
+- **The multi-language thesis**: the WASM spike proved one Rust build runs
+  near-native in Go (wazero) and Java (chicory) with batching amortizing
+  the boundary. Arrow IPC as the wire format + this columnar entry point
+  is that story's missing half — the same engine serves Python, Go, and
+  Java with no per-language marshalling code.
+
+### Expected win (to be verified by bench, not promised)
+
+Eliminates the ~1.4 µs/row of boxing/emit at titanic's width and the
+per-row call machinery; the remaining cost is compute. That should put
+`spec` decisively **ahead** of the handcrafted python twin at every batch
+size — the twin cannot go columnar without becoming numpy-vectorized
+code, which is exactly the authoring burden this library exists to remove.
+
+## Costs and risks
+
+- **Dependency**: arrow ingestion in Rust. Two routes: `arrow-rs` (heavy
+  crate, full ecosystem) vs the Arrow **C Data Interface** via pyo3
+  (lean — a few hundred lines for our four types + validity). Lean route
+  recommended; we only need 4 primitive layouts + utf8 offsets.
+- **Strings**: input utf8 can be read zero-copy from arrow buffers;
+  output strings copy once from the arena into an arrow buffer (already
+  cheaper than one PyUnicode per value).
+- **Chunked tables**: accept only single-chunk input at v0 (`combine_chunks()`
+  is the caller's one-liner) — named rejection otherwise.
+- **API surface pre-1.0**: v0-no-compat makes this cheap to adjust later.
+- **Bench honesty**: the comparison baseline for `infer_arrow` must be a
+  *vectorized numpy/pandas twin*, not the per-row python twin — new bench
+  rows, same three-way parity gate.
+
+## Open questions for the discussion
+
+1. **Entry point shape**: `infer_arrow(pa.Table) -> pa.Table`, or also
+   accept/return a dict of numpy arrays? (Arrow-only keeps one code path;
+   numpy is a `pa.table(...)` call away for callers.)
+2. **Non-map shapes**: for `"filter"`/`"many"`, is an output-only
+   `__row_index__` column the right re-alignment tool, or do we return
+   `(table, indices)`?
+3. **Zero-copy ambition**: v1 copies input buffers into `ColData` (simple,
+   already a big win) vs true zero-copy lanes reading arrow buffers in
+   place (bigger surgery in exec). Recommendation: copy first, measure,
+   then decide if zero-copy is worth it.
+4. **Does the columnar path also want `output_model`-style typed schemas**
+   (arrow schema declaration up front), or is the derived schema enough?
+5. **Multi-language sequencing**: is Python-only columnar the first ship,
+   with the WASM/arrow-IPC endpoint as its own later proposal?
+
+## Measured: us vs DuckDB-on-pyarrow (2026-07-28, titanic, p50 per call)
+
+DuckDB gets a PRE-BUILT arrow table each call (register+execute+fetch
+arrow = serving-realistic; "floor" = re-execute on an already-registered
+table, its absolute best). We run today's ROW path (spec_dict) — an upper
+bound for any columnar path of ours.
+
+| n/call | duckdb | duckdb floor | us (row path, today) | us vs duckdb |
+|---|---|---|---|---|
+| 1 | 6.58 ms | 5.52 ms | 3.3 µs | **2055× faster** |
+| 8 | 6.28 ms | 5.86 ms | 24 µs | **265×** |
+| 64 | 6.75 ms | 5.94 ms | 206 µs | **33×** |
+| 1024 | 7.75 ms | 6.93 ms | 3.42 ms | **2.3×** |
+| 16384 | 20.4 ms | 18.7 ms | 60.5 ms | 0.3× (duckdb wins) |
+| 131072 | 105 ms | 98 ms | 543 ms | 0.2× (duckdb wins) |
+
+Reading: DuckDB pays ~5.5–7 ms of per-QUERY cost every call regardless of
+batch size — we pay it once at build. The serving regime (1–1k rows/call)
+is ours by 1–3 orders of magnitude ALREADY, on the row path. Today's
+crossover is ~2–3k rows/call. The columnar path moves that crossover, not
+the small-batch story: boundary-only (compute stays ~1.7 µs/row) puts
+16k-row calls at ~29 ms vs DuckDB's 20 ms (near-tie); a vectorized
+columnar CORE (2–3× on compute) would put the crossover at ~100k+
+rows/call — competitive everywhere except true analytic scans, where
+DuckDB's parallelism should keep the crown (and that's fine: that's not
+serving).

docs/superpowers/specs/2026-07-28-stageB-multiplicity-pins.md

@@ -0,0 +1,72 @@
+# Stage B: join multiplicity under shape='many' — pins + design (TASK-59)
+
+Measured against DuckDB 1.5.5, 2026-07-28. Raw pins: `pins-stageB/*.json`
+(5 fleet agents, 57 pins). Gate: every construct here builds ONLY under
+`shape='many'` (TASK-58); under `filter`/`map` the rejections stay exactly
+as today.
+
+## The row-SET contract (stable everywhere — this is what we serve)
+
+1. **Inner equi-join**: one output row per (left row, matching right row)
+   pair; full per-key cross product when BOTH sides have duplicate keys
+   (2×3 = 6 verified).
+2. **LEFT**: per left row, `max(1, surviving matches)` rows — exactly one
+   null-extended row when zero matches survive, INCLUDING when a residual
+   ON predicate filtered every match away.
+3. **NULL keys match nothing** — under `=`, `>`, and crucially `<>`
+   (NULL <> NULL is NULL, not TRUE). A NULL-keyed left row = unmatched:
+   dropped by INNER, one null-extension by LEFT. (The entire 9-case
+   left_join_issue_1172 corpus family reduces to this — probe is NULL, so
+   every case is a single null-extended row; the dup keys are inert.)
+4. **Residual ON predicates** filter per match-pair; WHERE composes
+   orthogonally (left rows removed first).
+5. **Keyless joins** (comma, CROSS, `ON` inequality / one-sided /
+   constant): exactly cross-product-then-filter, verified bit-for-bit.
+   `ON NULL = 2` → zero matches (LEFT: null-extends every left row).
+   Empty right side: INNER → 0 rows, LEFT → all null-extended.
+6. **USING** merges the key into ONE leading output column carrying the
+   LEFT value (visible on null-extended rows: the probe key, not NULL).
+   With ON, both copies appear; the model/dict boundary applies the
+   wave-5 dedup rename (`id, id_1`) — same convention as DuckDB's own
+   `.df()`.
+7. **Self-join stars**: `SELECT *` = left cols then right cols in
+   declaration order; unqualified EXCLUDE strips EVERY copy; qualified
+   EXCLUDE strips one side's (USING survivor's position depends on which
+   copy went — pinned); excluding all of one side's star is legal; bare
+   names (and rowid) are ambiguity errors.
+
+## ORDER: an accident, not a contract (the fleet's central finding)
+
+DuckDB's join output order is a hash-join artifact on three independent
+axes: the optimizer picks the streamed side by cost (not FROM order, not
+reliably size), dup-key matches emit in REVERSE build-insertion order
+(LIFO chains) in per-2048-row lockstep passes, and at threads>1 with
+~500k+ rows the order differs RUN-TO-RUN on the same connection. Even
+single-threaded, one probe row's matches land ~2048 rows apart — a
+row-at-a-time engine cannot naturally reproduce it and must not try.
+
+**Decision**: parity for shape='many' is **multiset** (sort both sides
+before comparing — the corpus already does; duck_check gains a sorted
+mode for 'many' tests). The engine defines its OWN deterministic order,
+documented: probe rows in input order; all matches for one probe row
+contiguous; matches in build INSERTION order; the LEFT null-extension
+in place. This intentionally diverges from DuckDB's accidental order —
+SQL without ORDER BY promises none, and we reject ORDER BY anyway.
+
+## Engine design (validated against the emit-machinery study, TASK-59 plan)
+
+- Emission becomes explicit: `Inst::EmitRow` appends the out-lane values;
+  `CTerm::Emit` retires to end-of-row. Loops use the existing Brif/Jump
+  blocks with back-edges (verify must accept them — first change).
+- `StaticTy::Map` values become per-key row LISTS (flat arena +
+  (offset, len) per key). New ops `ProbeStart`/`ProbeNext` iterate them;
+  keyless joins are the degenerate one-bucket scan with the residual in
+  the loop body; LEFT keeps an `any_emitted` flag and emits the
+  null-extension when the loop closes dry.
+- Self-joins build the batch-side map per call before the row loop.
+- Cranelift: emission via an `h_emit_row` helper (helpers already write
+  through Cx), loops as normal CLIF blocks; the row_fn return code stays
+  for end/trap only. Fallback if heavy: interpreter-first for 'many'
+  (documented, backend-identity tests exempt 'many').
+- Corpus harness: replay retries with shape='many' ONLY on the named
+  multiplicity rejections, so the default-shape rejections stay proven.