Stage B part 1: join multipliplicity under shape='many' — corpus 529 -> 546 (TASK-59) - #45
Merged
Merged
Conversation
ahrzb
approved these changes
Jul 28, 2026
…ge-B discussion) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, engine-defined order (TASK-59) Five-agent fleet, 57 pins. Row-SET contract: per-pair emission with full per-key cross products; LEFT = max(1, surviving matches) incl. the all-matches-filtered null-extension; NULL keys match nothing (even <>); keyless joins are cross-then-filter; USING carries the LEFT value on misses. Central finding: DuckDB's join output ORDER is a hash-join accident (LIFO chains, lockstep vector passes, run-to-run divergence at scale) — parity is MULTISET, and the engine defines its own documented deterministic order. The 1172 family turns out to be pure NULL-semantics (no fan-out at all). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…me ours by 1-3 orders, crossover ~2-3k rows/call today Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tics, range probes (TASK-59 stage B-1)
- Term::EmitTo { to, args }: emit the completed row AND continue at a
block — the multiplicity loop's back-edge. Verifier: cycles are now
legal iff every block still reaches a row-ENDING terminator
(emit/skip/trap), which keeps rejecting degenerate spin loops; the
store dataflow excludes back-edges from its topo order and EmitTo
propagates the RESET all-zero store state to the loop header.
- StaticTy::MultiMap + Inst::ProbeRange/ProbeRead: duplicate-key maps
probed as a flat equal-key row range (stable sort keeps INSERTION
order among equal keys — the engine's documented 1:N emission order);
zero keys = the whole table (keyless cross/inequality joins).
- Interpreter executes all of it; cranelift pre-rejects multiplicity
programs so the existing interp fallback is the documented 'many'
path. Print/parse roundtrip (multimap header, probe.range/probe.read,
emit.to) + two machinery fixtures: dup-key fan-out with skip-on-miss,
and the keyless cross join.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nder shape='many' (TASK-59 stage B-2) lower() gains many: bool (one join per query in stage B, named restriction). The join lowers as a multiplicity loop: entry evaluates keys (NULL keys force an empty range) + ProbeRange; header/body iterate the equal-key run with ProbeRead lanes seeded into the block probe cache; the RESIDUAL gates match-ness (and the LEFT null-extension flag) while WHERE only gates emission — a match killed by WHERE still suppresses the null-extension, per DuckDB's join-then-filter order; the back-edge is emit.to. LEFT emits one null-extended row when nothing matched, itself WHERE-gated with default lanes. The multimap static declaration allows duplicate keys at materialization ONLY on this path; the default shapes keep the 1:1 map contract byte-identical (dup keys still error). E2E interp tests cover fan-out order, INNER drops, residual-filters-all, WHERE composition, and both named restrictions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…retry — 529 -> 546, 0 FAIL (TASK-59 stage B-3) Predicate/expression emission may SPLIT blocks (CASE machinery), so the multiplicity loop now rides its state and probe lanes on the LIVE stack (auto-rebound across splits; lanes are always the trailing entries) and re-seeds the per-block probe cache before every emission that can reference join columns — fixes the three corpus binder failures. The keyless path needed nothing: wave-4's empty-key binding already covers cross joins, inequality ON, and constant ON under 'many' (test added). Corpus replay retries 'duplicate map key' rejections with shape='many' (the default's rejection stays proven); fn.shape now reports 'many'. Corpus: 546 match / 132 clean-unsupported / 0 FAIL of 678 (was 529 — the full dup-key + cross/inequality bucket). Self-joins are B-4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ns doc/twin, corpus 546 (TASK-59 stage B-5) - tests/test_duckdb_stageb_many.py: sorted-multiset oracle tests (dup-key fan-out, residual/WHERE combos, cross + inequality + constant ON) plus the engine's OWN order contract pinned as a test. - known-limitations §1/§2: dup-key/cross/inequality joins now documented as serving under the shape='many' opt-in (multiset parity, engine- defined order); self-join row marked in-progress; corpus count 546. - Twin + shape tests flipped: dup keys serve under many, reject under the defaults; fn.shape == 'many'. Full gate: cargo 167, pytest 616 + 13 xfail, corpus 546/0 FAIL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
claude-agent-ahrzb
Bot
force-pushed
the
task-59-stage-b
branch
from
July 28, 2026 03:13
4960b50 to
3172a45
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Join multiplicity, behind the shape fence
Every construct here builds ONLY under the explicit
shape='many'opt-in (PR #44's fence) — the default shapes are byte-identical to before, dup keys still reject.What serves now (pins:
2026-07-28-stageB-multiplicity-pins.md, 57 pins / 5 agents)ON, constantON— the keyless one-bucket range (wave-4's empty-key binding meant this was nearly free).The order finding (fleet-measured)
DuckDB's join output order is a hash-join accident — LIFO chains in lockstep vector passes, flips with table size or projection, non-deterministic run-to-run at 200k+ rows with threads. So parity is multiset (sorted compare; the corpus already sorts), and the engine ships its own documented deterministic order: probe rows in input order, matches contiguous in build insertion order.
Machinery
Term::EmitTo(emit-and-continue back-edges; the verifier now allows terminating cycles only),StaticTy::MultiMap+ProbeRange/ProbeRead(stable-sorted equal-key runs), loop-structured lowering riding state on the live stack across block splits. Interpreter-only: cranelift pre-rejects multiplicity programs into the existing fallback.Not in this PR
Self-joins (10 corpus cases) — the batch-as-build-side piece, next PR on this ticket. The
left_join_issue_1172family turned out to be pure NULL-semantics (measured), so it landed here.Gate
Corpus 546 / 132 / 0 FAIL of 678 (was 529). cargo 167, pytest 616 + 13 xfail. Also carried: the columnar-path decision doc + measured us-vs-DuckDB-on-arrow numbers.
🤖 Generated with Claude Code