Skip to content

linq_fold: array _join splice → m3f join_* parity (chunk N+3) - #2913

Merged
borisbat merged 5 commits into
masterfrom
bbatkin/linq-fold-array-join-splice
May 28, 2026
Merged

linq_fold: array _join splice → m3f join_* parity (chunk N+3)#2913
borisbat merged 5 commits into
masterfrom
bbatkin/linq-fold-array-join-splice

Conversation

@borisbat

Copy link
Copy Markdown
Collaborator

Summary

  • Closes the m3f-vs-m4 parity gap on the entire join_* bench family. Mirror of PR linq_fold: plan_decs_join + LinqJoin auto-typed result lambda (Session 3) #2848 (plan_decs_join) + PR linq_fold: Theme 3 Phase 1 — cross-arm join+group_by splice (C3) #2861 (_join |> _group_by cross-arm) for arrays. Before this PR, every _fold(arr |> _join(...) ...) chain fell through to tier-2 cascade (join_impl at daslib/linq.das:1674) — 3 intermediate allocations, per-pair invoke overhead, m3f 2-3× slower than m4 across the board. Now m3f beats m4 on all 5 cells in both INTERP and JIT.
  • Phase 1 — new register_array_join_rows pattern + emit_array_join produces hashB-collect + srcA-probe inline (preserves the count-no-where bucket-length fast path; mirrors PR linq_fold: plan_decs_join + LinqJoin auto-typed result lambda (Session 3) #2848's D2-A guarantee). 2-source invoke wrap (like Zip); primitive equi-key gate via existing is_primitive_join_key_type; non-primitive keys + decs-bridge srcb cascade silently.
  • Phase 2 — new ArrayJoinShape struct + SourceAdapter.ArrayJoin variant. Extends 4 adapter helpers (adapter_bind_name, adapter_element_type, adapter_wrap_source_loop, adapter_wrap_invoke) + allocation prefix in plan_group_by_core. Adds upstream_join slot to group_by_array row + array_join_invariants requires predicate. plan_group_by_core is fully reused — ArrayJoin flows through the adapter abstraction the same way DecsJoin does.

Bench (full INTERP + JIT rerun, n = 100 000)

Cell m3f INTERP before / after m3f JIT before / after m4 INTERP m4 JIT
join_count 128.2 → 51.0 34.6 → 11.6 63.6 12.7
join_select 148.1 → 72.0 41.6 → 19.7 83.8 22.2
join_where_count 148.5 → 60.1 41.2 → 20.4 74.3 22.5
join_groupby_count 186.2 → 78.3 47.2 → 19.4 90.1 21.6
join_groupby_to_array 217.8 → 88.1 55.5 → 19.5 91.4 21.6

Drift across the non-join matrix is at measurement floor (±5% INTERP, ±10% JIT). The splice is gated on array_source + primitive equi-key + (for cross-arm) upstream_join capture, so non-join chains aren't touched.

Known typer-order subtlety

Bare-_select after _join on array sources (no trailing to_array()) leaves selCall._type.firstType as an unresolved typedecl(result_selector(type<TT>)) because nothing forces resolution in the chain — select's array overload returns array<...> directly. emit_array_join extracts the resolved type from the select lambda's body via peel_lambda_single_return instead of cloning selCall._type.firstType. emit_decs_join doesn't hit this because its bench chains universally end with |> to_array() (decs returns iterator-typed bridge). Test coverage includes both bare-select and iterator-chain-with-to_array forms.

Follow-up (next PR, agreed upfront with the author)

emit_array_join and emit_decs_join are ~70% structural duplicates; same for the DecsJoin / ArrayJoin branches of adapter_wrap_source_loop. Deferred to a follow-up refactor PR (extract emit_join_collect_probe_shape helper across all 4 callsites; ~80-100 LOC dedup) — same staging call PR #2848 made: ship the perf-fix at zero risk to the 4 already-closed decs cells, then refactor with both sides bench-locked.

Test plan

  • mcp__daslang__format_file + mcp__daslang__lint clean on every touched .das
  • mcp__daslang__run_test tests/linq/test_linq_join.das — 52/52 (10 new splice-exercising tests appended: 7 Phase 1, 4 Phase 2, 1 non-primitive-key cascade)
  • mcp__daslang__run_test tests/linq/test_linq_from_decs.das — 198/198 (decs splice not regressed)
  • mcp__daslang__run_test tests/linq/test_linq_fold_terminal_select.das — 28/28
  • mcp__daslang__run_test tests/linq/test_linq_fold_theme3_decs_join_groupby.das — 14/14 (PR linq_fold: Theme 3 Phase 1 — cross-arm join+group_by splice (C3) #2861 surface intact)
  • Bench acceptance: full INTERP+JIT rerun across benchmarks/sql/*.das; all 5 join_* m3f cells beat m4
  • mcp__daslang__compile_check benchmarks/sql/join_*.das — all 5 compile under the new splice

🤖 Generated with Claude Code

borisbat and others added 5 commits May 28, 2026 01:10
Mirror of plan_decs_join for arrays. New `register_array_join_rows`
pattern + `emit_array_join` produces hashB-collect + srcA-probe inline,
preserving the count-no-where bucket-length fast path (per PR #2848's
D2-A guarantee on the decs side).

Two sources bind as invoke parameters (mirrors Zip's 2-source wrap
rather than the 1-source Array helper). Primitive equi-key gate via
existing `is_primitive_join_key_type`. Asymmetric "array _join
decs-bridge" cascades via new `array_join_srcb_is_array` predicate.

Supports `_join + count`, `_join + _where + count`, `_join + _where +
to_array`, `_join + _select + to_array`, `_join + _select` (bare-select
on array chain).

Bare-`_select` typer-order subtlety: `selCall._type.firstType` may stay
as unresolved `typedecl(result_selector(type<TT>))` when the chain
doesn't end with `to_array()` (array-overload select returns array
directly, so no enclosing wrap forces resolution). Extract the resolved
type from the select lambda's body via `peel_lambda_single_return`
instead. emit_decs_join doesn't hit this because its bench chains
universally end with `|> to_array()` (decs returns iterator-typed
bridge).

10 splice-exercising tests added to `tests/linq/test_linq_join.das`,
including the bare-`_select` shape that exposed the typer-order
subtlety. All 198 from_decs tests + 47 join tests + 28
terminal_select tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirror of PR #2861's DecsJoin adapter for arrays. Adds `ArrayJoinShape`
struct + `SourceAdapter.ArrayJoin` variant arm, extends 5 adapter
helpers (`adapter_bind_name`, `adapter_element_type`,
`adapter_wrap_source_loop`, `adapter_wrap_invoke`, `plan_group_by_core`
allocation prefix), and wires `upstream_join` slot into the
`group_by_array` pattern row with new `array_join_invariants` requires
predicate.

`plan_group_by_core` requires zero further changes — `ArrayJoin` flows
through the existing adapter abstraction layer the same way `DecsJoin`
does. `adapter_wrap_source_loop`'s new `ArrayJoin` branch performs
hashB-collect + srcA-probe via plain `for` loops (no `for_each_archetype`
wrapping); `adapter_wrap_invoke`'s new `ArrayJoin` branch produces a
2-source invoke (mirrors Zip's wrap shape, not Decs's zero-arg form).

Closes the `_join |> _group_by(_.K) |> _select((K, agg))` array-side
gap. Bench m3f now matches m4 (within ±25% INTERP, ±15% JIT) on
`join_groupby_count` + `join_groupby_to_array`, completing the chunk N+2
parity target across all 5 `join_*` cells.

4 new cross-arm tests appended to `test_linq_join.das` (count, sum, min/
max, empty-srcb). All 14 existing decs `theme3_decs_join_groupby` tests
still pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- `doc/source/reference/linq_fold_patterns.rst`: add "Array-array
  equi-join" section parallel to the existing "Decs-decs equi-join"
  section. Same 5 row shape covering count / implicit-to-array /
  trailing _select / trailing _where / cross-arm _group_by.
- `tests/linq/test_linq_join.das`: add non-primitive-key cascade test
  verifying struct-keyed _join still produces correct results via
  tier-2 cascade (silent splice-bail, no error).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Full INTERP+JIT rerun across benchmarks/sql/*.das. m3f beats m4 on
all 5 join_* cells in both lanes:

| Cell | m3f INTERP before/after | m3f JIT before/after |
|---|---:|---:|
| join_count | 128.2 → 51.0 | 34.6 → 11.6 |
| join_select | 148.1 → 72.0 | 41.6 → 19.7 |
| join_where_count | 148.5 → 60.1 | 41.2 → 20.4 |
| join_groupby_count | 186.2 → 78.3 | 47.2 → 19.4 |
| join_groupby_to_array | 217.8 → 88.1 | 55.5 → 19.5 |

Drift across the non-join matrix is at measurement floor (±5% INTERP,
±10% JIT). The splice is gated on array_source + primitive equi-key +
(for cross-arm) upstream_join capture, so non-join chains aren't
touched. Header refreshed; Notes-on-missing-lanes section unchanged
(no — cells flipped).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 28, 2026 08:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an array-side _join splice in linq_fold mirroring the existing decs-side emit_decs_join, plus a cross-arm _join |> _group_by path via a new ArrayJoin SourceAdapter variant. Closes the m3f-vs-m4 parity gap across all five join_* SQL benchmarks (m3f now beats m4 in both INTERP and JIT).

Changes:

  • New ArrayJoinShape + SourceAdapter.ArrayJoin variant; extends adapter_bind_name, adapter_element_type, adapter_wrap_source_loop, adapter_wrap_invoke, and plan_group_by_core's name prefix.
  • New register_array_join_rows / emit_array_join (Phase 1) with array_join_srcb_is_array guard; new array_join_invariants guard added to group_by_array pattern with optional upstream_join capture (Phase 2).
  • 11 new tests in tests/linq/test_linq_join.das, docs row in linq_fold_patterns.rst, refreshed bench matrix in results.md.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
daslib/linq_fold.das Adds ArrayJoinShape/ArrayJoin adapter, array_join_* guard predicates, splice emit + registration, cross-arm group_by wiring.
tests/linq/test_linq_join.das Adds 11 tests exercising Phase 1 splice arms, Phase 2 join+group_by, and non-primitive-key cascade.
doc/source/reference/linq_fold_patterns.rst Documents the array-array equi-join patterns and the cross-arm group_by composition.
benchmarks/sql/results.md Refreshes the INTERP+JIT benchmark matrix to reflect the new m3f wins on join_* rows.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@borisbat
borisbat merged commit dba2c8a into master May 28, 2026
32 checks passed
pull Bot pushed a commit to forksnd/daScript that referenced this pull request May 28, 2026
The Helper B refactor (qmacro_block_to_array → qmacro_expr inside the
ArrayJoin adapter branch of adapter_wrap_source_loop) happens to drop
an extra ExprBlock layer the original inline qmacro_block produced.
m3f INTERP improves 88.1 → 78.1 ns/op (-11.4%). Other 4 join_* m3f
cells stay at noise floor (±2.5% drift, uniform with m4 — measurement-
floor noise, not refactor-induced).

Refreshes:
- bottom INTERP table cell + ratio (1.04× → 1.17× — m3f beats m4 even
  more decisively now)
- opening "before / after" summary table cell + attribution (chunk N+3
  + dedup follow-up)

JIT lane not re-measured this round — the JIT bench (`bin/daslang -jit
dastest/dastest.das ...`) currently fails with "can't delete locked
array" in clargs at startup, a pre-existing regression on master that
also affects the unmodified baseline (not a refactor-induced break).
The JIT cells reflect PR GaijinEntertainment#2913 values; refactor doesn't touch codegen
logic so JIT codegen runs on a functionally-identical final AST.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@borisbat
borisbat deleted the bbatkin/linq-fold-array-join-splice branch May 30, 2026 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants