sqlite_linq: _distinct_by composition + reverse() MAX(pk) + group_by+first projection

[borisbat]

Summary

Closes 4 SQL — cells from benchmarks/sql/sqlite_linq_gaps.md's "window-function lowerings" group. Plan-mode probing established that 3 of the 4 actually need composition over PR #2906's bare-aggregate trick, not window functions — the gaps doc's ROW_NUMBER prescription was conservative. The 4th (groupby_first) is a genuinely new arm but reuses the bare-aggregate machinery.

Bench	Chain	SQL lowering
distinct_by_order_take	_distinct_by(K) |> _order_by(S) |> take(N)	SELECT … FROM (SELECT *, MIN(pk) FROM t GROUP BY K) AS t0 ORDER BY S LIMIT N
distinct_by_order_to_array	_distinct_by(K) |> _order_by(S)	Same, no LIMIT
reverse_distinct_by	reverse() |> _distinct_by(K)	MAX(pk) variant — SELECT … FROM (SELECT *, MAX(pk) FROM t GROUP BY K)
groupby_first	_group_by(K) |> _select((K=_._0, R=_._1|>first()))	Outer SELECT expands row entry to all source cols + nested ExprMakeStruct in row builder

Bonus: closes a latent silent-drop bug from PR #2906 — _sql(db \|> select_from(T) \|> _distinct_by(K)) (no aggregate terminator) used to emit SELECT … FROM "T" with no GROUP BY. Now wraps in the bare-aggregate subquery via the new passthrough finalize.

Arms (3)

Arm A — _distinct_by(K) passthrough + composition with trailing _order_by / take / skip. New finalize_distinct_by_passthrough_wrap invoked by maybe_finalize_distinct_by_passthrough from every analyze_chain consumer (_sql / _try_sql / _sql_text / _each_sql / _create_view). Outer SELECT preserves source columns via fill_default_columns; outer ORDER BY / LIMIT / OFFSET piggyback on build_sql_select's existing emission around q.innerSql. Gating rejects _where / _join / _group_by / _having / set ops / additional distinct.

Arm B — reverse() recognition. New analyze_chain branch at PHASE_DISTINCT; flips MIN(pk) → MAX(pk) in the bare-aggregate when wrapped by _distinct_by. Strict gating: reverse() only legal immediately above _distinct_by (SQL has no inherent row ordering). Bare reverse() |> to_array rejects with a fixit pointing to _order_by_descending(...).

Arm C — _group_by(K) |> _select((K=_._0, R=_._1 |> first())) whole-row tuple projection. analyze_grouped_projection recognizes _._1 |> first() via the new is_first_of_group_call helper, tracks per-entry groupedSelectIsRow flags, and synthesizes q.distinctByCol from q.groupByCols[0] so the existing passthrough rail builds the inner subquery. build_sql_select's GroupedNamedTuple branch expands row entries to all source columns inline. build_row_builder emits ExprMakeStruct for row entries with per-field read_via_adapter(stmt, baseCol+j, type<FieldT>) calls (new build_row_make_struct helper mirrors make_sql_read_row_fn's pattern but with the offset baked in — no sqlite_boost change needed).

v1 constraints

• All PR sqlite_linq: _distinct_by as chain operator + first-row aggregates #2906 constraints carry forward (single @sql_primary_key, single source table, no _join, no repeat _distinct_by).
• Arm A passthrough rejects _where / _join / _group_by / _having / set ops / additional distinct.
• Arm B: reverse() immediately above _distinct_by; double-reverse rejects as a no-op.
• Arm C: single-key groups only; no mixing first() with column aggregates (length / sum / min / max / average); no computed-expression group keys with first().

Test plan

• mcp__daslang__run_test tests/dasSQLITE/test_32_distinct.das — 24 tests, 0 failures (11 from PR sqlite_linq: _distinct_by as chain operator + first-row aggregates #2906 + 13 new: 7 Arm A / 3 Arm B / 3 Arm C, each covering emit shape + runtime correctness)
• mcp__daslang__run_test tests/dasSQLITE/test_31_long_count_distinct_by_canary.das — 8 tests, 0 failures (PR sqlite_linq: _distinct_by as chain operator + first-row aggregates #2906 fast-path regression guard)
• mcp__daslang__run_test tests/dasSQLITE/test_05_sql_macro.das / test_07_sql_composability.das — sanity green
• Rejection probes compile-fail with code 50503 + expected message: failed_distinct_by_with_where.das, failed_reverse_without_distinct_by.das, failed_groupby_first_mixed_with_aggregate.das
• mcp__daslang__run_script tutorials/sql/12-distinct.das — tutorial executable + emits expected SQL strings (3 new chain shapes demonstrated end-to-end)
• All 4 m1 bench lanes report sane ns/op (240.5 / 240.6 / 312.5 / 255.1 INTERP)
• Full INTERP+JIT rerun across benchmarks/sql/*.das — drift documented in results.md header (mostly converges back toward pre-sqlite_linq: _distinct_by as chain operator + first-row aggregates #2906 baselines)
• mcp__daslang__lint clean on every touched .das file
• mcp__daslang__format_file no-op on every touched .das file

Deferred to chunk N+2

• _where(P) |> _distinct_by(K) |> ... — inner-WHERE composition (SELECT … FROM (SELECT *, MIN(pk) FROM t WHERE P GROUP BY K)); semantically distinct from outer-WHERE; needs design.
• Composite-PK support (still pending from chunk N).
• Mixed projection: _group_by(K) |> _select((K=_._0, N=_._1|>length, R=_._1|>first())) — would require inner subquery to add COUNT(*) AS group_count alongside MIN(pk), and outer row builder to interleave scalar + offset row reads.
• Multi-key groups with _._1 |> first() (single-key only today).
• Real window functions (ROW_NUMBER() OVER (PARTITION BY K ORDER BY S)) — not blocked by today's benches but useful for _distinct_by_min_by(K, S) shape if a future bench needs it.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Extends sqlite_linq’s SQLite “bare-aggregate” lowering to cover additional “first/last row per group” query shapes, closing remaining SQL gaps in the benchmark suite’s previously “window-function” bucket.

Changes:

• Add row-returning _distinct_by(K) passthrough wrap (and composition with trailing _order_by / take / skip), including a finalize step so non-aggregate _distinct_by no longer silently drops the GROUP BY.
• Add reverse() |> _distinct_by(K) lowering via MAX(pk) to select the “last” row per group by PK order.
• Add _group_by(K) |> _select((K=_._0, R=_._1 |> first())) support by expanding whole-row picks into source columns and reconstructing the struct in the row builder.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
modules/dasSQLITE/daslib/sqlite_linq.das	Implements the new analyzer/finalize paths, reverse() handling, group-by-first recognition, SQL emission, and row reconstruction.
tests/dasSQLITE/test_32_distinct.das	Adds runtime + SQL-shape coverage for the new _distinct_by passthrough, reverse_distinct_by, and groupby_first behaviors.
tests/dasSQLITE/failed_distinct_by_with_where.das	Adds compile-fail coverage for explicitly unsupported `_where
tests/dasSQLITE/failed_reverse_without_distinct_by.das	Adds compile-fail coverage for reverse() used in _sql without _distinct_by.
tests/dasSQLITE/failed_groupby_first_mixed_with_aggregate.das	Adds compile-fail coverage for mixing first() with grouped aggregates in a single projection.
tutorials/sql/12-distinct.das	Demonstrates the new chain shapes end-to-end in the tutorial.
doc/source/reference/tutorials/sql_12_distinct.rst	Documents the new query shapes and v1 constraints/caveats.
modules/dasSQLITE/API_REWORK.md	Updates the shipped/deferred tracking for the sqlite_linq rework.
benchmarks/sql/sqlite_linq_gaps.md	Marks the “window-function lowerings” group as fully closed by bare-aggregate + composition.
benchmarks/sql/distinct_by_order_take.das	Adds the SQL (m1) lane using the new passthrough lowering.
benchmarks/sql/distinct_by_order_to_array.das	Adds the SQL (m1) lane using the new passthrough lowering.
benchmarks/sql/reverse_distinct_by.das	Adds the SQL (m1) lane using the new MAX(pk) variant.
benchmarks/sql/groupby_first.das	Adds the SQL (m1) lane using the new group-by-first tuple projection.
benchmarks/sql/results.md	Refreshes published benchmark results and notes based on the newly populated SQL lanes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.