Skip to content

dasLLAMA: kq/q8 tile rework — vector bsums, mr8 perms, streaming tune fixtures; DAS_JOBQUE_THREADS validation - #3542

Merged
borisbat merged 8 commits into
masterfrom
bbatkin/jobque-threads-panic
Jul 22, 2026
Merged

dasLLAMA: kq/q8 tile rework — vector bsums, mr8 perms, streaming tune fixtures; DAS_JOBQUE_THREADS validation#3542
borisbat merged 8 commits into
masterfrom
bbatkin/jobque-threads-panic

Conversation

@borisbat

Copy link
Copy Markdown
Collaborator

CPU prefill kernel rework driven by a per-bucket das-vs-llama.cpp profile of the Mistral-24B Q4_K_M prefill. The gap concentrated in one place: the k4 batch GEMM on fat FFN shapes delivered 0.89x of ggml's repacked q4_K_8x4 kernel per lane, and IR-level accounting showed ~3x the loads per MAC (64 scalar bsum loads per superblock, 4-row amortization).

Kernel changes (LLVM emitter, dasllama_gemm_gen.das)

  • Vector bsums + epilogue min-bias fold (NEON sdot leg, k4/k5/q40): the superblock's 16 per-16 bsums load as 4 v4i32 with a pairwise fold instead of 16 scalar loads + 8 splats per token; the min-bias accumulation moves to a superblock epilogue so it never lives in the block loop. Bit-exact (same i32 math and order).
  • Fused lo/hi accumulator at mr >= 8 (the per-block fold adds them anyway, so fusing is i32-exact) — halves the live int-acc bank, which is what makes 8-row groups fit the NEON register file. New mr8 perm rows on all four kq tile grids.
  • Decline-rail fix: the sdot q-reg budget in perm_declines modeled the split-acc emitter and silently declined mr8/nrsplit4 to the reference body (~11 GMAC/s) — tests stay green on a declined perm since the reference is bit-exact, so the only tell is speed. The budget now models the fused shape at mr >= 8. This also un-blocked the pre-existing q8 mr8_budget perm, which then won its table and passed the e2e confirm gate (+7.5% on a Q8 model).
  • Tune fixtures re-aimed to streaming shapes (kq and q8 batch fixtures: 2048x512x64 L2-hot kv-projection -> 2048x8192x256, weights above the L2 budget): the hot probe systematically crowned perms that lose double-digit % at model scale. Divergent q8 crowns still pass the e2e confirm gate before shipping.
  • LLVM_JIT_CODEGEN_VERSION 0x47 -> 0x48 (generator emission changed for fixed args).

x64 legs are untouched: all new emission is gated behind the aarch64 sdot rail, and the new perms decline to reference on other ISAs until their own tuner re-ranks.

Results (M1 Max, records re-sweep committed in performance/records/m1.json)

Every CPU pp512 cell on the 8-model board is now green vs clean-cpu llama.cpp (1.06-1.79x); the two losing cells flipped (gemma-4-12B 0.96 -> 1.09, Mistral-24B 0.90 -> 1.06; quiet-window Mistral pair: pp 34.22 vs 30.69 = 1.115x, tg 8.26 vs 7.87). das now beats the Accelerate/AMX stock build on 7 of 8 pp cells. No tg regressions; decode A/B gate passed (mr8 tg 7.99 vs mr4 7.86). test_kquant 104/104 bit-exact on the emitted kernels at both mr4 and mr8.

Also in this PR

  • DAS_JOBQUE_THREADS <= 1 is now a fatal error (job_que.cpp): the value means total compute lanes (N-1 workers + the computing main), so N==1 cannot be honored — the old floor silently ran 2 lanes and inflated every "t=1" baseline 2x. Garbage values (atoi -> 0) fail loudly too.
  • mm_ffn_down profile bucket (das + the lcpp ggml_op_profile.patch mirror): the down projection is long-K and often a different format (Q4_K_M keeps it q6_K); lumping it with gate+up hid where the deficit lived. The lcpp patch classifies ffn_out nodes (the biasless down mm) into the mirrored bucket.
  • DASLLAMA_TOKEN_BLOCK env in prefill_perf.das — A/B rail for the batch-GEMM token block (used to refute the weight re-stream theory: TB=256 no change, TB=512 regresses).

Full preflight (--full, 17/17 gates) green.

🤖 Generated with Claude Code

borisbat and others added 7 commits July 22, 2026 00:26
The value means TOTAL compute lanes (N-1 workers + the computing main),
so N==1 cannot be honored: the old max(1, N-1) floor silently ran one
worker + main = 2 lanes, inflating every "t=1" baseline 2x. Values <= 1
(including garbage that atoi maps to 0) now fail loudly at jobque init
instead of lying. Sweep docs in bench_gemm_iso updated to start at 2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fn_down)

The down proj is long-K and often a different format (Q4_K_M keeps it
q6_K), so lumping it with gate+up hid where the prefill deficit lives.
das side: prefill + decode down spans now report as mm_ffn_down. lcpp
mirror patch: ffn_out/ffn_down mul_mat nodes get their own bucket (the
down mm node is named ffn_out on biasless models).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A/B rail for the batch-GEMM token block (weights re-stream ntok/TB
times). First use: refuted the weight re-stream theory for the k4
gate/up gap on M1 (TB=256 no change, TB=512 -7% model).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The k4 gate/up GEMM trailed ggml's repacked q4_K_8x4 kernel 0.89x
per-lane in-model; IR accounting showed ~3x the loads per MAC, mostly
64 scalar bsum loads per superblock and 4-row amortization.

- sdot leg (k4/k5/q40): the superblock's 16 per-16 bsums load as 4
  v4i32 + pairwise fold instead of 16 scalar loads + 8 splats per
  token; the min-bias fold moves to a superblock epilogue so bacc
  never lives in the block loop. Bit-exact (same i32 math and order).
- mr >= 8 fuses lo+hi nibble dots into one accumulator (the fold adds
  them anyway), halving the int-acc bank; new mr8 perm rows on all
  four kq tile grids, and the sdot q-reg budget rail now models the
  fused shape (the old formula silently declined mr8 to reference).
- LLVM_JIT_CODEGEN_VERSION 0x47 -> 0x48 (generator emission changed
  for fixed args).

M1 Max, k4 streaming rig (Mistral FFN shape, 2 lanes): 188-190 ->
198.7-200.9 GMAC/s from the bsum rework at mr4, 217.9 at mr8 (109.0
per lane — above ggml's in-model 106.2). test_kquant 104/104 at mr4
and emitted mr8.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The kq tile bench fixture was 2048x512x64 — an L2-resident kv-projection
shape, while ~90% of prefill time is fat streaming FFN GEMMs. Hot-shape
winners lose double-digit % at model scale (mr4 vs mr8 on M1). The
fixture is now 2048x8192x256: weights exceed the L2 budget and the walk
spans multiple token blocks, so the crowned perm reflects the shape
class the tile actually serves.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same over-fit as the kq fixture: the 2048x512x64 kv-projection probe is
L2-hot and under-ranks perms that win at model scale (the mr8_budget row
tops the streaming table). Divergent crowns still pass the e2e confirm
gate before shipping, so the fixture change is ranking-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full-catalog CPU re-run (das + clean-cpu + stock rows, 2026-07-22).
Every pp cell is green vs clean-cpu (1.06-1.79x); the two red cells
flipped (gemma-12b 0.96->1.09, Mistral-24B 0.90->1.06). No tg
regressions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 22, 2026 18:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR focuses on improving dasLLAMA CPU prefill performance on arm64 (NEON sdot path) by reworking kq/q8 tile codegen and retuning fixtures toward streaming FFN shapes, while also tightening DAS_JOBQUE_THREADS validation and updating profiling/bench artifacts.

Changes:

  • Reworks NEON kq tile emission (vector bsums, mr≥8 fused accumulators) and fixes perm decline budgeting; bumps LLVM_JIT_CODEGEN_VERSION.
  • Re-aims tuning fixtures to streaming FFN shapes and splits profiling into a new mm_ffn_down bucket (including an updated ggml profiling patch).
  • Updates M1 performance record data and adds an env override for token-block sizing in prefill_perf.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/misc/job_que.cpp Makes DAS_JOBQUE_THREADS <= 1 a fatal error and clarifies thread-count semantics.
modules/dasLLVM/daslib/llvm_jit_run.das Bumps LLVM_JIT_CODEGEN_VERSION to invalidate cached JIT artifacts after codegen changes.
modules/dasLLAMA/performance/records/m1.json Updates benchmark record entries (dates/cmds/results/tune metadata).
modules/dasLLAMA/harness/ggml_op_profile.patch Adds a dedicated mm_ffn_down profiling bucket mirroring das-side profiling.
modules/dasLLAMA/harness/gen_tune_probe.das Retargets tuning fixtures to streaming-heavy FFN shapes.
modules/dasLLAMA/dasllama/dasllama_math_gen.das Adjusts tune permutations to include mr=8 entries directly (not only nrsplit=2).
modules/dasLLAMA/dasllama/dasllama_gemm_gen.das Implements vector-bsum loading, epilogue min-bias fold, and fused-acc behavior for mr≥8 in kq tile emission.
modules/dasLLAMA/dasllama/dasllama_common.das Splits FFN profiling into mm_ffn vs mm_ffn_down for better attribution.
modules/dasLLAMA/benchmarks/prefill_perf.das Adds DASLLAMA_TOKEN_BLOCK env override for batch-GEMM token block.
modules/dasLLAMA/benchmarks/matmul/bench_gemm_iso.das Updates benchmark guidance to reflect new jobque thread semantics and fatal <= 1.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/misc/job_que.cpp
Comment thread modules/dasLLAMA/benchmarks/prefill_perf.das
…ctive token block

Copilot: exported-but-empty env previously behaved like unset and now
panicked (atoi("") == 0); restore unset semantics for the empty string,
keep the fatal error for real values <= 1. The token-block override log
now prints the effective (clamped) value, not the raw parse.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 22, 2026 18:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

@borisbat
borisbat merged commit 23b1ed5 into master Jul 22, 2026
37 checks passed
pull Bot pushed a commit to forksnd/daScript that referenced this pull request Jul 23, 2026
The M1 section was measured 2026-07-13 and badly understated das — the
cpu-pp (GaijinEntertainment#3542) and decode-tg (GaijinEntertainment#3544) arcs lifted CPU decode ~+5-8% since,
but the board still showed the old numbers, so the whole dense-q8 tg column
read as a systematic 0.92-0.96 deficit.

Re-swept das fresh in a clean Parsec-off window (correct mr8_budget q8q8
crown; lcpp pins unchanged — clean-cpu ebd048f is stable). tg128 is now
genuine parity: das wins most cells (1.00-1.15x) and loses only Phi-3.5-mini
(0.95x, the one real dense-q8 outlier) plus three within-noise cells
(SmolLM2-1.7B 0.98, Qwen3-4B/Llama-3.2-1B 0.99). pp512 stays all-green
(1.04-1.77x). Notably the 7-8B dense models flipped: Mistral-7B 0.96->1.00,
Llama-3.1-8B 0.96->1.03, gpt-oss-20b 0.94->1.01. Adds three gemma-4-26B-A4B
format rows (q8/q4k/q4_0, das-only until lcpp pins land).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XLduse7eHSS1iJ2BG2P4LT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants