dasLLAMA: kq/q8 tile rework — vector bsums, mr8 perms, streaming tune fixtures; DAS_JOBQUE_THREADS validation - #3542
Merged
Merged
Conversation
The value means TOTAL compute lanes (N-1 workers + the computing main), so N==1 cannot be honored: the old max(1, N-1) floor silently ran one worker + main = 2 lanes, inflating every "t=1" baseline 2x. Values <= 1 (including garbage that atoi maps to 0) now fail loudly at jobque init instead of lying. Sweep docs in bench_gemm_iso updated to start at 2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fn_down) The down proj is long-K and often a different format (Q4_K_M keeps it q6_K), so lumping it with gate+up hid where the prefill deficit lives. das side: prefill + decode down spans now report as mm_ffn_down. lcpp mirror patch: ffn_out/ffn_down mul_mat nodes get their own bucket (the down mm node is named ffn_out on biasless models). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A/B rail for the batch-GEMM token block (weights re-stream ntok/TB times). First use: refuted the weight re-stream theory for the k4 gate/up gap on M1 (TB=256 no change, TB=512 -7% model). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The k4 gate/up GEMM trailed ggml's repacked q4_K_8x4 kernel 0.89x per-lane in-model; IR accounting showed ~3x the loads per MAC, mostly 64 scalar bsum loads per superblock and 4-row amortization. - sdot leg (k4/k5/q40): the superblock's 16 per-16 bsums load as 4 v4i32 + pairwise fold instead of 16 scalar loads + 8 splats per token; the min-bias fold moves to a superblock epilogue so bacc never lives in the block loop. Bit-exact (same i32 math and order). - mr >= 8 fuses lo+hi nibble dots into one accumulator (the fold adds them anyway), halving the int-acc bank; new mr8 perm rows on all four kq tile grids, and the sdot q-reg budget rail now models the fused shape (the old formula silently declined mr8 to reference). - LLVM_JIT_CODEGEN_VERSION 0x47 -> 0x48 (generator emission changed for fixed args). M1 Max, k4 streaming rig (Mistral FFN shape, 2 lanes): 188-190 -> 198.7-200.9 GMAC/s from the bsum rework at mr4, 217.9 at mr8 (109.0 per lane — above ggml's in-model 106.2). test_kquant 104/104 at mr4 and emitted mr8. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The kq tile bench fixture was 2048x512x64 — an L2-resident kv-projection shape, while ~90% of prefill time is fat streaming FFN GEMMs. Hot-shape winners lose double-digit % at model scale (mr4 vs mr8 on M1). The fixture is now 2048x8192x256: weights exceed the L2 budget and the walk spans multiple token blocks, so the crowned perm reflects the shape class the tile actually serves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same over-fit as the kq fixture: the 2048x512x64 kv-projection probe is L2-hot and under-ranks perms that win at model scale (the mr8_budget row tops the streaming table). Divergent crowns still pass the e2e confirm gate before shipping, so the fixture change is ranking-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full-catalog CPU re-run (das + clean-cpu + stock rows, 2026-07-22). Every pp cell is green vs clean-cpu (1.06-1.79x); the two red cells flipped (gemma-12b 0.96->1.09, Mistral-24B 0.90->1.06). No tg regressions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR focuses on improving dasLLAMA CPU prefill performance on arm64 (NEON sdot path) by reworking kq/q8 tile codegen and retuning fixtures toward streaming FFN shapes, while also tightening DAS_JOBQUE_THREADS validation and updating profiling/bench artifacts.
Changes:
- Reworks NEON kq tile emission (vector bsums, mr≥8 fused accumulators) and fixes perm decline budgeting; bumps
LLVM_JIT_CODEGEN_VERSION. - Re-aims tuning fixtures to streaming FFN shapes and splits profiling into a new
mm_ffn_downbucket (including an updated ggml profiling patch). - Updates M1 performance record data and adds an env override for token-block sizing in
prefill_perf.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/misc/job_que.cpp | Makes DAS_JOBQUE_THREADS <= 1 a fatal error and clarifies thread-count semantics. |
| modules/dasLLVM/daslib/llvm_jit_run.das | Bumps LLVM_JIT_CODEGEN_VERSION to invalidate cached JIT artifacts after codegen changes. |
| modules/dasLLAMA/performance/records/m1.json | Updates benchmark record entries (dates/cmds/results/tune metadata). |
| modules/dasLLAMA/harness/ggml_op_profile.patch | Adds a dedicated mm_ffn_down profiling bucket mirroring das-side profiling. |
| modules/dasLLAMA/harness/gen_tune_probe.das | Retargets tuning fixtures to streaming-heavy FFN shapes. |
| modules/dasLLAMA/dasllama/dasllama_math_gen.das | Adjusts tune permutations to include mr=8 entries directly (not only nrsplit=2). |
| modules/dasLLAMA/dasllama/dasllama_gemm_gen.das | Implements vector-bsum loading, epilogue min-bias fold, and fused-acc behavior for mr≥8 in kq tile emission. |
| modules/dasLLAMA/dasllama/dasllama_common.das | Splits FFN profiling into mm_ffn vs mm_ffn_down for better attribution. |
| modules/dasLLAMA/benchmarks/prefill_perf.das | Adds DASLLAMA_TOKEN_BLOCK env override for batch-GEMM token block. |
| modules/dasLLAMA/benchmarks/matmul/bench_gemm_iso.das | Updates benchmark guidance to reflect new jobque thread semantics and fatal <= 1. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ctive token block
Copilot: exported-but-empty env previously behaved like unset and now
panicked (atoi("") == 0); restore unset semantics for the empty string,
keep the fatal error for real values <= 1. The token-block override log
now prints the effective (clamped) value, not the raw parse.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pull Bot
pushed a commit
to forksnd/daScript
that referenced
this pull request
Jul 23, 2026
The M1 section was measured 2026-07-13 and badly understated das — the cpu-pp (GaijinEntertainment#3542) and decode-tg (GaijinEntertainment#3544) arcs lifted CPU decode ~+5-8% since, but the board still showed the old numbers, so the whole dense-q8 tg column read as a systematic 0.92-0.96 deficit. Re-swept das fresh in a clean Parsec-off window (correct mr8_budget q8q8 crown; lcpp pins unchanged — clean-cpu ebd048f is stable). tg128 is now genuine parity: das wins most cells (1.00-1.15x) and loses only Phi-3.5-mini (0.95x, the one real dense-q8 outlier) plus three within-noise cells (SmolLM2-1.7B 0.98, Qwen3-4B/Llama-3.2-1B 0.99). pp512 stays all-green (1.04-1.77x). Notably the 7-8B dense models flipped: Mistral-7B 0.96->1.00, Llama-3.1-8B 0.96->1.03, gpt-oss-20b 0.94->1.01. Adds three gemma-4-26B-A4B format rows (q8/q4k/q4_0, das-only until lcpp pins land). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XLduse7eHSS1iJ2BG2P4LT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CPU prefill kernel rework driven by a per-bucket das-vs-llama.cpp profile of the Mistral-24B Q4_K_M prefill. The gap concentrated in one place: the k4 batch GEMM on fat FFN shapes delivered 0.89x of ggml's repacked
q4_K_8x4kernel per lane, and IR-level accounting showed ~3x the loads per MAC (64 scalar bsum loads per superblock, 4-row amortization).Kernel changes (LLVM emitter,
dasllama_gemm_gen.das)mr8perm rows on all four kq tile grids.perm_declinesmodeled the split-acc emitter and silently declined mr8/nrsplit4 to the reference body (~11 GMAC/s) — tests stay green on a declined perm since the reference is bit-exact, so the only tell is speed. The budget now models the fused shape at mr >= 8. This also un-blocked the pre-existing q8mr8_budgetperm, which then won its table and passed the e2e confirm gate (+7.5% on a Q8 model).LLVM_JIT_CODEGEN_VERSION0x47 -> 0x48 (generator emission changed for fixed args).x64 legs are untouched: all new emission is gated behind the aarch64 sdot rail, and the new perms decline to reference on other ISAs until their own tuner re-ranks.
Results (M1 Max, records re-sweep committed in
performance/records/m1.json)Every CPU pp512 cell on the 8-model board is now green vs clean-cpu llama.cpp (1.06-1.79x); the two losing cells flipped (gemma-4-12B 0.96 -> 1.09, Mistral-24B 0.90 -> 1.06; quiet-window Mistral pair: pp 34.22 vs 30.69 = 1.115x, tg 8.26 vs 7.87). das now beats the Accelerate/AMX stock build on 7 of 8 pp cells. No tg regressions; decode A/B gate passed (mr8 tg 7.99 vs mr4 7.86).
test_kquant104/104 bit-exact on the emitted kernels at both mr4 and mr8.Also in this PR
DAS_JOBQUE_THREADS <= 1is now a fatal error (job_que.cpp): the value means total compute lanes (N-1 workers + the computing main), so N==1 cannot be honored — the old floor silently ran 2 lanes and inflated every "t=1" baseline 2x. Garbage values (atoi -> 0) fail loudly too.mm_ffn_downprofile bucket (das + the lcppggml_op_profile.patchmirror): the down projection is long-K and often a different format (Q4_K_M keeps it q6_K); lumping it with gate+up hid where the deficit lived. The lcpp patch classifiesffn_outnodes (the biasless down mm) into the mirrored bucket.DASLLAMA_TOKEN_BLOCKenv inprefill_perf.das— A/B rail for the batch-GEMM token block (used to refute the weight re-stream theory: TB=256 no change, TB=512 regresses).Full preflight (
--full, 17/17 gates) green.🤖 Generated with Claude Code