dasLLAMA: Apple +AMX tier — dasAccelerate module, BNNS-f16 lane, 3-config board scoring (both cpu board reds green)

[borisbat]

The Apple "+AMX" CPU tier for dasLLAMA, and the board scoring that measures it honestly. Closes out the two cpu-vs-stock board reds (E4B 0.958, 26B 0.861) — both green under matched-pair scoring, with every M1 cpu cell now ≥ 1.0 in both cpu configs.

Three-tier product framing

Apple inference now ships three gated configurations, each with a real use:

1. raw CPU ("generic neon-64") — portable floor, any 64-bit NEON core, bit-unchanged when the accelerate arm is unarmed;
2. CPU + AMX (DASLLAMA_ACCEL=1) — Accelerate BLAS serving float-plane batch GEMMs, for when the game owns the GPU;
3. Metal — the full as-shipped backend (unchanged here).

modules/dasAccelerate (new)

Apple-only dyn module over Accelerate: accel_sgemm_nt / accel_sgemv_n (ggml-blas call shapes), accel_bnns_hgemm_nt (dynamic BNNS BroadcastMatMul filter, f16×f16→f32 with a probe-proven f32 accumulator, ~0.2µs create-per-call), accel_threading_mode diagnostics. Accelerate's internal pool starves against the spinning jobque (~40ms/call flat), and its threading mode is per-thread TLS state — so every calling lane pins BLASSetThreading(SINGLE) (BNNS pins via n_threads=1) and the das side strip-dispatches over the jobque instead. Default-ON on Apple like dasMetal (DAS_ACCELERATE_DISABLED opt-out), so it rides existing SDK packaging.

Driver (dasllama_math_accelerate.das) + the float-batch override rail

set_float_batch_override in dasllama_math — fp32/bf16 batch slots only; the raw-CPU tier is bit-unchanged when unarmed. The driver arms it from DASLLAMA_ACCEL=1:

• fp32 planes → strip-dispatched sgemm; bf16 planes → grow-only f32 widen + strip-sgemm;
• per-call work floor DASLLAMA_ACCEL_MIN_MMAC (default 512) — tiny GEMMs (MoE routers) net-lose on the AMX engagement tax and fall through to NEON;
• BNNS f16 lane (DASLLAMA_ACCEL_F16=1, default OFF): weight plane converts bf16→f16 once (exact for every bf16 value in f16 normal range; ±65504 clamp; pointer+sampled-fingerprint cache guards mmap address reuse), activations convert per call via the f16 KV-codec kernel, strips over the jobque (DASLLAMA_ACCEL_F16_STRIPS knob). On M1 it is a wash vs the sgemm arm (within ±1% across adjacent-pair reps) and ships uncrowned — the target is M4/M5 SME where f16 is 2× f32 throughput. Falls back to widen+sgemm on probe/filter refusal.

In-model, the arm's one real M1 engagement is the E4B/E2B bf16 per_layer_model_proj (14.1 GMAC/pass): clean-CPU NEON has no bf16 ISA on M1, stock llama.cpp serves it on AMX — this arm answers with the same silicon (E4B pp512 +13% over raw CPU, the board red closed).

Board scoring: three matched pairs, never best-on-backend

llama-bench at -ngl 0 still op-offloads big-batch host-weight ops to Metal on a stock build (26B: 194MB MTL0 compute buffer, +33% pp) — pairing das-pure-CPU against that row was a category error, and it was the whole 26B "red". The board now derives each das ratio against its named llama.cpp sibling:

config	das side	llama.cpp side
generic neon-64	raw CPU	clean-cpu build, -ngl 0 (kernel vs kernel)
cpu + AMX	--accel	stock, -ngl 0 -nopo 1 (Accelerate both sides, no GPU; -nopo ≡ no-Metal build, window-proven)
as shipped	Metal	stock, -ngl 99

Rails: lcpp_bench --accel (records flavor accel), gen_bench_records --legs neon|amx|metal with each pair's das cell and ref run adjacent, site derivation + prose updated, PUBLIC_BENCH_PLAN.md updated.

M1 records re-sweep (pp512, das/lcpp)

model	generic neon-64	cpu + AMX	as shipped
Qwen3.6-35B-MTP	1.39×	1.45×	0.98×
Qwen3.6-27B-MTP	1.03×	1.10×	1.11×
gemma-4-12B	1.11×	1.09×	1.04×
gemma-4-E4B	1.57×	1.08× (was 0.958)	1.06×
gpt-oss-20b	1.74×	1.75×	0.92×
Qwen3-30B	1.60×	1.63×	0.99×
Mistral-24B	1.06×	1.03×	1.01×
gemma-4-26B	1.20×	1.24× (was 0.861)	1.10×

tg128 is untouched by the arm (override gates at ntok ≥ 32) — accel ≡ tuned decode everywhere. The honest pairing surfaces two pre-existing cpu-decode gaps (gpt-oss 0.97, 26B 0.96–0.97 — the old derivation hid the first behind lcpp's own op-offload decode loss); those are follow-up work, tracked outside this PR.

Measurement integrity: the sweep exposed that the FIRST das process on a cold/churned .dlim map warms across its reps (35B: 150→198 monotone within one cell; the next process reads a flat 212 ± 1.7) — one in-process warmup pass cannot absorb a 19GB map fighting page-cache eviction. gen_bench_records now carries a self-healing variance tripwire: any das cell over 3% cv re-runs once warm and keeps the retry (it caught and healed three cells during this sweep, including a stale pre-existing 26B baseline). The product-level fix candidate (eager-prefault option in the dlim mapper) is deliberately not part of this PR.

Harness / instruments (committed for reproducibility)

bnns_hgemm_probe.c (BNNS dtype matrix, accumulator-width proof, strips scaling — all M1 verdicts in the header), accel_sgemm_bench.c (Accelerate ceiling curves), accel_contention_probe.das (+ BNNS phase), dasprof_cpu/blas.patch (ref-tree op profiling).

Rider fix: tests/aot irregular-table guards were dead (master, hours old)

The first full-preflight over today's 38a1077a2 (per-suite AOT loop collapse) failed 50101 on every dasPUGIXML/dasHV/dasSQLITE test: the table's row guards (NOT DAS_X_DISABLED) reach if() as a single space-embedded argument, which CMake reads as an undefined variable name — so every guarded reg row silently stopped emitting its AOT stubs (pugixml/dashv/dassqlite/strudel/live_host/minfft). Latent until a fresh configure regenerates the graph — which also means the nightly full-AOT sweep hits it. Fixed by splitting the guard into real tokens before evaluation; repro + fix verified in isolation on CMake 3.29.5, all six areas emit edges again.

Testing

• modules/dasLLAMA/tests/test_accel_backend.das 5/5: fp32/bf16 tolerance parity, BNNS f16 lane parity, below-gate bit-exact fall-through; feints cleanly off-Apple. The f16 flag is pinned off in setup so env arming cannot leak into the sgemm arms.
• E4B/35B warm adjacent A/Bs in-model; contention rig clean (no jobque starve in either arm).
• JIT smoke clean. Full preflight ran once (the once-per-PR policy): 15/17 gates passed, cpp-syntax skipped (module C++ has no local clang gate — CI's lane covers it), tests-aot failed on the master-side guard bug above — root-caused and fixed in-branch with an isolated CMake repro; pushed --no-verify per policy, CI validates the tip.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR adds an Apple-specific “+AMX/SME” CPU acceleration tier for dasLLAMA via a new das_accelerate dynamic module (Accelerate BLAS + optional BNNS f16 matmul lane), and updates the public benchmark tooling/site to score dasLLAMA vs llama.cpp using matched configuration pairs (generic CPU vs clean-cpu, accel vs stock -ngl 0 -nopo 1, Metal vs -ngl 99). It also fixes a recently-latent AOT CMake guard parsing issue that caused guarded irregular AOT rows to stop emitting.

Changes:

• Add modules/dasAccelerate (Apple-only) and wire a float-plane batch override (set_float_batch_override) to support an opt-in Accelerate-backed CPU tier for dasLLAMA.
• Update benchmark record generation and website presentation to use matched-pair denominators (and record llama.cpp CPU refs with -nopo 1 to avoid GPU op-offload at -ngl 0).
• Fix tests/aot/CMakeLists.txt irregular-table guard evaluation by tokenizing guard expressions before if() evaluation.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/aot/CMakeLists.txt	Fixes irregular AOT row guard evaluation by splitting the guard string into if() tokens.
skills/make_pr.md	Updates pre-push / preflight process guidance (single full-preflight per PR).
skills/babysit.md	Aligns post-review workflow guidance with the “one full-preflight per PR” policy.
site/files/dasllama/bench_records.json	Updates/extends published bench records (adds accel rows; records llama.cpp CPU refs with -nopo 1).
site/files/dasllama.js	Changes ratio computation to matched-pair denom selection per das configuration.
site/dasllama.html	Updates narrative to explain matched-pair scoring and -nopo rationale.
modules/dasLLAMA/tests/test_accel_backend.das	Adds tests for fp32/bf16 override parity + BNNS f16 lane behavior and below-gate fall-through.
modules/dasLLAMA/PUBLIC_BENCH_PLAN.md	Updates public methodology docs for accel flavor and matched-pair ratios.
modules/dasLLAMA/performance/records/m1.json	Updates/extends per-box official records (adds accel rows; updates CPU refs with -nopo 1).
modules/dasLLAMA/performance/profile_common.das	Documents new das run flavor accel in the bench run schema.
modules/dasLLAMA/performance/gen_bench_records.das	Adds --legs pairing, adjacent ref runs per leg, and high-variance auto rerun logic.
modules/dasLLAMA/harness/dasprof_cpu.patch	Adds reproducible ggml CPU-side profiling patch for section timing/chunk counts.
modules/dasLLAMA/harness/dasprof_blas.patch	Adds reproducible ggml BLAS-side profiling patch for dequant/sgemm timing split.
modules/dasLLAMA/harness/bnns_hgemm_probe.c	Adds BNNS dtype/accumulator/profiling probe used to justify the BNNS f16 lane design.
modules/dasLLAMA/harness/accel_sgemm_bench.c	Adds Accelerate SGEMM/SGEMV ceiling benchmark utility for Apple boxes.
modules/dasLLAMA/harness/accel_contention_probe.das	Adds contention probe for Accelerate/BNNS under jobque strip-dispatch.
modules/dasLLAMA/dasllama/dasllama_math.das	Introduces the float-plane batch override rail (set_float_batch_override) and hooks it into batch matmuls.
modules/dasLLAMA/dasllama/dasllama_math_accelerate.das	Implements Accelerate-backed fp32 and bf16 batch overrides (+ optional BNNS f16 lane) and env-controlled arming.
modules/dasLLAMA/dasllama/dasllama_common.das	Wires the accelerate driver into the dasLLAMA umbrella via require ?das_accelerate.
modules/dasLLAMA/benchmarks/lcpp_bench.das	Adds --accel flag to produce das “accel” rows for bench record generation.
modules/dasLLAMA/.das_module	Adds dasllama_math_accelerate to the module’s export path list.
modules/dasAccelerate/src/dasAccelerate.cpp	Adds Apple Accelerate/BNNS C++ shim exports (accel_sgemm_nt, accel_sgemv_n, accel_bnns_hgemm_nt, diagnostics).
modules/dasAccelerate/CMakeLists.txt	Adds Apple-only build wiring for the new dynamic module and Accelerate framework link.
modules/dasAccelerate/.das_module	Registers the dynamic module at runtime for DLL builds (silent skip off-Apple).
CMakeLists.txt	Adds DAS_ACCELERATE_DISABLED option to control building the new module.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.