specula -- current status

SESSION 2026-06-16 (latest) — Qualcomm w4a16 deep-dive → 4.5× prefill on our llama.cpp

Dissected the blessed Qualcomm Qwen3-4B w4a16 bundle end-to-end (how it hits ~2229 pp) and applied the learnings to our llama.cpp HTP backend for a measured 4.5× prefill win. Full writeup: docs/qualcomm_w4a16_deep_dive.md.

The deep-dive (5 phases, on-device tools): static graph dump (qnn-context-binary-utility), per-op HTP profiling (qnn-net-run + qnn-profile-viewer, unlocked via ORT's signed skel + --use_native_input_files), instruction-level skel RE (hexagon-llvm-objdump --mv81 --mhvx), literature sweep. Headline mechanism: bundle = w4 block-wise weights + uint16 fixed-point activations + uint8 KV; 10 graphs/part (5 ctx × ar128/ar1), 615 MB weights shared across graphs; the w4 matmul HVX-expands int4→fp16 then runs fp16 HMX (NOT native integer — block-wise scales can't use the native A16W4 MAC, which is exactly the per-block-scale wall our w4a8 probe hit). The 2229 pp is AR128 amortisation + whole-graph fusion + bandwidth, not a magic matmul. Skel reuse in llama.cpp: not feasible (proprietary FastRPC IDL + GPL license). See [[reference_qualcomm_w4a16_deep_dive]], [[reference_hmx_w4a16_no_native_primitive]].

🏆 APPLIED → 4.5× prefill. Our llama.cpp base 45cac7c (2026-04-17) was 34 hexagon commits behind master (HMX matmul rework #23368, op fusion #23835, HMX flash attention #22347, max-corner clocks #22334). Branch had no custom commits → fast-forwarded hotschmoe-npu-work → 74ade52, rebuilt signed HTP (build-new). Qwen3-4B Q4_0 HTP0: pp512 187 → 499 (-fa 0) → 844 (-fa 1); tg 19.9 → 21.6. -fa 1 now WINS on HTP (HMX flash attn flipped the old -fa 0 rule). Gap to Qualcomm (2229 pp) closed 11.9× → 2.64×. Remaining gap is architectural (QNN's finalised weight-resident whole-part graph + AR128). Results: results/csv/hexagon_upstream_update_2026-06-16.md. New production HTP build = build-new with -fa 1.

SESSION 2026-06-16 (late) — HMX single-tile probe: native w4a16 DISPROVEN, w4a8 MAC confirmed

Built an on-device HMX single-tile reverse-engineering probe to settle the two open kernel questions on real v81 silicon. Full writeup: docs/hmx_single_tile_probe_findings.md.

Headline (success criterion #1, definitive): there is NO native int4-weight × fp16-activation ("w4a16") HMX primitive on v81. On a harness validated by the proven fp16×fp16 all-ones tile reading back exactly 32.0, all three int4 weight loaders fail to dequantize in an fp16 pass: Q6_weight_ubit→0.0099, Q6_weight_n→0.0049, Q6_weight_sbit→0.0099 vs the required 32 (~3000–6500× too small; ubit also non-linear in the weight value). The REOPENED "plausible native w4a16" hypothesis is now empirically disproven — closed in docs/llama_hexagon_qwen35_w4a16_plan.md and memory [[reference_hmx_w4a16_no_native_primitive]].

w4a8 (int4 × uint8 → int32) MAC CONFIRMED WORKING + INTEGER DATA PATH PINNED in-engine. The early fold/scale was a tile-size bug: HMX tiles are always 2 KB and a uint8 tile is 8×8×32 flat (2048 elem), not 32×32/Rt=1023. Re-probed in the live backend with full 2 KB tiles + Rt=2047 + uh_2x1: all-ones → uniform 4.0, full 1024-coverage (fold gone). Decoded readout row r → activation M = 4·(r//2)+1; fixed ~÷8 readout scale. The exact integer data path (distinct from the fp16 32×32 geometry) is now known. Next: build the full looped integer matmul on this geometry (pack int4 weights R4Weights8x4 + uint8 act 8×8×32, loop passes like core_dot_chunk_fp16, calibrate scale + per-block Q4_0 scale), then validate vs CPU + bench. Full detail: findings doc.

Why Qualcomm w4a16 hits 2200+ pp with NO native int4×fp16 (resolved): "a16" = fp16 activations, and there's no int4×fp16 MAC, so their w4a16 decompresses int4→fp16 inline then runs plain fp16×fp16 HMX — same arithmetic llama.cpp already does. The 13× gap is data movement + pipelining: they keep weights int4 into VTCM and decompress fused (¼ the weight DDR bandwidth; prefill is bandwidth-bound), vs llama.cpp's separate q4_0_to_fp16_lut pass that materializes full fp16 weight tiles (2× bandwidth + extra pass), plus whole-graph VTCM residency/DMA double-buffering. ⇒ a cheap llama.cpp win is making the existing fp16 path decompress inline; w4a8 adds integer throughput + ¼ activation bandwidth on top, at an accuracy cost.

Reusable harness (branch npu-int4a16-hmx in llama-int4a16 worktree): hmx_w4a16_probe() builds KNOWN VTCM tiles and dumps RAW readout tiles to a host-visible dst (DSP FARF is NOT visible on stdout on Windows — verified — so dump-to-dst is the mechanism). Built with -DGGML_HEXAGON_W4A16_PROBE=ON; host reader tests/test-hmx-probe.cpp; offline analysis scripts/analyze_hmx_probe.py. Also confirmed this session: the signed-cat HTP stack loads on HTP0 and the fp16 dequant matmul path runs on-device (test-backend-ops -o MUL_MAT -b HTP0).

SESSION 2026-06-16 — 14B NPU runtime: fp16 build bug found + FIXED, decode in progress

Ran the 10-part Qwen3-14B w8a16 bundle on the X2E Hexagon. Found two runtime ceilings, fixed the first, drove a corrected rebuild on the Threadripper, and built a streaming engine for the second. Full writeup: docs/npu_engine_14b_runtime.md.

Headline arc: (1) decoder parts wouldn't load — root-caused to the "w8a16" build storing fp16 weights (no calibration → float HTP graph → 3.3 GB/part > ~2 GB per-context ceiling). (2) Drove a calibrated rebuild on the box (capture_calib_14b.py + requant_14b.sh --input_list) → int8, 1.66 GB/part, loads on the Hexagon. (3) Hit a second ceiling: the full ~16 GB bundle exceeds the HTP's ~10 GB / ~6-context budget via ORT-QNN; built engine_14b_swap.py (2-group streaming) but the X2E DSP transport crashes under context churn / >~8 GB execution, and no ORT-QNN runtime knob (shared-mem allocator / spill-fill) moves the wall (tested).

🏆🏆🏆 END GOAL HIT — Qwen3.6-27B (and 35B-MoE) RUN ON THE HEXAGON NPU. qwen35/qwen35moe are already implemented in llama.cpp master. Measured on 4 HTP sessions + hybrid -ngl:

Qwen3.6-27B Q4_0 (14.7 GiB, 26.9B dense): pp64 24.7 / tg16 2.8 t/s.
Qwen3.6-35B-A3B MXFP4-MoE (20.2 GiB, 34.66B): pp64 50.4 / tg16 5.6. The 27B GGUF needed its MTP head stripped (scripts/strip_mtp_head.py: drop blk.64.*, block_count→64) — the shipped MTP variant fails to load. TG is low (SSM/overflow layers on CPU); perf = w4a16 integer-HMX + -ngl/NHVX tuning. Full chain: 4B→14B→27B→35B all run on the NPU; ORT-QNN couldn't pass the 14B. See [[reference_llamacpp_hexagon_npu_works]], docs/llama_hexagon_qwen35_w4a16_plan.md.

🏆 BREAKTHROUGH — llama.cpp Hexagon NPU backend WORKING + 14B VERIFIED. Qwen3-4B-Q4_0 on HTP0: pp128 101.8, tg32 18.0 t/s. Qwen3-14B-Q4_0 (7.92 GiB) RUNS via 4 HTP sessions + hybrid -ngl 34: pp64 40.9, tg16 11.2 t/s — the 14B that ORT-QNN could NOT run (>10 GB wall) now runs on the NPU. Hard HTP limit found: exactly 4 sessions × ~2 GB = ~8 GB resident (5th fails 0x200) — the root of the ceiling; >8 GB models use hybrid -ngl offload to GPU/CPU. all layers offloaded; 4 HTP sessions open for multi-session split. The ORT-QNN ~10 GB ceiling is bypassed (llama.cpp splits across HTP sessions + GPU/CPU over unified mem). Unlocked by: signed skel catalog (WDK inf2cat + -DGGML_HEXAGON_HTP_CERT) and ADSP_LIBRARY_PATH→skel+cat dir (the final 0x80000406 fix). Repro: docs/llama_hexagon_build_setup.md, [[reference_llamacpp_hexagon_npu_works]]. 27B blocker: the Qwen3.6-27B-MTP-Q4_0.gguf is arch qwen35 — not yet supported by llama.cpp (fails to load on every backend); needs qwen35 arch support added. Next demo: convert Qwen3-14B (standard arch) → GGUF Q4_0 → run on NPU multi-session.

STRATEGIC PIVOT (for the 27B end-goal): ORT-QNN loads contexts fully resident → fundamentally capped at ~10 GB on this box → can never fit a 27B. Researched the ceiling (docs/htp_memory_ceiling_problem.md): it's a real, documented HTP/session limit (~2 GB/session) — llama.cpp's Hexagon backend is the answer (per-session layer split via GGML_HEXAGON_NDEV + -ngl NPU/GPU/CPU hybrid over unified 48 GB; builds libggml-htp-v81.so = our arch; GGUF Q4_0). Staged the build on the X2E (Hexagon SDK 6.6.0.0 + Adreno OpenCL SDK 2.3.2 downloaded + extracted to C:\Qualcomm; clang/cmake/ninja present; cmake --preset arm64-windows-snapdragon-release configures clean; build running). Remaining: user does test-signing + self-signed HTP cert + reboot (docs/llama_hexagon_build_setup.md), then run 4B→14B→27B with hybrid -ngl.

✅ Ground-truth IO contract extracted. Bundle shipped with no bin_info/; regenerated it with qnn-context-binary-utility per .bin. Confirms topology embed + 8×(5-layer) + lm_head-alone = 10 parts, live attention_bias[1,1,1,512] input (not the 4B folded ScatterND mask), rope_theta=1e6. Mirrored (tracked) to npu_engine/bin_info_14b/.
✅ General N-part ORT-QNN engine (npu_engine/engine_14b.py) — reads bin_info, wires seam + live attention_bias + full-dim RoPE + KV ring, supports separate-session and combined-EPContext-group loading, has a ceiling-probe mode. Validated: part1 embed lookup runs; combined 2-EPContext-node-per-session wrapper works (the >7-session fix — part1+part10 in one QNN session, finite logits). The 4B name-collision failure does NOT recur for these distinct AR1 parts.
✅ CPU fp reference (npu_engine/ref_cpu_14b.py, transformers 4.57): coherent — '<think>\nOkay, the user is asking…', first-decode argmax 151667. Saved results/pathb_eval/ref_cpu_14b.npz for cos-sim.
🛑 BLOCKER — the bundle can't run as built. Decoder parts (2-9) fail to create an HTP context (QNN 1002) alone — not a session-count issue. Root cause (3× confirmed): the "w8a16" build stored fp16 weights, not int8 (constSize 3.30 GB = exactly 2 B/param; all IO FLOAT_32; qairt-quantizer ran with no --input_list calibration + --preserve_io_datatype). No activation calibration → HTP compiled a float graph → weights materialized to fp16 → 2× oversize (3.30 GB) → exceeds the ~2 GB X2E runtime per-context ceiling (part1/10 at 1.56 GB load; Qualcomm's 7B parts ≤1.09 GB load). Calibration here is required for loadability, not just accuracy.
⏭ NEXT — rebuild on the Threadripper from saved 06_split ONNX: re-quantize with --input_list calibration + quantized (uint16/uint8) IO → ~1.65 GB int8 decoder parts that load under the ceiling. Then engine_14b.py runs the full chain end-to-end + logit cos-sim vs the CPU ref. Spec in docs/npu_engine_14b_runtime.md §5.

AUTONOMOUS RUN 2026-06-15 (build server) — stood up the Threadripper

While the user is away (~8h), built out a remote build server on the unRAID Threadripper (192.168.10.5, 32t/125 GB/8 TB SSD) to run the heavy NPU-bundle steps the X2E's 48 GB can't. Full playbook: docs/threadripper_build_server.md. Progress:

✅ Build server live — key-based SSH, Docker, code on the 8 TB SSD, .venv-box (torch/onnx/transformers/optimum + optimum-onnx==0.1.0), Qwen3-14B downloaded (28 GB), QAIRT 2.45.40 Linux x86_64 extracted from Qualcomm.zip (exact version match, no download needed), and a reusable specula-qairt:2.45 Docker image (libc++ + numpy-1.x) with all three QAIRT tools verified working.
✅ Stages 1–5 running natively (export → rewrites) — the exact steps that OOM'd/thrashed on the X2E now run with headroom (export peaked ~114 GB; added a 64 GB SSD swap backstop, untapped).
✅ Full ONNX pipeline ran on the box — stages 1–5 (export → 4 rewrites)
- split into 8 parts. Every step that OOM'd/thrashed/capped on the X2E now runs (export peaked ~114 GB, 64 GB SSD swap backstop untapped).
✅ 🏆 part1.bin (1.56 GB) — a real X2 Elite HTP context binary built END-TO-END on the box (export→rewrite→split→convert→quantize→context-bin), no AIMET, no cloud. The no-AIMET w8a16 chain (qairt-converter → qairt-quantizer --weights_bitwidth 8 --act_bitwidth 16 → qnn-context- binary-generator) is proven. The build-server architecture works.
✅ Fixed many 14B-scale bugs (all committed): optimum-onnx pin; protobuf 2 GiB cap ×3 (rewrite prune ×2 + split extract_part — the 3.1 GB embed/lm_head; in-place / streaming fixes); split mask threading (transformers 4.57 emits a live attention_bias, not the old folded mask); QAIRT env (onnx==1.18.0 for onnx.version, onnxsim to fold the rotary rotate_half head_dim 127→128, QAIRT_TMP_DIR on the SSD, libc++).
✅ 🏆🏆 COMPLETE Qwen3-14B w8a16 NPU bundle built on local hardware — runs/.../10_bundle/qwen3_14b-w8a16-specula-x2e/: 10 ordered HTP context binaries (part_1..10_of_10.bin, 28 GB) + genie_config.json + htp_backend_ext_config.json + tokenizer/config + metadata, genie-shaped. Built end-to-end on the Threadripper, no AIMET, no cloud. On the X2E at models/qwen3_14b-w8a16-specula-x2e/, sha256-verified 10/10 vs the bundle manifest (the first tar-over-ssh pull stalled on the WiFi link; a per-file scp loop with ServerAliveInterval keepalive + size-resume + retry completed it bit-perfect — single-stream-no-keepalive was the zombie cause). Bundle also lives on the box (runs/.../10_bundle/). 385 GB of regeneratable intermediates (01-04, 07-08) cleaned from the SSD; kept 05/06 (rebuild checkpoints) + 09_bin + 10_bundle.
Split-balancing findings (per-context HTP limits): a 10-layer part (~13 GB) failed qnn-context-binary-generator with QNN 1002 (graph finalize) — the HTP per-context ceiling is ~5 layers / ~3.3 GB; and the 3.1 GB lm_head must be its OWN part (breaks the converter's symbolic shape inference if mixed with attention layers). Final layout: embed + 8×(5-layer) + lm_head = 10 parts. Directly informs how to split the 27B.
⏭ DEPLOY caveat (next, for the user): loading all 10 parts on the X2E NPU will hit the ~7 ORT-QNN HTP session ceiling ([[reference_ortqnn_session_limit]]) — needs the combined-wrapper / sidecar to load. The build is done; running the 14B on-device is the next runtime step. No blocker on the build side; all findings in docs/threadripper_build_server.md.

▶ NEXT SESSION STARTS HERE: docs/next_session_npu_engine_14b.md — run the 10-part 14B bundle on the Hexagon (solve the >7-session ceiling). Also queued: docs/qwen3_14b_w4a16_plan.md — try w4a16 by re-quantizing the saved 06_split on the box (qairt-quantizer --weights_bitwidth 4; AIMET only for higher quality + needs a GPU; AI Hub is int8-only).

Last updated: 2026-06-15 (session 36 — kicked off the Qwen3.6-27B NPU port. New direction: push a 27B (then 35B-A3B) onto the Hexagon NPU — uncharted, a deliberate community/clout play. On-device-first; AI Hub for the physically-impossible pieces (free, key on box); w8a16 first so a first bundle needs no GPU/cloud.)

Landed this session:

Snag 1 done — end-to-end/lib/model_config.py is hybrid-aware (qwen3_6 family + block_types/attention_layer_indices/SSM dims), pinned against the real Qwen/Qwen3.6-27B config.json. It's a VLM (qwen3_5 arch, LLM dims under text_config), 16 full_attention @ [3,7,..,63] + 48 linear_attention, partial rotary 0.25, mRoPE [11,11,10]. Verified + dense-4B regression clean. Commits 9899603, real-config pin, doc updates.
Op-compilability probe (end-to-end/probes/op_compilability_probe.py, qwen3_next proxy, 0.79M params, no download/GPU): the gated-delta-net SSM op does not export to ONNX with stock exporters — dynamo hits a data-dependent .item() in _update_linear_attn_mask; legacy hits the vmap chunked-delta custom autograd. Full writeup + path-forward: docs/qwen3_6_27b_op_compilability.md.
Option A — SSM WALL CLEARED ✅ (the win of the session). Two math-equivalent patches (static linear-attn mask dropping the .item(); chunk->recurrent gated-delta-net) + dense-FFN config (real target is NOT MoE) → dynamo export SUCCEEDS: 918 nodes, custom domains NONE, no Scan/Loop/NonZero/If. The gated-delta-net decomposes to standard ONNX. Verdict flipped from "doesn't export" to "exports to standard ops." Only a few ops to HTP-validate (Where×4, ScatterElements, IsNaN, Softplus); the per-step unroll is O(seq) so production prefill still needs a chunked/Scan/windowed recurrence — an engineering problem, not an op-support wall.
Added onnxscript 0.7.0 to .venv-arm-export (dynamo exporter dep). Left .venv-qairt/.venv-ort21 frozen (QNN version lock).
Stage 2 — HTP COMPILE PASSES ✅✅ (the headline). Submitted the self-contained Option-A ONNX to AI Hub submit_compile_job on Snapdragon X2 Elite CRD (qnn_context_binary). Job j5qw8d6m5: CREATED -> OPTIMIZING_MODEL -> SUCCESS. The X2 Elite QNN compiler accepts the gated-delta-net op set (incl. Where/ScatterElements/IsNaN/ Softplus/Conv) and emits an HTP context binary. So the SSM op both exports to standard ONNX AND compiles to the Hexagon HTP — it is NOT a fundamental wall. Probe: end-to-end/probes/aihub_compile_probe.py. Caveats: proxy (qwen3_next) not real qwen3_5; op-support not numerics/perf; seq=8 unroll (production needs chunked/Scan/windowed recurrence).
Stage 3 — HTP NUMERICS MATCH ✅✅✅ (the capstone). Ran the compiled binary on real X2 Elite silicon (AI Hub submit_inference_job, job jp38krql5) and compared logits to eager torch: cos 0.99999, max abs diff 0.0045, last-token argmax MATCH, top5 5/5. The gated-delta-net computes correctly on Hexagon. Probe: end-to-end/probes/aihub_inference_probe.py. All three stages green (export ✅ / compile ✅ / numerics ✅) — the single biggest unknown of the 27B NPU port is answered YES.
Qwen3-14B w8a16 stepping stone — downloaded + routing decided. Got the 8-shard HF model (models/Qwen3-14B, standard dense Qwen3, 40 layers). Finding: the e2e quantize_to_npu.py is RunPod/CUDA-designed — stage 6 (aimet_onnx, no ARM wheel) + DEFAULT_VENV=/workspace/...cu121 + seq-mse/ada-scale defaults make w8a16 fully local impossible via that script. Local-first route instead: optimum export + pathb rewrites on-device (stages 1-5, CPU), then AI Hub submit_quantize_job (w8a16, free, no GPU) + submit_compile_job — matches the on-device-first / AI-Hub strategy (AI Hub already proven end-to-end on the SSM probe). EXPORT DONE ✅ — the optimum_export_4b.py wrapper produced a valid 111 GB fp32 ONNX (runs/qwen3_14b_w8a16/01_optimum/model.onnx{,_data}, [post-fix] OK: 83 inputs, 81 outputs, loads in ORT). 14B export survives on 48 GB ARM — the wrapper streams weights per-tensor (peak process RAM ~1.2 GB). Scaling question answered. Route refinement: the 111 GB fp32 ONNX is too big to upload to AI Hub, so w8a16 quant goes local via qairt-quantizer (on-device, ~14 GB output; the 4B reproduction already runs it on Prism) — AI Hub only for the final compile if needed. Even more on-device than planned.

Two scaling walls found; one solved. (1) Disk — SOLVED. The 118 GB ONNX was NOT 2× weights (only 59 GB is referenced — 443 inits, correct for 14.8B fp32); the post-fix onnx.save left ~59 GB dead space. New tool end-to-end/scripts_helper/repack_onnx_external_data.py streams the referenced bytes into a tight 59 GB file (≤1 tensor in RAM, 113 s; validated byte-identical). Deleted the 118 GB original → 284 GB free. (2) RAM — OPEN (the live blocker). rewrite_qwen3_htp.py:100 does onnx.load(load_external_data=True) → pulls all 59 GB into RAM; on 48 GB it thrashes on swap (sys free pinned ~0.1 GB, no progress in 60 s) — confirmed + killed. Fix design: make the pathb rewrites graph-only (load_external_data=False, manipulate structure, save proto-only) sharing ONE external-data file across all stages — the rewrites never touch weights, so the 59 GB file is written once and every stage references it. That makes the chain near-zero extra disk AND RAM, and is exactly the infra the 27B needs too. Real refactor of rewrite_qwen3_htp / rewrite_qwen3_pathb / pin_shapes (not yet done).

Next (14B): build the graph-only / shared-external-data rewrite infra (the RAM-wall fix above), then pathb rewrites → split → qairt-converter + qairt-quantizer w8a16 (local PTQ, no AIMET) → context-bin-gen → bundle. Next (SSM): recurrence-structure at real seq (chunked/Scan/windowed — the seq=8 unroll is the last gap to a production prefill graph); repeat the 3-stage SSM proof on the real qwen3_5 arch when transformers ships it. Workstream map in README "Active workstream"; charter in docs/qwen3_6_27b_npu_kickoff.md.

Last updated: 2026-06-12 (session 35 — scheduled backend refresh: llama.cpp +489 commits (one month) → rebuilt all 4 ARM backends at e37abd6b5; smoke-passed on battery; real perf sweeps deferred to AC per user.)

Stale-fetch gotcha first. git status reported "behind 10" and a first git fetch confirmed 10 — both stale. A verbose fetch + GitHub API cross-check showed origin/master was actually e37abd6b5 (2026-06-12), i.e. 489 commits / ~1 month ahead of our HEAD 856c3adac (session 25). Lesson: cross-check the API tip when a fetch count looks implausibly small.

Updated + rebuilt. Discarded the local kleidiai.cpp SME-detect mod (regenerated by the build script), git pull --ff-only to e37abd6b5, then rebuilt build-cpu / build-cpu-kleidiai / build-opencl / build-vulkan via scripts/build_llama_cpp.ps1 (all exit 0). The patch_kleidiai_detect.py Windows SME patch still applies cleanly — upstream rewrote detect_num_smcus (+272 lines, now has an if (!ggml_cpu_has_sme()) return 0; guard + a NEON-fallback warning) but kept the exact #else return 1; Windows fallthrough the patch targets, and base SME is still present on Oryon v2 so the patch is still load-bearing.

What's in the 489 commits (backend-relevant):

MTP self-draft MERGED to mainline — llm_graph_input_mtp #23643, qwen35: post-norm hidden state for MTP #24025, backend-sampling path #23287, padding/D2D cleanup #24086, plus Gemma-4 MTP. This is the PR #22673 session 27 was waiting on → build-opencl-mtp is now redundant (mainline build-opencl should run MTP); pending on-device verification.
Adreno OpenCL MoE kernels — MoE q4_k/q5_k/q6_k #23303, generalize MoE on M #23449, OP_GATED_DELTA_NET #23312 (the Qwen3.6 SSM op), MoE warmup-crash fix #22876, q5_0/q5_1 gemm/gemv. Directly targets session-27's "Qwen3.6 MoE won't GPU-offload on OpenCL (clCreateImage -40) + TG collapse to 13 t/s".
Vulkan — ~14k lines of shader work (mul_mat_vec, mul_mm, gated_delta_net, ssm_conv).
ARM CPU / KleidiAI — kleidiai.cpp +272, ops/vec/simd changes.
spec-decode CLI args changed (#22964) — verify llama-speculative flags before MTP runs.

Mini smoke tests (battery, NOT perf records — p8/n8/r1 single rep):

backend	pp8	tg8	note
CPU `-t 16`	292	58.7	coherent gen confirmed (4B Q4_0)
OpenCL/Adreno `-ngl 99`	42.5	27.4	Adreno device init + offload OK
Vulkan `-ngl 99`	145.2	36.8	device OK (fp16:1, KHR_coopmat); see correction below

CORRECTION (AC perf run, same session): the Vulkan p8 smoke number was misleading — Vulkan prefill is STILL BROKEN. At a realistic pp512 the Vulkan figure collapses to 6.36 t/s (the same ~6-7 t/s "broken-F16" signature from sessions 25-26). The p8 smoke (145 t/s) hid it because the broken path's per-token overhead doesn't dominate at 8 tokens. Lesson: never conclude "prefill fixed" from a tiny-p smoke — measure pp512. Vulkan TG is fine (~38 t/s). Vulkan remains a research curiosity, not a usable prefill backend.

Note: llama-completion now also drops into conversation mode at EOF (hangs like llama-cli — see [[feedback_llama_cli_hang]]); used llama-bench for non-interactive smoke instead.

Drivers. Adreno 32.0.149.0 (Feb), Hexagon NPU 30.0.220.11010 (Jan — the one suspected behind the Genie DSP transport break). Windows Update offers only an irrelevant audio APO driver; the MU extended catalog service is unregistered (0x80248014). No Adreno/Hexagon driver update is available via WU — newer NPU/GPU drivers must come from the OEM (MyASUS / Qualcomm), a manual elevation+reboot install the agent can't push. Worth a manual check given the Genie break.

AC PERF RESULTS — the update is a MIXED BAG on this hardware. Full data: results/csv/backend_refresh_2026-06-12.md. AC, Balanced plan (only scheme available).

GAINS:

4B Q4_0 TG new high: 55.65 t/s (OpenCL -ngl0 -t16), beats the prior all-time 50.80 (+9.5%). TG up across all backends.
Qwen3.6-35B-A3B now GPU-offloads on OpenCL -ngl99 (pp512 191.7 / tg128 22.8) — session 27 had this FAILING with clCreateImage -40 on SSM tensors. The new Adreno MoE kernels (#23303/#23449) + OP_GATED_DELTA_NET (#23312) fixed it.
MTP runs on mainline (Qwen3.6-27B-MTP Q4_0): +60% TG at n4 (7.75→12.40). → build-opencl-mtp retired. Caveat: mainline MTP accept is far lower than the PR build (37% vs 95.8% at n8), so the sweet spot shifted n8→n4 and absolute MTP numbers need a multi-sample re-run (these are r=1). Likely a mainline MTP-path rework (#23287/#24025) and/or PR-era GGUF MTP-head mismatch.

LOSS (A/B-CONFIRMED REGRESSION):

Dense prefill ~halved. Built the OLD 856c3adac into build-opencl-old and A/B'd it against the new build, same session/power/model: 4B Q4_0 pp512 ngl0 369→195 (−47%), ngl99 -ub512 544→232 (−57%). OLD reproduces the session-26 records today → genuine regression in the 489-commit window, NOT power/thermal/ubatch (all ruled out). Both ngl0 (CPU matmul) and ngl99 (OpenCL matmul) regress ~equally → shared upstream cause (batching / graph-build / common op). 35B ngl0 PP shows it too (190→151). Kept build-opencl-old as the known-good prefill build.
Vulkan prefill still broken (pp512 6.36; the p8 smoke 145 was misleading — never trust tiny-p for prefill).

Next: bisect the prefill regression across [856c3adac..e37abd6b5] (suspect a batching/graph or common-op change; both matmul backends hit equally) and report upstream; multi-sample re-run of mainline MTP accept; decide adopt-new (decode + 35B offload) vs stay-old (prefill). For now: new build for decode-heavy / 35B GPU offload; old build for prefill-heavy / long-context. NPU/ORT-QNN stack left untouched (version-pinned to QAIRT per [[reference_ort_qnn_qairt_match]]).

Last updated: 2026-05-22 (session 34 — long-context prefill sweep for Qwen3.6-35B-A3B: the -ngl 0 "blended" default is a TTFT loss for long prompts. Track D, results/csv/track_d_longctx_pp_2026-05-22.md.)

Triggered by external advice to try --n-cpu-moe (experts-on-CPU / attention-on-GPU split). That specific recommendation was already disproven — results/csv/track_b_ncmoe_sweep_2026-05-13.md shows -ncmoe loses monotonically (TG 17.3→12.0 as N rises). But the advice's underlying point — GPU offload should help prefill more at long context — is correct, and the lever is plain -ngl 99, not the split.

Measured (OpenCL, -t 16, FA-off; -fa 1 is a ~35% prefill regression on Adreno OpenCL — pp2048 119 vs 178 t/s — so FA stays off):

`-ngl`	`-ub`	pp512	pp8192	pp32768
0	512	~190	148.8	90.0
0	2048	—	129.4	81.8
99	512	~197	159.5	114.6
99	2048	—	174.4	(pending)

-ngl 99 beats -ngl 0 on prefill, gap widens with context — +4% at 512, +7% at 8K, +27% at 32K (114.6 vs 90.0). For a coding agent's TTFT (long prompt, prefill-bound) the GPU path wins.
-ub cuts opposite ways — bigger -ub hurts the -ngl 0 coprocessor path, helps -ngl 99. Best prefill config measured: -ngl 99 -ub 2048 = 174.4 t/s at pp8192, +17% vs the blended -ngl 0 default (148.8 t/s).
TG unchanged: -ngl 0 still owns decode (~31 t/s vs ~13–17 for ngl99). So the 35B default should be context-dependent — -ngl 99 -ub 2048 for prefill-heavy / long-context turns, -ngl 0 for decode-heavy / short-prompt.

Sweep stopped early (laptop needed for other work). Pending — overnight via scripts/track_d_overnight_2026-05-22.ps1: the ngl99/ub2048/pp32768 cell (A1 dropped it — investigate, likely a runtime error), PP at 131072 for ngl {0,99} (and whether ngl99 OOMs: 20.2 GB model + 128K KV in 24.4 GB GPU memory), and TG-128 at depth 4K/32K/128K. Run powershell -File scripts/track_d_overnight_2026-05-22.ps1; results land in results/csv/track_d_longctx_pp_2026-05-22_tail.md.

Last updated: 2026-05-22 (session 33 — side quest: Gemma 4 → Hexagon NPU. Scaffolded a new gemma-pipeline/ subdirectory (sibling of end-to-end/) for a non-RunPod attempt at converting Gemma 4 E2B to a w4a16 NPU bundle for the npu_engine on the Hexagon laptop. Gemma 4 released 2026-04-02; E2B is the smallest (35-layer text decoder, 128k native ctx). Built + tested lib/model_config.py (Gemma-4-aware ModelInfo — dual RoPE, Per-Layer Embeddings, KV sharing, sliding window); wrote ARCHITECTURE_NOTES.md (Qwen3-vs-Gemma4 difference matrix), an honest orchestrator, an AI Hub launcher, and the spec for 4 Gemma-specific rewrite scripts. Key upside: Gemma 4's native sliding-window attention removes the 32k VTCM wall that killed the dense Qwen3-4B long-context effort (sessions 31–32). Blockers on the x86 dev box: AIMET needs CUDA (Intel B50 is not), the NPU is the other machine, no AI Hub token found yet. Next: dig into Qualcomm AI Hub — whether it has a Gemma 4 recipe, else do the graph surgery ourselves and submit jobs. See gemma-pipeline/STATUS.md.

Last updated: 2026-05-22 (session 32 — decision: pivot the NPU long-context target off dense Qwen3-4B onto the Qwen3.6-27B hybrid.

Session 31 closed on a user checkpoint with three options (ship the uint8-KV ctx-4096 A/B, probe the dense ceiling, or start SWA design). Resolution: none of the three as scoped. The session-31 TCM-tiling wall (long_context_scaling.md §8.8) makes dense 32k/64k structurally impossible, and a 4k dense ceiling is not a useful long-context deliverable. SWA is still the fix — but building it on Qwen3-4B is wasted work: Qwen3-4B was not SWA-trained (quality-lossy), and the GGUF geometry shows the per-attention-layer KV slice is identical on the real target anyway.

The call: take the RunPod conversion pipeline straight to Qwen3.6-27B, then 35B-A3B. Verified from the GGUF (gguf_dump.py on models/Qwen3.6-27B-MTP-Q4_0.gguf): arch qwen35, 65 blocks, full_attention_interval=4 → only ~16 global-attention layers carry an O(ctx) KV cache; the other ~48 are SSM/Mamba2 with O(1) state and scale to the native 256k context for free. SWA is still needed on the 16 attention layers (same 32 MiB KV slice → same TCM wall) but is now "with the grain" of the hybrid, not a quality hack — the SSM layers already carry the long-range path.

Kickoff brief committed (docs/qwen3_6_27b_npu_kickoff.md, commit b962a77) for the RunPod team: lists the hybrid-awareness snags (lib/model_config.py per-block type map, rewrite_qwen3_pathb.py SSM skip, lib/split.py KV-on-attention-layers-only, SWA on the 16 layers, genie/ORT-QNN KV manager) and the first milestone — a loadable, on-device-correct 27B bundle at ctx 32768.

Next. RunPod team works the kickoff brief: hybrid-aware model_config/rewrite/split first (verify at ctx-512), then SWA, then the ctx-32768 build and the ctx sweep to 256k. The uint8-KV ctx-4096 Qwen3-4B A/B is the honest close-out of Qwen3 NPU (matches Qualcomm's cl4096 limit) and can be shipped opportunistically — it is no longer the long-context goal.

Last updated: 2026-05-22 (session 31 — long-context build campaign: ctx-parametric pipeline landed, but the many-parts approach is a dead end — uint8 KV is the required fix. Goal was the first real long-context Qwen3-4B NPU bundles (ctx 32768 / 65536). Full design + implementation log: docs/long_context_scaling.md §8; on-device evidence: docs/2026-05-21_specula_bundle_npu_testing.md.

Pipeline fixes landed (commit 66feb23, "e2e: ctx-parametric pipeline + decoupled-calibration long-ctx builds"):

--ctx beyond 512 never actually worked — real bug fixed. pin_shapes_qwen3_4b.py only rewrote symbolic ONNX dims; the frozen attention_mask initializer stayed at the traced ctx, so the folded causal mask was capped at [1,1,1,512] and qairt-converter died on the first broadcast. Fix: pin_shapes now also rewrites the attention_mask initializer to [1, ctx]. --ctx is genuinely parametric for the first time (verified at 65536). The cl{1024,2048,3072,4096} ctx sweep (e2e task 11) had never been run — it would have hit this immediately.
Decoupled calibration ctx from compile ctx. AIMET at long ctx is a non-starter (calibration samples carry the full fp32 KV — ~1.2 TB at 32k). Fix: calibrate once at ctx 512, compile any ctx tier from that one encodings file — sound because the additive mask makes activation ranges ctx-invariant, and it is what Qualcomm does (one calibration → 5 shipped tiers). compile_split_bundle.py --pathb-dir re-pins the pre-AIMET graph and splits that, paired with the ctx-512 encodings; ctx-aware stage dirs; --jobs for parallel qnn.
NTK rope-theta for ctx beyond the 40960 trained window (theta' = theta · s^(d/(d-2)), a genie-config knob; 32k is in-window, 64k → theta' ≈ 1.61e6).

The dead end (the important finding). To dodge qnn-context-binary-generator's ~3.5 GiB per-part serializer ceiling at long ctx, the build had to split into many parts — 19 for ctx 32768, 37 for ctx 65536. The on-device test proved this is unloadable: the HTP has a hard ~4-5 co-resident context-binary session ceiling, and ORT-QNN fails to load the 19-part ctx32768 bundle at part 5 (QNN error 1002). The 37-part 64k build was abandoned. Splitting finer to clear the compile ceiling only trades it for the session ceiling — there is no part count that satisfies both at 32k/64k with the current fp32-KV graph.

Root cause + real fix. The pathb graph passes the KV cache as fp32 (past_key_values.* / present.*). That fp32 KV (a) blows the per-part compile estimate (~1.66 GB/layer at 64k) → forces the unworkable part count, (b) makes pathb decode ~4.5× slower than the Qualcomm reference (the on-device test measured ~150 MB in + ~150 MB out of KV IO per decode step). Qualcomm's reference bundle uses uint8 KV (their metadata.json: past_key/value uint8, asymmetric, zero_point 128). uint8 KV is the fix — 4× smaller → 32k/64k fit in ≤8 loadable parts AND ~4× faster KV IO. For ctx beyond ~8-12k it is mandatory, not an optimization.

Scope decision. Re-exposing attention_mask as a graph input (which would make bundles Genie-loadable) was dropped — the project uses its own ORT-QNN engine (npu_engine/bench_pathb_ortqnn.py), not Genie, and that engine already works with the folded mask; attention_mask-as-input is purely a Genie requirement.

uint8 KV implemented — and it cleared the allocation ceiling but exposed the real wall. lib/aimet.py::_apply_uint8_kv (commit 36b7a3f) quantizes the 72 KV-cache I/O tensors to 8-bit via qai-hub's _set_tensors_to_output_8b_sym + _tie_quantizers_for_kv_cache; qairt.py gained a preserve_io flag (compile_split_bundle --quantize-io) so the 8-bit encoding reaches the graph boundary. w4a16 ctx-512 uint8-KV calibration: probe cos 0.9703 (only −0.005 vs 0.9758 — KV-only quant, no 16x8-matmul lever; argmax still ' Paris').

The decisive wall (commit ea320bc, long_context_scaling.md §8.8). The w4a16 ctx-32768 uint8-KV build (6 parts, --quantize-io) cleared the 3.5 GiB allocation ceiling — then failed HTP graph-prep at tcm_migration.cc: a per-layer uint8 KV InputSlice ([1,8,32768,128] = 32 MiB) cannot tile into on-chip VTCM (~8 MiB). This is a fundamental per-op HTP limit — not part count, not KV dtype (uint8 already applied) — and it caps dense global-attention ctx at ~4–8k. It is why Qualcomm's reference stops at cl4096.

Conclusion: 32k/64k dense global attention is NOT HTP-compilable with the current pathb graph. The genuine fix is sliding-window attention (long_context_scaling.md §5.2) — a fixed W-token window makes the KV slice always W-sized — now promoted from optimization to the load-bearing requirement. The flag-level fixes are conclusively exhausted (5-wall ceiling stack, §8.8).

Shippable now (not the 32k/64k goal, but real): uint8-KV bundles at ctx ≤4096 — validates uint8 KV end-to-end + the ~4× KV-IO throughput win. Next session direction (user checkpoint): decide between shipping the uint8-KV ctx-4096 w4a16/w8a16 A/B, probing the exact dense ceiling (ctx 8192), or starting the SWA design — the real long-context project. The w8a16 ctx-512 uint8-KV calibration was left running to completion (reusable artifact for a future ctx-4096 build).

Last updated: 2026-05-21 (session 28 — first on-device test of the three specula-built Qwen3-4B pathb NPU bundles from the RunPod cloud-GPU pipeline. Full writeup: docs/2026-05-21_specula_bundle_npu_testing.md.

Results. The two ctx512 bundles run through our ORT-QNN runtime and decode coherent on-topic text; w8a16 is visibly more fluent than w4a16. ORT-QNN throughput (256-tok prefill + 128-tok greedy decode, AC): specula w4a16 ctx512 PP 5.26 / TG 5.21 t/s; w8a16 ctx512 PP 4.48 / TG 4.40 t/s; Qualcomm control (re-measured today) PP 1604 / TG 23.4 t/s. First-decode logit cos w4a16-vs-w8a16 0.972, same argmax.

Three blockers found:

Genie can't load any pathb bundle — fold-pathbmask removes the attention_mask input Genie's KV-cache manager requires (Failed to create the dialog).
Genie DSP transport is broken on this machine today — even the Qualcomm control fails (DspTransport.openSession qnn_open failed 0x80000406, skel-load 1002) despite the v81 skel being present. ORT-QNN works on the same HTP — looks like NPU-driver drift (driver 30.0.220.11010, 2026-01-26). Genie produced zero numbers.
ctx32768 bundle (19 parts) exceeds the HTP session ceiling — ORT-QNN load fails at part 5 (QNN 1002). 19 co-resident contexts is not achievable under any runtime; fix is re-splitting into ≤8 parts, not a swap engine (Genie has no swap mode anyway).

Why pathb TG is ~4.5× slower than control — measured: added fully-static IOBinding to the harness + per-part profiling. IOBinding changed TG by nothing (5.2→5.5 t/s, noise). Per decode step: parts run on-device 1.6/52/52/60 ms, host KV-roll only ~10 ms — i.e. ~165 of ~175 ms is the QNN graph on the HTP. The gap is 100 % bundle design (FP32 KV IO, FP32 cross-part IO, AR1-only prefill), not harness overhead. Each 12-layer pathb part is ~3.7× slower on-device than the control's. Fix is entirely upstream on RunPod.

New tooling: npu_engine/bench_pathb_ortqnn.py (generic pathb ORT-QNN driver, static IOBinding, per-part profiling). Next: quantize KV to uint8 + use quantized cross-part IO + emit AR128 graph + re-expose mask in the pipeline; root-cause Genie's DSP transport break.

Last updated: 2026-05-13 (session 27 — Qwen3.6-27B MTP first numbers. Built PR #22673 (gg/spec-mtp-experiments rebased) at e7b484815 into build-opencl-mtp/ so the mainline binaries stay intact at 856c3adac. Downloaded unsloth/Qwen3.6-27B-MTP-GGUF Q4_0 (16 GB) and Q8_0 (29 GB; resumed once after silent truncation). Mainline cannot load these GGUFs — block_count=65 includes the MTP head, which has attention-only tensors that mainline expects to be full SSM+attention. Only the PR build can load them.

MTP TG wins: Q4_0 27B -ngl 0 -t 18 --spec-type draft-mtp --spec-draft-n-max 8 → TG 12.17 t/s vs 8.39 baseline (+45%), acceptance 95.8%. Q8_0 at same config → TG 7.98 t/s vs 5.13 (+55%), acceptance 95.8%. Accept rate plateaus at 98-100% from n_max=2 through n_max=6, only drifting to 94% at n_max=16. The unsloth MTP head is well-trained. Peak Q4_0 win at n_max=12 (TG 12.25, +46%); peak Q8_0 win at n_max=8 (TG 7.98, +56%).

PP takes a ~22% hit when MTP is on (Q4_0 PP 63→49, Q8_0 PP 44→36) — prefill has to initialize the MTP head's KV state. Crossover prompt length where MTP-on still wins overall (for 256-token answer): ~1.8k tokens for Q4_0. MTP is the right call for any prompt shorter than that, which covers chat / structured-output / shortlist RAG.

Hardware correction from session 26. User caught a mistake: I quoted Adreno's 24 GB cap as a hardware limit. The 24 GB is the default OpenCL view; GGML_OPENCL_ADRENO_USE_LARGE_BUFFER=1 unlocks the cl_qcom_large_buffer extension which is present on our driver, allowing access to the full 44 GB BIOS allocation. Rerun on 4B Q4_0 / 35B MXFP4 at -ngl 99 shows the flag is ~5% overhead for models that already fit; it matters only when the model exceeds 24 GB. The session-26 numbers stand within normal variance (rerun 4B Q4_0: PP 564 / TG 24.7 without, PP 524 / TG 26 with — vs session-26 single-run 586 / 26.7).

Qwen3.6-27B does not GPU-offload on OpenCL independent of LARGE_BUFFER. -ngl 99 fails with clCreateImage error -40 at ggml-opencl.cpp:11358 — the SSM (ssm_a, ssm_conv1d) tensors have shapes the Adreno image-memory path doesn't accept. -ngl 0 (coprocessor) is the only OpenCL path for 27B.

Companion writeup: docs/2026-05-13_qwen3_6_27b_mtp.md. CSVs: results/csv/qwen3_6_27b_{Q4_0,Q8_0}_mtp_sweep_2026-05-13.md, results/csv/track_c2b_large_buffer_reruns_2026-05-13.md.

Next: monitor PR #22673 for merge (cleanest path is to fold MTP into mainline once it lands); MTP perplexity sanity-check (verify accept rate translates to bit-equal outputs at temp=0); concurrency-4 with MTP enabled (np > 1 interaction unmeasured); compare 27B-MTP TG vs 14B CPU+CPU spec decode for the quality/speed Pareto.

Last updated: 2026-05-13 (session 26 — overnight perf sprint on 856c3adac (no rebuilds). Three new records on this hardware. Headline #1: Qwen3-4B Q4_0 on build-opencl with -ngl 0 -t 16 → PP 379 / TG 50.80 t/s (r=5, σ_TG=0.4%). New all-time TG, beats NPU ORT-QNN (29.03 t/s) by +75%. Adreno backend registered but no layers offloaded — appears to act as a coprocessor while weights live on CPU. Headline #2: same model with -ngl 99 -t 16 -ub 512 → PP 586 t/s — new non-NPU prefill record. Headline #3: Qwen3.6-35B-A3B MXFP4_MOE on OpenCL -ngl 0 -t 18 → PP ~190 / TG ~31 — equivalent to pure CPU on prefill, ~12% faster on TG, DOUBLE the GPU-offload TG; new "blended" default for 35B inference. Partial -ngl (8..56) collapses TG into a 16-19 t/s valley (sequential per-layer CPU↔GPU sync); -ncmoe hurts monotonically. Concurrency-4 (4 agentic streams × 512+128 toks) on OpenCL -ngl 0: 4B Q4_0 hits aggregate S_TG 126.94 t/s, 35B MXFP4 hits 65.63 t/s — agentic-scale TG is real on this laptop.

Other findings tonight:

CPU thread sweet spot: -t 16 (not default -t 18) wins for Qwen3-4B Q4_K_M on build-cpu-kleidiai (PP 257.68 / TG 40.37); for 35B-A3B -t 18 still wins. Rule of thumb: 4B = phys_cores−2, 35B = phys_cores.
Q4_0 beats Q4_K_M for Qwen3-4B on this hardware on every backend. CPU-kleidiai: Q4_0 TG 42.55 vs Q4_K_M 31.66 (+34%); OpenCL -ngl 0: 50.50 vs 44.60 (+13%); OpenCL -ngl 99: 26.84 vs unstable. Migrate production target to Q4_0 (pending perplexity sanity-check).
Heterogeneous spec decode loses to pure-CPU spec decode. Adreno draft for Qwen3-14B CPU target = 11.97 t/s (-19% vs 14.84 baseline); CPU draft + CPU target = 16.97 t/s (+14% vs baseline). Adreno's small-model dispatch cost on 0.6B Q8_0 outweighs spec savings. Smaller draft (0.6B) beats larger (1.7B) on CPU consistently.
Vulkan TG path partly recovered: `GGML_VK_PREFER_HOST_MEMORY=1
- GGML_VK_DISABLE_MMVQ=1 + GGML_VK_DISABLE_FUSION=1gets Vulkan TG to 41.09 on 4B Q4_0 (vs 38 last session). PP path still broken (~7 t/s). OpenCL-ngl 0` (TG 50.8) cleanly beats this so Vulkan stays a research curiosity.
xmem F16xF32 GEMM (GGML_OPENCL_ADRENO_XMEM_GEMM=1) doesn't help MXFP4 quantized MoE — F16xF32 path doesn't apply to quantized weights. PR #22755 still valuable for dense F16 (none on disk to test).
MTP punted. PR ggml-org/llama.cpp#22673 still OPEN. Many MTP-preserved Qwen3.6 GGUFs now on HF (havenoammo / unsloth / am17an / localweights) but useless without runtime support. Watch the PR; re-bench havenoammo/Qwen3.6-35B-A3B-MTP-GGUF when it merges.

Full results: docs/2026-05-13_overnight_perf_results.md (per-track tables + reproducer commands). CSVs: results/csv/track_{b1,b2,b,c1,c2,c3,c4,c5,c6,c_thread_sweep}_*_2026-05-13.{md,log}.

Next: perplexity comparison Q4_0 vs Q4_K_M on 4B (validate the production-target migration); ngram-cache spec decode via llama-server (no extra model needed); watch PR #22673; resume last_side_quest sequence (SQ1 / SQ2 / SQ6).

Last updated: 2026-05-12 (session 25 — scheduled backend refresh + MTP/DFlash/PFlash landscape audit. llama.cpp bumped f53577432 → 856c3adac (186 commits). NPU got ~10% faster across both runtimes on the unchanged Qualcomm w4a16 bundle — Genie PP 1725.65 t/s / TG 26.14 t/s, ORT-QNN PP 2167.11 t/s / TG 29.03 t/s — likely Qualcomm driver / Windows update. CPU PP regressed ~−8% (RMS_NORM+MUL fusion likely suspect); TG flat. OpenCL Q4_0 a wash (+3% / −3.6%). Vulkan regression: prior workaround GGML_VK_DISABLE_F16=1 now STATUS_ACCESS_VIOLATIONs at load. GGML_VK_PREFER_HOST_MEMORY=1 alone still loads but PP stays at the broken-F16 ~6 t/s; TG goes to 38.01 t/s.

Qwen3.6-35B-A3B probe (new model row). Mainline llama.cpp now registers LLM_ARCH_QWEN35MOE etc. The Q4_K_M and MXFP4_MOE GGUFs we already had on disk load on the rebuilt binaries. CPU TG 34.19 t/s on a 35B parameter model is genuinely strong (A3B's 3B active per token doing its job). OpenCL MXFP4 prefill 210.47 t/s beats CPU by +45%, but TG collapses to 13.08 (Adreno MoE dispatch- bound). Vulkan stalls at <1% device utilization with 45 GB resident — the broken-F16 path can't drive 35B MoE prefill. CPU is the right single-backend choice for Qwen3.6-35B-A3B inference today.

MTP landscape. Mainline llama.cpp does not yet have MTP self-draft consumption (PR #22673 still draft on gg/spec-mtp-experiments). Standard --spec-draft-model flags ARE on master. Of the five DFlash/PFlash/MTP-fused forks, all five are unbuildable on this hardware: four are CUDA-only (Indras-Mirror TBQ4, croll83 Blackwell, z-lab DFlash, Luce-Org PFlash); the fifth (antirez) builds on CPU but targets only DeepSeek-V4-Flash. vLLM / SGLang have no Windows ARM64 + Vulkan/OpenCL/Adreno/Hexagon build. Right path: wait for PR #22673 to merge, then re-bench against an MTP-preserved GGUF.

Companion writeup: docs/2026-05-12_sweep_and_mtp_landscape.md covers full sweep + Qwen3.6 probe + MTP/DFlash/PFlash/vLLM landscape + recommendations. Baseline doc update log entry added to docs/qwen3_4b_baseline_all_backends.md. CSVs: qwen3_4b_baseline_2026-05-12_ac.csv, qwen3_4b_gpu_q4_0_2026-05-12_ac.csv, qwen3_4b_ortqnn_2026-05-12_ac.csv, qwen3_6_35b_a3b_baseline_2026-05-12_ac.csv. Local llama.cpp stash stash@{0} holds the obsolete SME-detect patch (no longer needed on 856c3adac — upstream has its own early-return path); safe to drop after patch_kleidiai_detect.py runs cleanly.

Next: file Vulkan upstream issue; watch llama.cpp PR #22673; optional MLC-LLM probe as a non-CUDA vLLM alternative; resume last_side_quest sequence (SQ1 / SQ2 / SQ6).

Last updated: 2026-04-27 (session 24 — last_side_quest umbrella opens; SQ5 long-context NPU closes POSITIVE. New last_side_quest/ workspace at repo root frames the final 6 deliverables before the Qwen3 → Qwen3.5/3.6 graduation: SQ1 heterogeneous-demo (NPU 4B draft

Qwen3-14B CPU target), SQ2 AIMET local venv survey, SQ3 smallest- Qwen-MoE for AIMET → NPU, SQ4 cloud-compute sizing decision, SQ5 long-context NPU, SQ6 small-model server harness for opencode.

Plan doc: last_side_quest/last_side_quests.md. User locked SQ1 target to Qwen3-14B-Q4_K_M, SQ2 first model to Qwen3-0.6B, starting order SQ5 first.

SQ5 result. npu_engine generalized from hardcoded ctx=512 to take any of the bundle's 5 ctx tiers (512 / 1024 / 2048 / 3072 / 4096) via a --ctx-tier flag — backward compat preserved. Pure-Python metadata smoke passes for all tiers; AC NPU benches landed at cl=1024/2048/4096.

ctx	AR1 step	AR1 TG (t/s)	AR128 PP (t/s)	warm-sidecar wall (640 toks)
512 (prior)	36 ms	27.81	2229	(baseline)
1024	36 ms	27.23	—	—
2048	40 ms	25.27	1629	~6 s
4096	47 ms	20.28	1284	~9 s

(All AC, AR128 swap-mode for cl=2048/4096, pp=512 + tg=128.)

Both phases scale sublinearly with ctx. Per-tier 4-partition load cost is FLAT at ~8 s — HTP context init dominates over past-KV size, so mmap/cache wins are unlikely. The cl=512 ~7-session ceiling worry does NOT apply to AR1-alone or AR128-alone at cl=4096; only AR1+AR128 coexistence remains untested at higher tiers (and irrelevant for swap-mode).

Strategic answer for the user's "are we dead in the water for coding-assistant context" question: no. 4K context at 20 t/s decode is comfortably interactive for system-prompt + 1-3 file reads. >4K still routes to the cloud pipeline (docs/one_pipeline_cloud_gpu.md). SQ1's heterogeneous demo can ship at cl=2048 default (25 t/s) with cl=4096 available.

Findings + per-tier analysis in last_side_quest/sq5_long_context_npu/findings.md. Engine edits in npu_engine/{qualcomm_qwen3_4b_oracle,bench_qwen3_4b_ortqnn,sidecar}.py. TODO #1 of docs/npu_engine_todos.md has shipped.

Commits: 29853b5 (engine generalization), c9d8242 (AR1 sweep), 5f30fba (AR128 swap-mode + SQ5 close-out).

Next: pick from SQ1 (heterogeneous demo — needs Qwen3-14B-Q4_K_M download), SQ2 (AIMET venv on a fresh isolated env), or SQ6 (small- model server harness). User to direct.)

Last updated: 2026-04-25 (session 23 — NPU engine v0 sidequest lands. Our ORT-QNN stack now beats Genie at PP (+39%, 2229 vs 1598 t/s) and TG (+19%, 27.81 vs 23.30 t/s) on the same Qwen3-4B w4a16 binary. Sidecar process amortizes the ~15 s HTP context-init across requests (51% faster on pure-AR1 workloads). Phase-batched execution (vLLM-style prefill_all → decode_all) gives 4.00× speedup on N=5 AR128 batched requests. Full writeup in docs/npu_engine_prefill_sidequest.md. Empirical AR128-vs-AR1 swap crossover: 576 prompt tokens.

Last session (22) — Qwen2.5-7B side-quest closes; concurrency-4 (agentic) matrix + NPU-concurrency cliff finding land. Per-parameter scaling story for 4B → 7B is in docs/qwen2_5_7b_baseline_all_backends.md.

What this session set out to do. Test how each compute island scales 4B → 7B on the same X2E silicon (the W1.b roadmap question "what does NPU prefill look like at the 8B target?"), and characterize agentic-workload (concurrency=4) throughput.

Side-quest model picked: Qwen2.5-7B-Instruct. Closest arch neighbour to Qwen3-4B that AI Hub Workbench will compile end-to-end for X2 Elite. Llama-3.1-8B was the alternative but its license blocks Qualcomm from publishing a pre-quantized intermediate, forcing the heavy local FP16 export path — punted to Scenario A in the rewritten docs/rent_cloud_compute.md.

Pipeline learnings while building the bundle. Qwen3-4B was a freebie (precompiled X2 Elite Genie bundle on Qualcomm's public CDN via huggingface.co/qualcomm/Qwen3-4B/raw/main/release_assets.json). Qwen2.5-7B is NOT precompiled — but qai-hub-models auto-downloads a pre-quantized AIMET ONNX intermediate (~30 GB) from the CDN, so the local FP16 materialization step (the OOM-prone 150 GB-RAM warning) got skipped. Workbench compile + link runs in ~20 min total. Three distinct conversion paths now documented in docs/rent_cloud_compute.md (precompiled bundle / pre-quantized intermediate / full FP16 → cloud Linux), with rental scenarios for the cases that need them.

This session landed:

docs/qwen2_5_7b_baseline_all_backends.md — full matrix doc mirroring the 4B doc, covering AC, BAT, AC↔BAT consistency, per- backend detail, concurrency=4, NPU-concurrency experiment, and a per-parameter scaling 4B → 7B post-mortem.
docs/rent_cloud_compute.md rewritten with a decision tree
- two scenarios (Scenario A high-RAM CPU box for FP16 export when no pre-quant intermediate exists; Scenario B existing CUDA SEQ_MSE / AdaScale work, scope unchanged but reframed as one of two).
scripts/bench_qwen2_5_7b_all_backends.py forked from the 4B runner, parser fix to detect last-partition decode graph by regex on _N_of_N$ (4B hardcoded _4_of_4, missed 7B's _6_of_6).
scripts/bench_concurrency4_all_backends.py — drives llama-batched-bench -np 4 -npp 512 -ntg 128 -npl 4 on CPU/KleidiAI/OpenCL for both 4B and 7B. NPU absent (Genie has no concurrency knob).
npu_engine/bench_concurrency4_npu_ortqnn.py — NPU concurrency via spawn-N-procs of bench_qwen3_4b_ortqnn.py. 4B only (the 7B Workbench bundle ships only raw context binaries; no wrapper ONNXs for chained ORT-QNN).
scripts/gen_pp512_prompt_qwen2_5_7b.py — prompt scaffolding fork using the bundle's tokenizer (or upstream cache while the bundle is in flight).
Surgical fix to npu_engine/bench_qwen3_4b_ortqnn.py — skip build_wrapper when the file already exists. Avoids a write-race when N processes spawn simultaneously against the same bundle. Single-stream behavior unchanged.
9 CSVs in results/csv/ covering Qwen2.5-7B AC, BAT, AC NPU rerun (parser fix), conc=4 CPU/GPU on both models, and per-stream conc=N NPU streams.
Genie bundle scaffolding (models/qualcomm-qwen2_5-7b-ref/..., gitignored): hand-built genie_config.json for 6-partition Qwen2.5 (n-vocab 152064, ctx-bins listing all 6 .bin files), the 4B's htp_backend_ext_config.json copied as-is (same SoC v81/88), tokenizer.json from upstream HF.

Headline measurements.

Per-parameter scaling 4B → 7B (AC):

metric	4B	7B	Δ	takeaway
NPU partition count	4	6	+50%	spec-decode handoff cost grows
NPU bundle size	3.1 GB	4.7 GB	+52%	mostly w4a16 → w8a16
NPU PP	1566	1219	-22%	sublinear; better than W1.b's 700-900 projection
NPU TG	23.30	22.91	-1.7%	dispatch-bound, not weight-BW-bound
NPU J/gen-tok (BAT)	0.615	0.967	+57%	partition count is the cost driver
CPU PP	188	123	-35%	proportional to params
CPU TG	39.5	24.2	-39%	same
OpenCL TG	22.9	10.7	-53%	catastrophic

Concurrency=4 (agentic), AC:

model	CPU agg TG	OpenCL agg TG	scaling vs N=1
4B	82.0 t/s	15.7 (worse than N=1)	CPU 2.08×, OCL 0.68×
7B	62.8 t/s	13.2	CPU 2.60×, OCL 1.23×

NPU concurrency (Qwen3-4B via ORT-QNN spawn-N-procs):

N	per-stream TG	aggregate TG	scaling	status
1	25.78	25.78	1.00×	baseline
2	~14.76	~29.5	1.14×	works
3	~10.45	~31.4	1.22×	works (plateau)
4	—	—	—	unstable — QNN error 1003

The major non-obvious findings.

NPU TG is dispatch-bound, not bandwidth-bound. 4B and 7B measure within 1.7% of each other on AR=1 decode (23.3 vs 22.9 t/s). Per-token cost grows with partition count, not with weight bytes. Implication: rolling-our-own runtime's KV-stitch overhead is the budget, not the matmul throughput.
NPU PP scaling is gentler than the roadmap predicted. 1219 t/s @ 7B + sublinear scaling means 8B should land near 1000+ t/s, vs the W1.b projection of 700-900. The W1.b investment remains the obvious play.
NPU is single-tenant or low-tenant on this stack. Aggregate decode plateaus at ~31 t/s regardless of concurrent stream count, and 4 simultaneous ORT-QNN context groups (4 streams × 4 partitions = 16 contexts) hit a QNN HTP backend resource ceiling. At concurrency=4, CPU's 82 t/s aggregate beats NPU's plateau by 2.6× AND is stable. For agentic workloads at N≥4, CPU owns the compute.
KleidiAI flipped from regression at 4B to small win at 7B under single-stream AC (+5% TG). On battery and under concurrency it goes back to a wash or slight loss vs plain CPU. Pick the build per power state and concurrency mode, not by silicon.
OpenCL is dead at 7B+. TG halved 4B→7B; concurrency=4 is barely positive (1.23×) on a tiny absolute number (13 t/s). Mean power ↑31% to 58 W. Don't include OpenCL in any 7B+ deployment.

Decision gate updates.

W1.b (NPU prefill of 8B target) stays high priority. Confirmed by sublinear PP scaling — actual 8B PP should beat the roadmap's lower-bound projection.
W4 (heterogeneous sidecar / async orchestration) just got a clearer reason-to-exist. The NPU-concurrency cliff at N=4 means any "serve multiple agents from NPU" use case requires an in-process multi-context runtime; spawn-N-procs hits the QNN resource ceiling. The C++ sidecar with shared QnnContext + KV scheduling is the only path to N>3 NPU concurrency.
CPU promoted to the agentic-workload backend. 2.6× TG scaling at 7B / N=4 is the headline. Until W4 lands an NPU multi-context runtime, agentic deployments target CPU.
OpenCL retired from the W1.a candidate list at 7B+. The 4B's W1.a gate (>10× CPU prefill) was already not met; at 7B it's worse. Don't compile-target OpenCL for prefill on 7B+ models.
Llama-3.1-8B AI Hub run deferred. Requires Scenario A cloud Linux rental (~$2-4 one-time) per the rewritten docs/rent_cloud_compute.md. Not blocking — the 4B/7B scaling curve already lets us extrapolate the W1.b 8B answer with confidence. Pick this up when next addressing W4/W1.b directly.

Genie 4× async (item #1 of the user's two NPU-concurrency questions) — feasibility-only, not run. Spawning 4 genie-t2t-run.exe processes works mechanically but is strictly worse than the ORT-QNN multi-process path:

No inter-process weight sharing (weight_sharing_enabled only deduplicates within a process). Memory cost ~4× the bundle size.
No per-step timing visibility — Genie's CLI gives only aggregate PP/TG.
HTP context-switch cost between independent Genie processes vs the in-process multi-context model the W4 sidecar would use.

Skipped because the ORT-QNN spawn-4-procs run already shows the NPU's hard ceiling.

Next session — pick one:

Llama-3.1-8B baseline via Scenario A (cloud Linux rental, ~$2-4, ~3-5 hr): real 8B point on the same matrix, confirms (or adjusts) the W1.b extrapolation. Roadmap-aligned.
W4 sidecar scoping (no measurements; design + scope a C++ in-process multi-context QNN runtime that can serve N>3 streams from one Hexagon engine). The NPU concurrency cliff just turned this from "future" to "actually-blocking-NPU-deployment-ever".
Resume Phase 5.5 Lever C (paused per session 21). The 4B/7B matrix has already produced enough data to reprioritize, but Lever C's w4a16 PTQ work is still a real cost-of-quality investigation if we ever want to ship our own draft compiles.

current_status.md size note: this entry brings the file to ~1800 lines. Per docs/repo_hygiene.md, when this exceeds ~2000 lines or becomes hard to navigate, archive sessions ≤19 to docs/archive/current_status_archive_TBD.md and trim the head of this file. Not yet warranted.)

Last updated: 2026-04-23 (session 21 — Pivot to all-backends Qwen3-4B baseline matrix; repo cleanup + hygiene rules landed. Phase 5.5 Lever C is paused (see session 20 below — session ended with ARM-side prep for A.2/A.1 shipped and x86 compile ask pending; that pending work is not cancelled, just deprioritized until the baseline tells us where the compute bottleneck actually is).

Rationale for the pivot. The 5.5 investigation has been drilling into one island (Hexagon) at one model size (0.6B draft × 8B target) on one partial axis (w4a16 PTQ). The roadmap workstreams that come next (W1 GPU prefill, W2 NPU utilization, W4 heterogeneous async) all depend on a category × backend matrix we have never actually measured at a common model size. Qwen3-4B is the right common size: Qualcomm ships a blessed w4a16 Genie bundle for NPU, unsloth ships a Q4_K_M GGUF for CPU / GPU, weights are ~2.5 GB both sides, and the bundle has already loaded cleanly under ORT-QNN per the side-quest in results/reference/qwen3_4b_genie_w4a16_probe.md. One session of baseline measurement produces the data that reprioritizes W1/W2/W4.

This session landed (no measurements yet — infrastructure only):

Repo cleanup, ~200 GB staged for deletion. Moved intermediate ONNX export dirs (qwen3-0.6b-*, qwen3-4b-arm-*), negative-result and dominated binaries (w4a16-local{,-mse,-tfe}, w8a16-local-pr, fp16-local, old ctx512 artifacts), the calibration bundle dir (17 GB, regenerable from scripts + manifest), and the Phase 3 / 5 compile output bundles (phase3_qwen3_4b_compile 21 GB, phase5_qwen3_4b_bundle 28 GB) to marked_for_deletion/. models/ is down from ~280 GB to ~24 GB: four baseline Qwen3 GGUFs, qualcomm-qwen3-4b-ref/ (Genie bundle), plus two NPU exports kept as Lever B (pathbmask.bin — 18.12 t/s baseline) and Lever C (pathb.w8a16-local.bin — full-gate PTQ pass). marked_for_deletion/ is gitignored; rm -rf it after a soak if nothing comes up missing.
results/ reorganized. All 26 CSVs → results/csv/. Oracle .md + .npz pairs, probe summaries, npu_env_snapshot.txt, and the self-contained shotgun_drop/ bundle → results/reference/. All .log, .stdout, .stderr, per-run subdirs, and intermediate .json dumps → marked_for_deletion/results/ after confirming findings are captured in investigation docs or CSV rows.
docs/ reorganized. 12 closed-phase docs moved to docs/archive/ (all phase5_* subphase docs, SME_investigation, adreno_{debugging,opencl}, exporting_on_arm, npu_scoping, upstream_issue_body). Active top-level is down to 12 docs focused on the priority path. Markdown is never hard-deleted per the new hygiene rules.
docs/repo_hygiene.md written. Three-bucket rule (keep / archive / marked_for_deletion), per-directory policies, regeneratability test, when-to-tidy triggers, soak rules for staged deletions.
CLAUDE.md written (repo root, new) — session orientation pointing at current_status.md (detailed), docs/roadmap.md (big-picture), and docs/repo_hygiene.md. Plus current priority path and working rules.
README.md updated with a "For contributors / agents" section pointing at the three orientation docs.

Next session — run the baseline. Plan lives in docs/qwen3_4b_baseline_methods.md; results table in docs/qwen3_4b_baseline_all_backends.md. Measure Qwen3-4B PP512 + TG128 at ctx=2048 on AC across:

NPU (Genie) — genie-t2t-run against the Qualcomm bundle, w4a16. Primary tool.
CPU (ARM64 NEON) — llama-bench against Q4_K_M, -t 8 then scan {6, 8, 10, 12} if saturated.
CPU+KleidiAI — same, build-cpu-kleidiai binary (tests whether i8mm / DOT kernels help at 4B scale; on 0.6B they were a small win).
GPU (Adreno OpenCL) — llama-bench -ngl 99 against the Adreno-tuned OpenCL kernels. Expect a big PP number, likely middling TG.
GPU (Vulkan) — llama-bench -ngl 99. Cross-vendor future- proof path; our Adreno Vulkan driver has known correctness issues (docs/archive/adreno_debugging.md), so this row may stay empty or produce a "fast-but-wrong" warning row.

Optional secondary cell if NPU Genie refuses to load the 2.42-compiled bundle under QAIRT 2.45: the ORT-QNN chained 4-partition probe fallback per docs/qwen3_4b_baseline_methods.md §Fallback.

Decision gate after the matrix. The matrix tells us:

If NPU PP ≫ CPU PP and GPU PP, W1.b (8B target NPU prefill) moves up the priority list and W1.a (GPU prefill) becomes the cheap comparison point.
If GPU PP ≫ NPU PP, W1.a goes first and the 8B NPU prefill compile investment (W1.b) is gated on W4 (heterogeneous async) first justifying per-layer NPU/GPU hand-off.
If all three PPs are within 2× of each other, prefill is not the bottleneck — W2 (NPU utilization + tree drafts) takes priority over W1.
If NPU TG ≫ CPU TG, the Hexagon draft path is worth investing in for Qwen3.5 / 4B-class drafts even at w4a16 PTQ cost — Lever C's sub-gate result on 0.6B may not transfer up.

Only after the matrix + decision gate do we reopen w4a16 Lever C (session 20's A.2 / A.1 compile ask) or commit to W9.b cloud pipeline work. Baseline first; targeted investment second.)

Last updated: 2026-04-23 (session 20 — Phase 5.5 Lever C REOPENED via w4a16_investigation_continued.md; Phase 5.5.1 A.2 + A.1 in flight. Rationale: the Qualcomm Qwen3-4B side-quest (results/qwen3_4b_genie_w4a16_probe.md) measured 7.22 ms median for 12 Qwen3-4B w4a16 layers via ORT-QNN — projecting ~17 ms/step for our 0.6B if we match Qualcomm's uint8-past_kv + uint16-rest IO convention, vs our current 21-24 ms/step with uint16-past_kv. Two investigation-axis leads were never executed: A.2 (drop preserved-IO, pin past_kv to 8-bit via --quantization_overrides) and A.1 (A.2 + V-projection + O-projection weights pinned to w8 — targeting the session-17-localised V-collapse at w4 precision). This session shipped the full ARM-side prep: IS_LOCAL_FULL_QUANT_IO flag, quant_to_uint8 / dequant_from_uint8 helpers, bitwidth-aware quant_tensor / dequant_tensor dispatchers, past_kv_dtype / present_kv_dtype params on _describe_{inputs,outputs}_pathb_local, migrated four probes (npu_short_prompt_probe, probe_npu_steady_state_latency, probe_w4a16_quant_roundtrip, probe_w4a16_vs_fp16_differential) to the dispatcher. Committed 168- and 112-entry pre-authored --quantization_overrides JSONs at models/calibration/quant_overrides_{mixed,fqio}.json. x86 compile ask landed in docs/phase5_lever_c_x86_ask.md Update 3 with both recipes + NAS drop paths (phase5_step15_local_qairt_out_qairt242_{fqio,mixed}\). AST parse green across all five modified scripts; full-pipeline load gated on binary arrival. Existing variants (w4a16-local, w8a16-local, fp16-local, etc.) are explicit-whitelist-gated so they keep their uint16-everywhere schema unchanged. Decision tree on outcome in w4a16_investigation_continued.md §"Decision tree after measurement": A.2 beats 18.12 t/s → Lever C positive; A.1 clears cos 0.95 AND beats 18.12 → Lever C positive on mixed precision; both lose → memory- bandwidth thesis empirically disproven at 0.6B/8B ratio, pivot to Axis B (Qwen3-1.7B draft) or W1.a (GPU prefill 8B target). Commit 3f366db. See also the expanded roadmap (B9 EAGLE-3 ↔ w4a16 compounding, B20 custom multipath verifier, W4 heterogeneous 3-island pipeline with layer-wise KV streaming per exolabs). Prior status snapshot preserved below.)

Last updated: 2026-04-22 (session 19 — Phase 5.5 Lever C closes NEGATIVE as a product: w8a16-local AC sweep mean 12.83 t/s k=2 vs Lever B's 18.12 t/s (−29%), 71.65% vs 81.91% accept. Correctness pipeline is fully delivered (every stage works, local QAIRT compile bypasses AI Hub's preserve-list bug entirely) but PTQ noise on a 0.6B draft costs ~10 pp of accept rate, and per-step latency savings don't compensate. Lever B's 18.12 t/s fp16 pathbmask AC remains Phase 5.5's high-water mark.

Sessions 15-18 delivered the entire Lever C runtime stack via x86 local QAIRT (plan docs/phase5_local_qairt_compile.md, findings docs/phase5_local_qairt_compile_findings.md): pathb rotary-hoisted w4a16 binaries load cleanly on ORT-QNN 1.24.4, quant formula validated (RMS 0.001%), IS_LOCAL_COMPILE dispatcher pattern-matches any *-local variant, quant_specs threaded through sync + async outer loops + sweep. Session 17 differential probe localised the w4 PTQ collapse to layer-1+ V-projection weights (value tensor cos 0.957 at layer 0 → 0.130 at layer 1 → <0.2 all the way through layer 27; keys degrade gracefully via rotary smoothing). Session 18 shotgun (7 variants, 6 distinct MD5s): w8a16-local = first full gate pass (cos 0.963/0.979, argmax ✓, multi-step 100%); w4a16-local-pr soft pass (cos 0.888, 100% greedy match, 620 MB — 32% smaller binary); w4a16 mse/tfe/cle all confirmed negative (activation-cal not the lever; CLE is a no-op on MatMul graphs); w8a16-local-pr soft pass (per-row hurts at w8). Session 19 on AC: steady-state latency (scripts/probe_npu_steady_state_latency.py, 5 warmup + 25 measured per variant) shows all quantized variants cluster 21-24 ms/step (w4a16-local-pr fastest at 21.4 ms), fp16-local at ~50 ms — session-18's uniform "50 ms on battery" was cold-HTP + thermal noise. The 40-cell AC sweep on w8a16-local (async-pipelined, n_predict=200, 14.2 min) broke out as k=2 mean 12.83 / k=3 11.79 / k=4 10.12 / k=8 6.74 t/s. Best cell p2 k=2 = 14.39 t/s / 78.2% accept. Worst p6 k=8 = 4.73 / 26.9%.

Decision: ship Lever B's 18.12 t/s AC baseline as Phase 5.5's final number. Document Lever C as a structurally-working PTQ pipeline that didn't clear the throughput bar at 0.6B draft size — forward-compatible with Qwen3.5 graduation where the draft is larger (per-step costs grow, per-step savings become worth more vs fixed HTTP verify overhead) and where the same local-QAIRT toolchain drops in unchanged. w4a16-local-pr AC sweep result (18.1 min): k=2 mean 12.22 t/s, 54.51% accept — worse than w8a16-local (−17 pp accept, −5% t/s). The CPU-ref 100%-greedy-match DID NOT predict 8B-target greedy-match; top-5 overlap (3/5 vs w8a16's 4/5) was the more predictive signal. Lower accept → more rounds → HTTP-verify overhead swamps the per-step latency edge. Decision unchanged. Commits 48301d9 (quant_specs plumbing + steady-state probe), 435abf1 (session-18 shotgun probes), {pending} for session-19 sweep writeups.)

Last updated: 2026-04-22 (session 14 -- Phase 5.5 Lever C — pathb w4a16 compile SUCCEEDED but runtime blocked by an AI Hub compile driver bug. Rotary hoisting cleared the op-validation failure that killed j563xme75 (job jg93r1jqg reached SUCCESS in 100 min; .bin 876 MB at models/qwen3_0_6b_draft_v81_ctx256.pathb.w4a16-a.bin). But the ORT-QNN forward pass fails with "ORT Tensor data size does not match QNN tensor data size": AI Hub's driver mis-formats the --preserve_io_datatype list for qairt-quantizer, dropping exactly the first entry (past_key_values.0.key) — the converter gets 116 names, the quantizer gets 115. Layer-0 key is therefore uint8-quantized at the IO boundary while every other past_kv stays fp32, causing a 4× byte-count mismatch at runtime. Evidence: direct grep of both invocations in results/aihub-compile-jg93r1jqg-pathb-w4a16-a/jg93r1jqg.log. fp16 binaries are unaffected (no quantizer step invoked). Session-14 X2E plumbing is still sound and reusable: 61-input schema wired through compile_qwen3_ai_hub.py, prep_onnx_for_ai_hub.py, capture_calibration_samples.py (+ rope_tables(pos) with rope_theta=1e6), npu_load_qwen3_bin.py, probes + sweep (commit 1423f6c). Bundle A calibration captured (60 samples × 61 inputs, 3.27 GB). Lever B's 18.12 t/s AC baseline remains Phase 5.5's high- water mark. Next session picks a workaround from docs/qwen3_perf_levers_investigation.md §Lever C: prepend a sacrificial preserve-guard input, do ORT-side uint8 quant of just past_kv.0.key, or file a Qualcomm AI Hub bug ticket. See commit {pending} and the same doc for the full workaround matrix.)

Last updated: 2026-04-22 (session 13 -- x86 delivered Path B (rotary hoisted). models/qwen3-0.6b-pathb/: 61 inputs (was 59), 7,131 nodes, zero /model/rotary_emb/* nodes. CPU-equivalence probe vs optimum source: cos = 1.000000 on both pos=0 zero-KV and pos=5 synthetic-past_kv probes (numerically exact, not just within tolerance). Transferred to Z:\exposed\junk\phase5_step12_pathb\ with MD5 verified end-to-end. New scripts: rewrite_qwen3_pathb.py (pure protobuf rewrite) + probe_pathb_equivalence.py. X2E follow-up unchanged from session 12: extend compile_qwen3_ai_hub.py for the pathb 61-input schema, regenerate calibration, submit --quant w4a16. See status_x86.md session 2 for handoff details and the canonical runtime cos/sin formula.)

Last updated: 2026-04-22 (session 12 -- Phase 5.5 Lever C handed off to x86. Levers A + B closed on battery + AC (k=2 async-pipelined, ctx=256): AC baseline 18.12 t/s mean, 19.07 best, 81.91% accept (+127% over Phase 5 baseline 7.98 t/s). Lever C W4A16 compile attempted twice, both failed at AI Hub — root cause diagnosed by inspecting Qualcomm's shipping Qwen3-4B w4a16 bundle: our graph computes rotary_emb inline; Qualcomm hoists it out. Fix is a new x86-side export (Path B: rotary hoisted + additive mask) per docs/phase5_export_on_x86.md §"Path B implementation contract (2026-04-22 revision)". See also Phase 5.5 section below and Lever C detail in docs/qwen3_perf_levers_investigation.md.)

Last updated: 2026-04-21 (session 11 -- Phase 5 CLOSED. Full sweep landed: 40 cells (k ∈ {2,3,4,8} × 10 humaneval prompts, n_predict=256) in 25.9 min. k=2 wins with 7.98 t/s mean, 81.0% accept (best cell 8.44 t/s at p8). Structural regression vs Phase 2 CPU-spec 40.2 t/s, driven by NPU per-step latency; accept rate matches CPU-spec exactly. Writeup in docs/npu_results.md. w4a16 quantisation identified as biggest Phase 5.5 lever.)

Living document. Update every few turns. Anyone picking this up cold should be able to read this page, skim the README, and resume work.

Where we are in the phase plan

Phase 0 -- Infrastructure: in progress.

Note (2026-04-19): initial combined vulkan-opencl preset split into standalone vulkan and opencl presets so a missing OpenCL SDK doesn't block Vulkan work. Combined preset kept for when both are satisfied.

Note (2026-04-19, session 2): the vulkan build does not include llama-perplexity or llama-completion (our LLAMA_BUILD_TOOLS subset is narrower than default). This matters because the shipped llama-cli silently ignores -no-cnv on newer llama.cpp and always enters conversation mode — it prints "please use llama-completion instead" and falls back. For scripted correctness/perplexity assays we need to widen the build tool set (or drive llama-server over HTTP). Not blocking the OpenCL pivot; bundle with the next rebuild.

Note (2026-04-20, session 3): target list in build_llama_cpp.ps1 now includes llama-completion and llama-perplexity. They're in the build-opencl/bin/ output. Open caveat: llama-completion on HEAD fd6ae4c… also defaults to conversation mode and hangs waiting for interactive input after -n 64 tokens are generated — the generation itself is correct (we verified coherence), but scripted runs still need a Ctrl-C or a stdin close. For fully hands-off automation the safer tool is llama-server over HTTP.

Note (2026-04-19, session 2): Qualcomm's own GPU-compute path for llama.cpp is the OpenCL backend, not Vulkan. ggml-opencl has Adreno-specific optimizations landed by Qualcomm engineers (lhez, max-krasnyansky). Vulkan remains useful as a second GPU path, but OpenCL is the vendor-blessed one and should be unblocked soon.

Phase 1: CPU + OpenCL baselines landed in results/baseline-*.csv (sweep covers 0.6B / 1.7B / 8B at 8/12/18 threads, PP128/512 + TG64/128). Reference numbers the rest of this document assumes:

8B Q4_K_M CPU @ 18t: PP512 164 t/s, TG128 25.91 t/s, TG64 26.48 t/s
0.6B Q8_0 CPU @ 18t: TG128 149.66 t/s (draft ceiling)
8B Q4_K_M OpenCL: PP512 much stronger, TG lower than CPU -- CPU wins TG

Phase 2 -- Stock speculative decoding (session 4, 2026-04-20):

Six sweeps complete. Fixed rig throughout: target Qwen3-8B-Q4_K_M + draft Qwen3-0.6B-Q8_0, greedy (temp=0), --draft-min 0, -n 256, 10-prompt fixtures. Baselines from Phase 1:

8B CPU TG: 25.91 t/s -- the reference for every speedup below.
8B OpenCL TG: 13.50 t/s (PP is strong, TG is weak on Adreno).

Summary table (all humaneval, best k per config)

Config (k shown)	mean accept	mean decode t/s	vs CPU TG	vs own TG baseline
CPU spec, k=3 (winner)	79.6%	40.19	1.55×	1.55×
CPU spec, k=4	74.6%	37.51	1.45×	1.45×
CPU spec, k=2	82.3%	29.93	1.16×	1.16×
CPU spec, k=6	65.1%	32.32	1.25×	1.25×
CPU spec, k=8	58.4%	36.02	1.39×	1.39×
CPU spec, k=16	43.0%	30.43	1.17×	1.17×
CPU spec, k=32	27.5%	21.23	0.82×	0.82×
Mixed tgt=OpenCL dft=CPU, k=3	77.1%	9.52	0.37×	0.71×
Mixed tgt=OpenCL dft=CPU, k=8	--	14.37	0.55×	1.06×
Mixed tgt=OpenCL dft=CPU, k=16	--	16.14	0.62×	1.20×
OpenCL-all spec, k=3	77.7%	9.14	0.35×	0.68×
OpenCL-all spec, k=8	59.2%	13.19	0.51×	0.98×

Key findings

k=3 is optimal for draft-model spec on this hardware. Not k=4 as the coarse {4,8,16,32} sweep suggested. k=2 has highest accept (82%) but lowest decode (30 t/s) -- too little amortization per verify batch. k=3 hits 40.2 t/s mean (1.55×) with 79.6% accept.
Draft-model spec caps near ~1.6× on this rig regardless of workload. JSON accept is higher than humaneval (82.0% vs 79.6% at k=3) but decode barely moves (40.83 vs 40.19 t/s). Peak single run is JSON prompt-7 (git commits) at 44.88 t/s (1.73×). The ceiling is per-round overhead, not accept-limited. Phase-3+ techniques need to either drastically raise accept with the same k or cut the round-trip cost; higher accept alone will not break through 1.6×.
Mixed-device placement (tgt=OpenCL, dft=CPU) is a regression. Monotone improvement with k (9.5 → 14.4 → 16.1 for k ∈ {3, 8, 16}) pinpoints per-round CPU↔OpenCL sync as the bottleneck -- larger verify batches amortize it but never enough to beat CPU-alone TG (26 t/s), let alone CPU speculative (40 t/s). Peak mixed run was 26.6 t/s at k=16, only matching (not beating) CPU-alone. Same story for fully-on-OpenCL spec (9.1 t/s at k=3, 13.2 at k=8) -- converges to OpenCL-alone TG baseline, never wins.
Per-prompt variance is large. binary_search (p5) accepts at 55--91% across all k and gave the 1.88× peak (49.0 t/s at k=8). flatten (p6) is pathological: 58% accept at k=8 collapses to 13% at k=32, decode drops from 31.7 to 10.8 t/s (0.41×). A single pathological prompt can drag a whole sweep mean noticeably.
Encoded (prompt-eval) speed is invariant to k (~115 t/s across all CPU runs). PP is target-only and one-time; k only affects TG.
OpenCL per-call kernel-launch overhead is the killer. Adreno crushes large-batch PP (0.6B Q8_0 at 2674 t/s PP512) but the tiny 4-10-token verify batches in spec decode don't amortize kernel dispatch. This is the architectural lesson behind the negative mixed-device and OpenCL-all results.

Research implications (feeding Phases 3-5)

The 1.6× ceiling observed here is the draft-model spec ceiling on this hardware. The negative OpenCL results redirect the research plan:

DFlash+DDTree (Phase 4) is the primary lever. Confirmed by the lucebox-hub RTX 3090 paper (see new_spec_decode_example_to_research.md and docs/reference-projects.md): AL ≈ 8.9 and 3.43× with block- diffusion draft into tree verify. It attacks both binding axes -- K tokens drafted in one pass AND K-fat verify batches -- exactly matching the two constraints session 4 measured.
EAGLE-3 (Phase 3) becomes a viability probe, not an anchor. It touches only the accept-rate axis. Cheap to try because the PR exists, but unlikely to beat the overhead ceiling alone. Plan: build the PR on CPU + OpenCL, run one sweep, decision-gate on ≥2×.
NPU drafting (Phase 5) becomes more important. Hexagon and Adreno share LPDDR; an NPU-drafted block can be consumed by Adreno without the CPU↔GPU DMA round-trip that torpedoed mixed-device here. Also, NPU draft in parallel with GPU verify converts the per-round sync into pipelined overlap. The lucebox paper's top-3 perf wins included exactly that class of optimisation on PCIe (D2D copy, +3.3%).
CPU speculative is the working baseline to contribute upstream. 1.55× on code, 1.58× on JSON, stable, clean. llama.cpp's Adreno spec story is currently worse than CPU-alone; the data above is probably worth a docs/discussion contribution even before new techniques land.
Unified-memory / buffer-model optimisation is a latent lane. ggml-opencl uses plain clCreateBuffer(CL_MEM_READ_WRITE) with no zero-copy flags -- Snapdragon X2's shared LPDDR5X is available but not exploited. See docs/reference-projects.md ("Unified memory vs zero-copy") for the breakdown. Not blocking any phase yet, but may matter if we build a custom runtime (trident/lucebox-style) later.

Input artefacts

Prompt fixtures (prompts/): humaneval_subset.jsonl (10 code completions), structured_json.jsonl (10 JSON generations). prose_longform.jsonl and chat_multiturn.jsonl still TODO.

CSVs + per-run logs (all under results/):

spec-cpu-...-125354 : k ∈ {4,8,16,32} CPU humaneval
spec-cpu-...-131451 : k ∈ {2,3,4,6} CPU humaneval
spec-cpu-...-132358 : k ∈ {3,4} CPU JSON
spec-opencl-tgt-ocl-dft-cpu-...-132850 : k=3 mixed humaneval
spec-opencl-tgt-ocl-dft-cpu-...-133742 : k ∈ {8,16} mixed humaneval
spec-opencl-...-134935 : k ∈ {3,8} OpenCL-all humaneval

Phase 3 onward: not started.

CPU build validated

Command:

.\llama.cpp\build-cpu\bin\llama-cli.exe `
  -m .\models\Qwen3-0.6B-Q8_0.gguf `
  -p "The Snapdragon X2 Elite Extreme is" `
  -n 64 -t 18 -no-cnv

Result: coherent generation, PP 826 t/s / TG 111 t/s on Qwen3-0.6B Q8_0 at 18 threads. Consistent with gguf_models/LOCAL_LLM_NOTES.md scaling vs Qwen3-4B Q4_K_M (PP 248 / TG 42 at 18t).

Build recipe on this machine (captured in `scripts/build_llama_cpp.ps1`)

llama.cpp rejects MSVC for ARM64, so the build uses clang invoked from a vcvarsarm64 environment:

VS BuildTools 2022: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools
LLVM: C:\Program Files\LLVM (clang 22.1.3)
clang-rt: ...\lib\clang\22\lib\windows\clang_rt.builtins-aarch64.lib

Paths with spaces must be passed to cmake as 8.3-short, forward-slashed, unquoted (PS 5.1 argv parsing strips embedded quotes, so the reference .bat trick of inner \"..\" doesn't translate). See comments in the script. build_llama_cpp.ps1 -DryRun prints the fully resolved cmake invocation without executing.

Per-build metadata is recorded in llama.cpp\build-<preset>\SPECULA_BUILD.txt.

Outstanding issues / known gotchas

OpenCL build — SDK survey done, plan locked. See docs/adreno_opencl.md for the full writeup. Summary of what session 3 established:
- Runtime is already on disk. The Adreno driver (qcdx8480.inf_arm64_e11dd2e33e0b42d3 in the Windows driver store) ships both OpenCL.dll and OpenCL_adreno.dll. QAIRT bundles the same two DLLs under lib\aarch64-windows-msvc\. C:\Windows\System32\OpenCL.dll (the Khronos ICD loader) is also present.
- ICD registry key missing. HKLM\SOFTWARE\Khronos\OpenCL\Vendors does not exist. Without an entry under that key pointing at OpenCL_adreno.dll the loader sees zero platforms. Admin PowerShell one-liner in docs/adreno_opencl.md §Step 3.
- No QCOclIcd.dll on this Adreno gen. The earlier note that called for QCOclIcd.dll was based on older-gen Adreno naming; on this driver the ICD DLL is OpenCL_adreno.dll.
- Headers ship with QAIRT — just in a sample-app path. C:\Qualcomm\AIStack\QAIRT\2.45.40.260406\examples\QNN\SampleApp\SampleAppGPUFencing\src\CL\ contains cl.h, cl_ext.h, cl_ext_qcom.h, cl_platform.h, cl_version.h. Usable in a pinch; still no OpenCL.lib.
- SDK gap = import library only. Unblock routes ranked:
  1. vcpkg install opencl:arm64-windows (preferred — cleanest).
  2. Build Khronos OpenCL-Headers + OpenCL-ICD-Loader for ARM64 with the existing clang-via-vcvarsarm64 recipe, then pass -DOpenCL_INCLUDE_DIR=... -DOpenCL_LIBRARY=....
  3. Generate an import lib from OpenCL.dll (dumpbin /exports → .def → lib /def:) and consume QAIRT's sample-app headers. Only if (1) and (2) fail.
- Gotcha caught in source: ggml-opencl.cpp:222 get_adreno_gpu_gen() matches only A7X (730/740/750), A8X (830/840), and X1E (X1 substring). X2-90 falls to ADRENO_UNKNOWN. Non-fatal (init still proceeds) but may skip gen-specific tuning branches. Pre-emptive patch is a one-line strstr(name, "X2") add; leave it until we have a first correct run so we can see whether unknown-gen works out-of-the-box.
KleidiAI / SME2 crashed at runtime in prior project. See gguf_models/LOCAL_LLM_NOTES.md. Will retry as a tracked task in Phase 1; scripts/build_llama_cpp.ps1 -Preset cpu-kleidiai is wired up and applies the clang-on-Windows .S patch automatically.
Good sign for the SME2 retry: the vulkan configure pass showed HAVE_SME - Success for the compiler feature probe, meaning the toolchain thinks SME codegen works. The runtime-trap suspicion from LOCAL_LLM_NOTES.md (ZA-tile user-mode state not enabled) may still bite, but the build side is not the problem.

Directory layout snapshot

specula/
├── README.md                   # phase plan, hardware assumptions, rationale
├── current_status.md           # <-- this file
├── pyproject.toml, .python-version, .gitignore, .gitattributes
├── docs/
│   └── reference-projects.md   # trident / voice_project / gguf_models pointers
├── scripts/
│   ├── build_llama_cpp.ps1     # multi-preset native ARM64 builder
│   ├── patch_kleidiai.py       # clang-on-Windows .S patch for KleidiAI
│   ├── download_models.ps1     # HF GGUF fetcher (resumable)
│   ├── sweep_baseline.ps1      # Phase 1 autoregressive matrix
│   └── sweep_speculative.ps1   # Phase 2 spec-decode matrix
├── prompts/
│   └── humaneval_subset.jsonl  # 10 code-completion prompts (other workload files TODO)
├── models/                     # GGUFs (gitignored)
├── results/                    # CSVs + logs (empty)
├── notebooks/                  # analysis (empty)
└── llama.cpp/                  # sibling checkout, gitignored
    └── build-cpu/              # built; binaries in bin/, DLLs copied

Phase 5 status (session 9, 2026-04-21)

Step 4 CLOSED. models/qwen3_0_6b_draft_v81_ctx512.bin (1438 MB) is on disk. AI Hub job jgzx6xlz5 compiled cleanly on the first attempt off the x86-produced nomask ONNX: CREATED → OPTIMIZING_MODEL → SUCCESS at t=400s. This cleared the 440-465s CTX-BIN wall that killed the prior 8 attempts — the x86-side nomask variant (onnxsim + aggressive mask-subgraph removal, commit adbbbd4) eliminated attention_mask, Where, and IsNaN entirely, so HTP had no BOOL tensors to reject. See qlcom_compile_status.md for the full retro.

Current local state:

models/qwen3-0.6b-nomask/ — x86 handoff input (3 GB onnx + data)
models/qwen3-0.6b-nomask-ai-hub/ — staged for upload (2.87 GB)
models/qwen3_0_6b_draft_v81_ctx512.bin — compiled HTP context binary, signed via AI Hub's QAIRT stack. Target for step 5.

Steps 5-10 (per docs/npu_scoping.md §7) still ahead:

[DONE]    1. Environment snapshot                 (commit 7230210)
[DONE]    2. ORT-QNN sidecar skeleton             (commit 282e84a)
[DONE]    3. Qwen3-0.6B ONNX sourced + CPU-valid  (commit 106c756)
[DONE]    4. AI Hub compile -> Hexagon .bin       (session 9 ★)
          5. Load .bin via NPUSession, shape-check <-- next
          6. Correctness vs CPU, single greedy prompt
          7. Pipe first drafted token through llama.cpp verify
          8. External-drafter bridge for llama.cpp spec decode
          9. First NPU-spec number on 10-prompt humaneval
         10. Sweep k values, write up, close phase

Step 5 risk: NPUSession currently loads .onnx files. Context binaries need a different ORT-QNN load path (either via qnn_context_binary_file provider option or via an ONNX-EPContext wrapper that AI Hub emits alongside). Small extension expected.

Step 5 CLOSED (2026-04-21). First forward pass running on the Hexagon NPU. Step 6 diagnosis (session 10, 2026-04-21) REOPENED STEP 4 — the binary runs cleanly but produces catastrophically wrong logits. See "Step 6 diagnosis" section below.

session providers   : ['QNNExecutionProvider', 'CPUExecutionProvider']
inputs (58) / outputs (57) — match the binary signature exactly
run latency       : 109.81 ms   (decode step, empty KV, ctx 512)
logits shape      : (1, 1, 151936)
logits finite frac: 1.0000      (no NaN/Inf)
logits min/max    : -4.500 / 3.059
=== STATUS: ok ===

Per scoping doc §7 step 5 exit criterion ("one forward pass completes without error") — done. 110 ms/step is in-line with NPU expectations (slower than CPU's ~9 ms/tok for Qwen3-0.6B Q8_0; the value lands at step 7 when drafts pipeline alongside CPU verify). Commit <TBD>.

Two walls hit + cleared on the way:

ORT-QNN ↔ QAIRT version mismatch. AI Hub default was QAIRT 2.45; onnxruntime-qnn 1.24.4 bundles 2.42 → LoadCachedQnnContextFromBuffer error 5000. Bumping to onnxruntime-qnn 2.1.0 (bundles 2.45.40, ships Genie.dll) cleared the version match but its context-binary loader has unrecoverable bugs on the X2E94100 driver (both file-mapping retry path AND embed_mode=1 path segfault with no Python traceback — only the plain-ONNX path works in 2.x). Working fix: stay on 1.24.4 + recompile via AI Hub with --qairt_version 2.42. Recompile (job jp34dq03g, also 400s) reused the upload from mng5oj90m. Full writeup in docs/npu_ort_qnn_version_match.md, cross-linked from docs/npu_scoping.md §3.8.
EPContext wrapper IO names + dtypes had to match the compiled binary. QAIRT's converter normalises dotted names to underscored (past_key_values_0_key, not past_key_values.0.key) and renames all outputs to output_0..output_N in declaration order. Also, --preserve_io_datatype keeps past_key_values at FP32 even when --quantize_full_type float16 is set for the graph interior. Real names + dtypes captured from qnn-context-binary-utility.exe inspection (results/bin_inspect.json); wrapper builder in scripts/npu_load_qwen3_bin.py updated to match.

Updated 10-step tracker (after session 11 step-7 close):

[DONE]    1. Environment snapshot                 (commit 7230210)
[DONE]    2. ORT-QNN sidecar skeleton             (commit 282e84a)
[DONE]    3. Qwen3-0.6B ONNX sourced + CPU-valid  (commit 106c756)
[DONE]    4. AI Hub compile -> Hexagon .bin       (session 11 jperqy07g,
                                                   patha binary 1.4 GB)
[DONE]    5. Load .bin via NPUSession, shape-check (session 9, re-verified
                                                   session 11 on Path A wrapper)
[DONE]    6. Correctness vs CPU, single greedy    (session 11 Path A:
                                                   cos=0.9999, 100% match)
[DONE]    7. Pipe first drafted token through     (session 11 step-7
              llama.cpp verify                     plumbing script passed;
                                                   draft=target=264 at anchor)
[DONE]    8. External-drafter bridge for          (session 11: short-prompt
              llama.cpp spec decode                probe + outer loop;
                                                   6.23 t/s, 65% accept on
                                                   humaneval p0, coherent text)
[DONE]    9. First NPU-spec number on 10-prompt   (session 11: 40-cell sweep,
              humaneval                            k=2 optimal at 7.98 t/s
                                                   mean, 81.0% accept; best
                                                   cell 8.44 t/s at p8)
[DONE]   10. Sweep k values, write up, close      (session 11: docs/npu_results.md
              phase                                — documented loss, 0.31× of
                                                   CPU-alone TG, w4a16 lever
                                                   flagged for Phase 5.5)

Phase 5.5 status (session 12, 2026-04-22)

Full detail in docs/qwen3_perf_levers_investigation.md — this is the project-wide summary.

Closed levers:

lever	commit	k=2 mean t/s	vs baseline	notes
Phase 5 baseline	7e10670	7.98	—	ctx=512 fp16 sync
Lever A (async draft∥verify)	64de69f	10.93	+37%	pipelined verify-ahead landed in 56b375b
Lever B (ctx=256) × A	f755d6d	14.28	+79%	battery + CAD load; best cell 17.78
Lever B AC rerun (this session)	90594d9 CSV	18.12	+127%	clean AC, other programs closed; new reference baseline
R4 (zero-copy / shared-mem)	557c59e	—	no win	parked; per-step dominated by compute not copy

Battery→AC delta on identical binary is +26.9%, larger than any single lever's gain — future comparisons must be AC-vs-AC.

Lever C — W4A16 quantization — in flight, x86 handoff.

Two AI Hub compile attempts this session, both failed:

jp4x74ll5 (FAILED, ~120s): AI Hub's PTQ validator rejects calibration_data dicts whose key order doesn't match ONNX graph.input order. Fixed in commit 372e17a (compile script now iterates specs to rebuild DatasetEntries; capture script puts attention_bias AFTER past_kv to match graph order).
j563xme75 (FAILED, 6010s): pipeline got deep — ONNX→DLC ✓, quantizer ✓, quantized DLC saved ✓, then QNN backend op-validation rejected /model/rotary_emb/MatMul with "has incorrect Value 0, expected equal to -32768" (INT16_MIN, the offset QNN's backend hard-codes for rotary outputs). Full AI Hub log archived at results/aihub-compile-log-j563xme75-w4a16-a-FAILED.log.

Root cause confirmed by inspecting Qualcomm's shipping Qwen3-4B w4a16 Genie bundle (models/qualcomm-qwen3-4b-ref/.../metadata.yaml): their graph does not contain rotary_emb internally. position_ids_cos and position_ids_sin are declared as top-level graph inputs (shape [1,1,N,head_dim/2], dtype uint16, offset -32768 — exactly the value the AI Hub error expected). Same QAIRT 2.42, same X2 Elite target. The conclusion: for w4a16 compile to succeed, our export must hoist rotary out, matching Qualcomm's recipe.

Infrastructure landed this session (commits 90594d9, 372e17a, 11fe8fa):

scripts/capture_calibration_samples.py — CPU FP32 prefill + greedy decode on humaneval + structured_json fixtures, snapshots model inputs at selected decode positions into stacked-per-input .npz. Reusable for Qwen3.5 cutover.
scripts/compile_qwen3_ai_hub.py extended: --quant {float16,w4a16,w8a16}, --calibration-npz, --calibration-dataset-id, --quant-tag. fp16 path backward-compatible.
SPECULA_NPU_VARIANT env var wired through npu_load_qwen3_bin.py + npu_vs_cpu_correctness.py so probe/outer_loop/sweep target variant binaries transparently. Mirrors SPECULA_NPU_CTX's pattern.
Calibration bundles (models/calibration/, gitignored): Bundle A (60 realistic samples, 3.27 GB) + Bundle B (20 step-0 samples, 1.09 GB). Both at ctx=256 for the pathbmask schema; both need regeneration once pathb lands.

x86 team work — DELIVERED (session 13). Artifact: models/qwen3-0.6b-pathb/ (61 inputs, 7,131 nodes, zero /model/rotary_emb/* nodes). CPU-equivalence cos = 1.000000 on both probes vs optimum source. Shipped 3D shape [batch_size, sequence_length, 128] for cos/sin (doc said 4D [1,1,1,128] but that was for a different seam — see status_x86.md session 2 for the seam choice). Bundle on NAS at Z:\exposed\junk\phase5_step12_pathb\qwen3-0.6b-pathb\ with MD5 verified. CPU-equivalence probe gate: cos ≥ 0.9999 vs optimum source. ~0.5 session estimate.

Next — X2E team work (after pathb arrives):

Add pathb to build_paths + build_input_specs in compile_qwen3_ai_hub.py (61 inputs, includes cos/sin).
Regenerate Bundle A + B calibration for pathb schema (compute cos/sin per sample using Qwen3's rope_theta=1e6).
Submit --quant w4a16 --calibration-npz bundle_a_pathb_ctx256.npz — expected to succeed this time based on Qualcomm-reference alignment.
Wire cos/sin computation into the runtime caller (probe, outer_loop, sweep).
Correctness probe (cos ≥ 0.95 tolerated post-w4a16) + AC sweep vs 18.12 t/s baseline. Optionally run Bundle B for the cheap-vs-realistic calibration A/B.
Phase 5.5 writeup + close.

Step 6 diagnosis (session 10, 2026-04-21)

Summary: the compiled .bin loads + runs, but the nomask ONNX it was compiled from is computationally broken. The NPU is faithfully reproducing a corrupted graph.

Harness: scripts/npu_vs_cpu_correctness.py — drives CPU prefill on the optimum ONNX (standard-ops, FP32 KV) until past_len=511, then compares one more decode step on both backends with identical past_kv + input_ids + position_ids. Also runs a 16-step sliding-window greedy comparison.

Single-step result (prefilled KV):

cosine sim: 0.546 (expected > 0.99)
argmax: CPU=264 (' a') vs NPU=133927 (Arabic glyph)
top-5 overlap: 0/5
max |logit delta|: 23.97

Zero-KV + BOS control probe (isolates graph vs KV-handoff):

cosine sim: -0.183 (anti-correlated)
max logit magnitude: CPU=+14.09, NPU=+4.80

Zero-KV failing rules out KV-handoff semantics — the NPU graph itself is wrong. Localized bug:

Root cause: models/qwen3-0.6b-nomask/model.onnx run on CPU-ORT gives cos = -0.18 vs its optimum source. All earlier intermediate artifacts (optimum, optimum-frozen, optimum-frozen-ortopt, optimum-ortopt, patched) produce cos = +1.0000 with the source. Only nomask is broken.
Bisected the two simplify_qwen3_no_mask.py transforms against the clean patched graph:
- Mask-promote-to-constant alone: cos = +1.0000 (safe)
- IsNaN/Where guard elision alone: cos = +1.0000 (safe)
- Mask-promote + onnxsim(with shape overrides): cos = -0.18 (BROKEN)
Breakage comes from onnxsim.simplify() folding with overwrite_input_shapes pinning + attention_mask pre-promoted to constant [1,512] all-ones. Something in that combination constant-folds a position-dependent subgraph incorrectly. Both transforms are individually safe; their combination with onnxsim is not.
Verified fix direction: skip onnxsim. Starting from patched (2185 nodes) and applying mask-promote + isnan-elide only produces a graph with cos = +1.0000 vs source. But 2 BOOL Cast nodes remain in the attention_mask subgraph (HTP will reject), and both trace to ops whose inputs are now known constants. They need a targeted surgical constant-fold of just that subgraph (not a whole-graph onnxsim pass). See the updated docs/phase5_export_on_x86.md for the recommended x86-side fix.

Qualcomm Qwen3-4B NPU reference (inspected session 10)

Downloaded qualcomm/Qwen3-4B Genie w4a16 bundle (~3 GB zipped). Local copy at models/qualcomm-qwen3-4b-ref/qwen3_4b-genie-w4a16-qualcomm_snapdragon_x2_elite/. Inspected via metadata.yaml + genie_config.json. Their architecture choices are substantially different from ours:

4-part binary split per variant (embed + 3 transformer chunks). Each part compiles as its own QNN context binary. This almost certainly keeps each individual AI Hub compile under the op-lowering complexity budget that bit us repeatedly at 4a-4f.
40 variants in one bundle: 5 context tiers (512 / 1024 / 2048 / 3072 / 4096) × 2 AR batch sizes (128 prefill, 1 decode) × 4 parts. Weight-sharing across all of them.
RoPE externalized. position_ids_cos and position_ids_sin are INPUT tensors of shape [1, 1, seq_len, 64], uint16 quantized, pre-computed on CPU. Graph contains zero Cos/Sin/Range ops. This eliminates a whole class of HTP lowering issues.
Attention mask is runtime input, additive, uint16. Shape [1, 1, seq_q, seq_k], quant scale 0.00153 / offset -65535. Graph adds this to attention scores pre-softmax. Zero BOOL tensors anywhere. This is the mechanism for expressing the causal pattern without any of the Range/Gather/Cast/And/Where subgraph that onnxsim is supposed to fold for us.
Full w4a16 quantization. Activations and KV at the IO boundary are uint8/uint16 quantized, not FP16 or FP32. Needs AIMET (or AI Hub's quant path) and ~50-100 calibration prompts.
Transposed key layout. past_key is [heads, batch, head_dim, seq] (head_dim BEFORE seq); past_value is [heads, batch, seq, head_dim]. A Qualcomm-specific layout that the graph surgery must emit.
Tool versions: QAIRT 2.42.0 (matches our pin for ORT-QNN 1.24.4 compatibility).

Relevance to our fix: we cannot replicate the full Qualcomm pipeline (w4a16 + 4-way split + 5 ctx tiers + 2 batch modes is out of scope for a quick fix). But the two load-bearing architectural choices — externalized RoPE and additive-FP16 attention mask — are the principled fixes that make the graph naturally HTP-friendly without needing constant-folding tricks. Our current path (optimum + onnxsim) takes a graph that contains these problem subgraphs and tries to fold them out; the Qualcomm path never introduces them in the first place.

The docs/phase5_export_on_x86.md doc now recommends two paths for the x86 re-export:

Path A (minimal): keep current pipeline (optimum + --no-post-process), drop onnxsim entirely, apply mask-promote + isnan-elide + targeted surgical fold of the residual attention_mask subgraph. Preserves cos = 1.0 and emits zero BOOL Casts. Smallest change; may still hit compile-time op-lowering issues on monolithic graph complexity.
Path B (Qualcomm-style): externalize RoPE + use additive-FP16 attention mask. Larger surgery but the architecturally robust path. Separate from w4a16 quantization (which can layer on top later for the perf win).

Path A is the recommended starting point since it's a smaller delta from what's known to produce a compileable graph (we already got one compile through on nomask, just semantically wrong).

Session 11 (2026-04-21): first compile cycle on patha + pathbmask

X86 team delivered two CPU-ORT-verified ONNX variants (both cos=1.0 vs optimum source, zero BOOL casts on both, zero BOOL tensors on pathbmask). Staging + AI Hub compile scripts parameterized by --path {patha,pathbmask}.

Both first-cycle compiles failed; both for the same root cause — dynamic shapes in the uploaded ONNX. Full retro in docs/phase5_step6_compile_retro.md. Load-bearing findings:

AI Hub's OverrideFoldConstantsPass folds BOOL subgraphs cleanly when the shapes around them are concrete. Path A's compile log shows Cast 348→1, ConstantOfShape 60→0, Equal 58→0, Where 86→0, Range 3→0 in a single pass. The x86 team's "BOOL rejection" hypothesis was wrong — surgical BOOL removal was nice-to-have, not required.
AI Hub's compiler has a pass that doesn't handle SymbolicDim (_op_identity.py:30 calls np.broadcast_shapes on dims). Our ONNX declares input_ids: (-1, -1), past_kv: (-1, 8, -1, 128), etc. — the rewriter blows up on the first symbolic dim it sees after the fold pass.
Session 9's nomask compile only worked because onnxsim had pinned shapes statically. The x86 team replaced onnxsim (correctly — it corrupted numerics) with pure protobuf surgical folds, but those preserved dynamic shapes.

Fix: a 30-LOC pure protobuf edit in prep_onnx_for_ai_hub.py that pins every graph input's TensorShapeProto dims to the static values from compile_qwen3_ai_hub.build_input_specs. Idempotent, no numerical impact (session 10 bisection established that protobuf-only edits preserve cos=1.0).

Jobs (both artifacts):

path	upload id	job id	status	failure kind
patha	mnzpwoe6q	jp83n13kg	FAILED	SymbolicDim in rewriter
pathbmask	mqe19804m	jp01w619g	FAILED	pre-compile shape mismatch

Local compile logs: results/phase5_step6_compile_{patha,pathbmask}.log. Workbench logs: results/ai_hub_logs/{jp83n13kg,jp01w619g}.log/.

Next: resubmit both serially with the pinned ONNX. Serial avoids halving upload bandwidth on this link (~25 min/path single-stream vs ~50 min parallel). Predicted outcome: both compile (the fold pass already handled the BOOL region for Path A; pinning resolves the SymbolicDim bug for both).

Session 11 cont. — what actually unblocked the compile

Predicted wrong. Shape pinning + dim_param resolution was not sufficient — iterations v2, v3, v4 produced byte-for-byte identical op histograms on the failing compile side. AI Hub's pipeline discards or re-derives our provided value_info and re-triggers the same SymbolicDim crash in its identity-op rewriter.

The load-bearing fix was ORT's ORT_ENABLE_BASIC graph-optimization pass applied locally before upload. Sequence that worked (v5):

pin_input_shapes — graph.input dims concrete
resolve_dim_params — substitute batch_size/sequence_length/ past_sequence_length + sequence_length with concrete ints
ORT-BASIC constant-fold (node count 7580 → 2061)
resolve any unk__N placeholders ORT generated → 1
final onnx.shape_inference(data_prop=True) — 0 tensors with symbolic dims

Why ORT-BASIC wins where AI Hub's own fold pass didn't: ORT actually replaces Range/Shape/ConstantOfShape/Gather nodes with Constant initializers and eliminates them from the graph. AI Hub's OverrideFoldConstantsPass also folds them but keeps residual tensor shape annotations marked SymbolicDim from its own shape inference, which the downstream rewriter then chokes on. By pre-folding with ORT, the symbolic-shape annotations never enter AI Hub's pipeline in the first place.

Why not use ORT ENABLE_EXTENDED: that level introduces operator fusions (GELU/LayerNorm/Attention) that emit com.microsoft ops — the exact ones we spent sessions 7-9 removing. BASIC is strictly constant-folding + redundant-node elimination, numerically equivalent to source per session 10's optimum-ortopt cos=1.0 probe.

Jobs in session 11:

path	job id	result
patha v1	jp83n13kg	FAILED (SymbolicDim in rewriter)
pathbmask v1	jp01w619g	FAILED (pre-compile shape mismatch)
patha v2	jgj09qwep	FAILED (same)
patha v3	jg93rddmg	FAILED (same)
patha v4	jpr4rw7vg	FAILED (same)
patha v5	jperqy07g	SUCCESS @ 430s, 1.4 GB binary
pathbmask v5	j563xwkv5	FAILED (input-spec order mismatch)
pathbmask v6	jpx7q4o9g	SUCCESS @ 400s, 1.4 GB binary

Correctness on both v5/v6 binaries (npu_vs_cpu_correctness.py --path {patha,pathbmask}):

Single-step prefilled KV: cos = 0.999916 (gate ≥ 0.95) — both paths
16-step sliding-window greedy: 100% match rate (gate ≥ 50%) — both paths
NPU text matches CPU verbatim: " a 5G network. It is a smartphone with a smartphone"
Path A and Path B-mask produce byte-identical NPU output (expected — same 2061-node graph after ORT fold)
Zero-KV + BOS probe: cos 0.94 (edge; non-BOS zero-KV = cos 1.0000)

Step 6 CLOSED on both paths. x86's 2×2 hypothesis matrix (Path A: "BOOL casts removed, tensors remain" vs Path B-mask: "zero BOOL tensors") is answered: HTP accepts either. The load-bearing issue was graph complexity (dynamic shapes + unfolded Range/Shape/Expand chains), not BOOL op types.

Operational recommendation: Path A is the primary NPU binary (one less runtime feed — no attention_bias zeros tensor per step). Path B-mask stays in the repo as the documented equivalent path, useful if a future regime needs a non-zero additive bias (partial-window prefill with padding).

Step 7 (llama.cpp verify wiring) now unblocked.

Session 11 cont. — step 7 plumbing checkpoint PASSED

scripts/npu_spec_step7_plumbing.py drives the three-way comparison the scoping doc §7 step 7 calls for: NPU draft vs CPU reference vs llama-server target, all at the step-6 validated anchor position.

Design choice: pass raw token ids (not detokenized text) to llama-server's /completion endpoint. llama.cpp/tools/server/ server-common.cpp:767 accepts "prompt" as a JSON array of ints via json_is_array_of_numbers(json_prompt), so both sides compare purely at the id level with zero risk of detok->retok divergence between HF's Qwen3 BPE (what the NPU draft uses) and the llama.cpp GGUF vocab (what the target uses). A lightweight sanity probe up front confirms the two tokenizers agree on ids for a sample string — they match byte- for-byte on the 11-id Fibonacci probe.

Run outcome (results/phase5_step7_plumbing.log):

tokenizer probe      : 11/11 ids match between HF 0.6B and server 8B
NPU draft (Path A)   : token 264 (' a')
CPU reference (0.6B) : token 264 (' a')   [step-6 anchor, known good]
Target (8B, CPU)     : token 264 (' a')
draft == target      : True   (accept)
draft == CPU 0.6B    : True   (sanity)

All three backends converge on the same next-token id at the 511-token anchor. The plumbing exit criterion — one drafted token returned, one target token returned, one accept/reject decision logged — is met. The accept=True here is a bonus: it demonstrates the small draft genuinely predicting what the large target would, not a coincidence of anchoring.

Invocation recipe (captured in the script):

llama-server.exe
  -m models/Qwen3-8B-Q4_K_M.gguf
  --host 127.0.0.1 --port 8088
  -c 576                 # CONTEXT_MAX (512) + n_predict slack
  -t 18                  # match Phase 2 CPU-spec baseline
  --no-warmup

with /completion body {"prompt": [512 ids], "n_predict": 1, "temperature": 0.0, "top_k": 1, "seed": 1, "cache_prompt": false, "return_tokens": true}. Target call latency: 4.60 s for PP=512 + 1 generated token (about 111 t/s prompt-eval, matching Phase 1's 8B Q4_K_M CPU PP512 of 164 t/s once server overhead is accounted for).

Caveat carried into step 8: the script anchors at past_len=511 so the NPU never has to mask out invalid KV positions. The production spec-decode outer loop wants short prompts (20-50 tokens, drafting 3+ tokens per round), which needs Path B-mask's non-zero attention_bias with -65504 for invalid slots. That masking pattern is numerically unvalidated on the NPU today; step 8's outer loop has to prove it out before the first end-to-end run.

Step 8 (external-drafter outer loop) now unblocked.

Session 11 cont. — step 8 PASSED end-to-end

Two scripts landed, both green:

scripts/npu_short_prompt_probe.py — short-prompt NPU probe gate. Encodes humaneval p0 (16 tokens), CPU-prefills, then NPU single-step at position 16 with slots 16..510 zero-padded and attention_bias set to -65504 over padded slots + 0 over valid slots + 0 over the self-slot. Also validates the multi-step KV rearrangement primitive (3 consecutive NPU steps, each growing valid_past_len by 1, moving the K/V from slot 511 to slot P after each step). Both gates passed byte-clean: single-step cos = 0.999960, argmax match, top-5 5/5; multi-step 100% match (3/3 tokens identical to CPU greedy), NPU text " if n".

scripts/npu_spec_outer_loop.py — first NPU-drafted spec decode end-to-end. Sidecar-as-driver; per round:

Draft k tokens on NPU via short-prompt mask + slot-511→slot-P rearrangement between steps. Keep k+1 past snapshots so any accept count j ∈ [0, k] can roll back cleanly.
POST committed ids to /completion with n_predict=k+1, cache_prompt=true, greedy pinned. Read k+1 target tokens.
Longest-common-prefix accept: j matching drafts + 1 bonus target token committed.
Absorb bonus into past via one more NPU step → next round's state + first candidate for next round.

Run on humaneval p0, k=3, n_predict=64:

rounds              : 22
decoded tokens      : 65
mean accept rate    : 65.2%   (43/66 drafts accepted)
wall generate       : 10.43 s
  NPU draft total   :  5.22 s
  target verify     :  3.53 s
  NPU absorb        :  1.67 s
decode rate         : 6.23 t/s

Generated text (the NPU-drafted, target-verified continuation of the Fibonacci stub):

    # Initialize a memoization dictionary
    memo = {0: 0, 1: 1}

    # Define a helper function to compute the Fibonacci number
    # recursively with memoization
    def fib(n):
        if n not in memo:
            memo[n] = fib(n-1) + fib

Functionally correct memoized Fibonacci. Proves draft + verify + KV rearrangement are all semantically sound; the question now is purely performance.

Why 6.23 t/s (not faster than 25.91 t/s CPU-alone, let alone Phase 2's 40.2 t/s CPU-spec)?

Bottleneck is NPU per-call latency:

110 NPU calls / 22 rounds = 5 calls/round (3 drafts + 1 final- snapshot step that only pays off when j == k + 1 absorb-bonus).
At ~63 ms/call that's 6.9 s of the 10.4 s wall budget.
Target verify is 3.53 s = 160 ms/round. At CPU target TG ~26 t/s and k+1=4 tokens per round, that's ~155 ms of decode + negligible HTTP. Cache-prompt keeps prefill cost off the critical path after round 1.
Draft phase of 5.22 s + absorb of 1.67 s = 6.89 s strictly sequential with the target verify's 3.53 s. 10.4 s total.

CPU-spec baseline used ~25 ms/token-generated with 0.6B at ~9 ms/draft-step on CPU. NPU's 63 ms/step is ~7× slower per step. Without overlap or a fatter draft tree, there's no way to recover the cost of running draft + target sequentially when draft alone costs more than target's per-token decode.

Step 9/10 levers (in priority order):

Drop the final-snapshot step when j < k. Saves 1 NPU call (22 × 63 ms = 1.4 s in this run). Easy lazy-compute refactor — only materialise past_snapshots[k] if j==k actually happens. Would push us toward ~7.2 t/s, still below 25.91 CPU-alone TG.
Overlap NPU draft with target verify. Kick off /completion for round N's verify and NPU draft for round N+1 in parallel. Caveat: round N+1's draft depends on round N's accepted tokens, so this only overlaps the target-side portion of round N with round N+1's BEFORE-accept drafts (which we'd speculatively compute assuming drafts[0] accepted). Saves up to 3.5 s on the verify side. Complex but can double throughput if it works.
Pipelined k+K drafting. Draft k tokens, but also opportunistically start the next-round's first draft step during verify. Structural.
Reduce NPU per-step latency. Would need ORT-QNN EP-side work (cl_qcom_ion_host_ptr-style zero-copy KV, per-call bind caching) or a recompiled binary with a smaller compiled context. Deep dive, Phase 5.5 or 6 territory.

Step 9 (sweep k + multi-prompt) will produce the real CSV for the writeup. Lever (1) is easy and should land before step 9 so the numbers we report don't carry an obvious waste. Levers (2-4) are explicitly out-of-scope for Phase 5 close; noted for Phase 6.

Immediate next steps (next session)

Current live plan: see the session 21 block at the top of this file. This section below is the Phase 5 close-out planning from session 11 and is preserved as historical record — some items (W4A16 lever, async overlap) have since been attempted and resolved, and the priority has shifted to the Qwen3-4B all-backends baseline matrix.

Phase 5 is CLOSED. 40-cell sweep banked, writeup in docs/npu_results.md. Headline: k=2, 7.98 t/s mean (8.44 best), 81.0% accept — a 5× structural regression vs Phase 2 CPU-spec 40.2 t/s, with accept rate identical to CPU-spec (81.0% NPU vs 82.3% CPU at k=2). NPU per-step latency is the root cause, not drafter quality.

Two branches open next. Both can be pursued independently; the Qwen3 close-out items have priority so Qwen3.5 graduation (Phase 4) is unblocked.

(A) Phase 5.5 — NPU performance levers (if we want a better NPU-spec number before graduation). Ranked by impact × effort:

W4A16 quantisation — biggest per-step lever. Qualcomm's own Qwen3-4B Genie bundle ships W4A16 (models/qualcomm-qwen3-4b-ref/); expected ~2-3× NPU per-step speedup + 4× weight BW reduction. Would push decode to ~20 t/s (model). Cost: AIMET or AI Hub quant pipeline run against humaneval + structured_json as calibration set. ~1 session.
Async NPU-draft ↔ target-verify overlap. Makes wall = max(NPU_round, verify_round) rather than sum. Expected ~11 t/s at FP16 (57% improvement); stacks with W4A16 to ~28 t/s. Cost: ~200 LOC async rework. ~1 session.
Smaller past_len compile tier. Our binary bakes past_len=511. For code drafting at ~256-token generation, a past_len=256 tier would ~halve attention FLOPs per step. Cost: one AI Hub recompile + tiered loader. ~half session.
Zero-copy KV handoff via cl_qcom_ion_host_ptr. Small (~10%) NPU-side win but unblocks DFlash-on-OpenCL's later Phase-4 perf tuning. Phase 6 territory.

(B) Qwen3 close-out before graduation (the stashed Phase-2 items that become orphaned at Qwen3.5 cutover):

--draft-p-min tightening at k=3 on CPU-spec (kept; cheap data point to sharpen our CPU-spec baseline).
prompts/prose_longform.jsonl + prompts/chat_multiturn.jsonl stub content (fill the four-workload matrix before writing up).
Ngram spec (--spec-type ngram-*) A/B on JSON (floor baseline for "dumbest draft").
Negative-result upstream contribution to llama.cpp — the NPU-spec + OpenCL-spec stories together are publishable data on the Snapdragon X2 heterogeneous-exec question.

Recommended order: (B) close-out items first (cheap, graduates Qwen3 cleanly), then decide whether (A) W4A16 is worth chasing on Qwen3 or save for Qwen3.5. Given production-target is Qwen3.5/6 and Phase 4 DFlash on Qwen3.5 is the next big milestone, the W4A16 lever probably lands on Qwen3.5's draft directly rather than Qwen3's.

Strategy (session 5, 2026-04-20): close out Qwen3, then graduate fully to Qwen3.5 for all further work.

The Qwen3 family has been our scaffolding model because it has more public literature (EAGLE PRs, community benchmarks, llama.cpp coverage). Production target is Qwen3.5 → Qwen3.6 -- see memory/project_target_model.md. Rather than running two parallel tracks (keep poking at Qwen3, start Qwen3.5), we treat this as a clean graduation:

Use the Qwen3 window to land everything adventurous that only makes sense on pure-attention + smaller models: NPU drafting is the headline item (simplest path into QAIRT), but any stashed Phase-2 experiments (see below) that we care about should finish in this window too -- they'll be orphaned after graduation.
Once NPU drafting on Qwen3-0.6B + Qwen3-8B has a stable baseline (win, lose, or tie against Phase 2's 1.55× CPU-spec), declare Qwen3 closed and move all subsequent phases (DFlash, NPU-on-dense-small, MoE) to Qwen3.5+.
This keeps the codebase from accumulating two model-family branches and keeps our attention on Qwen3-era questions while the tooling is still fresh.

Revised phase order: NPU-first (close out Qwen3), then Phase 4 DFlash (opens Qwen3.5 era). Session 5 scoped Phase 4 against lucebox-hub and then re-sequenced:

Phase 5 (NPU drafting on Qwen3) moves ahead of Phase 4. The Phase-2 mixed-device negative result told us CPU↔OpenCL sync breaks small-batch heterogeneous exec on this hardware. It did NOT tell us whether heterogeneous exec works at all -- only that ggml-opencl's per-round launch profile loses in the specific CPU-draft + OpenCL-target pairing. The NPU path is the structurally different bet: pipelined async dispatch + ION-backed buffers + parallel-to- target draft. Answering "does heterogeneous work on X2 at all" is a prior-scope question that gates DFlash's eventual Phase-5-style async design. If NPU-draft also regresses, we learn that before committing to a DFlash-on-OpenCL port with the same sync shape.
Phase 4 (DFlash + DDTree on Qwen3.5) follows NPU work. The scoping pass already landed (see "Phase 4" section below); the assets can be downloaded in parallel with NPU bring-up so Phase 4 is ready to start the moment NPU drafting answers its core question. Session 5 scoping remains valid regardless of ordering.
CPU spec peaks at 1.55× (40.2 t/s) at k=3; ceiling ~1.6×. This remains the target-side baseline for both phases.
lucebox-hub hit 3.43× on RTX 3090 with DFlash+DDTree (pure-CUDA, single device). Our win condition is different: we're stitching heterogeneous compute (NPU + CPU + GPU) via shared LPDDR5X, not competing on single-device throughput.
Phase-2 stretch items (--draft-p-min, 14B target, prose/multi- turn, ngram spec, upstream writeup) stay stashed -- none break 1.6×.

Phase intersection — why NPU-first is cheap information

The two tracks share less code than they look to at first, but the one thing they DO share is the work that matters most.

Shared, high-value: zero-copy buffer model on shared LPDDR5X. NPU drafting needs NPU↔CPU/GPU shared allocations via cl_qcom_ion_host_ptr or QAIRT buffer handoff so the NPU draft doesn't pay a cache-flush round trip to hand tokens back to the target. DFlash-on-OpenCL needs the same pattern for its per- round target_feat → draft → verify loop -- the Snapdragon analogue of the D2D copy that gave lucebox +3.3% on PCIe. Doing NPU first forces us to solve ION-backed allocation + async dispatch first; Phase 4's OpenCL port then inherits that infrastructure intact. This is the single biggest intersection.
Shared, low-value: benchmark harness + prompt fixtures + metric logging. sweep_speculative.ps1, the humaneval/json JSONL fixtures, the AL/accept/tok-s columns in our CSVs all carry over. Already built in Phase 2.
NOT shared: kernel ports. NPU drafting doesn't touch ggml-opencl (draft runs on QAIRT, target uses stock llama.cpp). DFlash-on-OpenCL doesn't touch Hexagon. Two independent skill trees; doing one doesn't accelerate the other's kernel work.
NOT shared: target-side code. NPU spec decode uses llama.cpp's stock --draft pipeline -- zero target patching, just swap which binary runs the draft forward. DFlash needs the custom non-libllama target loader + 5-layer hidden capture hooks.
NOT shared: draft/target pairing. NPU standard spec decode only needs tokenizer compatibility (Qwen3-0.6B-draft + Qwen3-8B-target works). DFlash requires the drafter to be trained on the specific target's hidden states -- Qwen3-trained drafter cannot verify a Qwen3.5 target. So NPU-on-Qwen3 and DFlash-on-Qwen3.5 are independent experiments, not a shared lane.

Session 5.5 prep (do in parallel with NPU bring-up)

To keep Phase 4 unblocked once NPU drafting answers its core question, stage these in the background:

Download unsloth/Qwen3.5-27B-GGUF Q4_K_M (~16 GB) into models/. Verify arch=qwen35 in the GGUF metadata.
Download z-lab/Qwen3.5-27B-DFlash safetensors (~3.5 GB BF16).
Read delta_net_chunked.cpp (237 LOC) and list every ggml op it calls. Cross-check against ggml-opencl's supports_op. This is the Phase-4 gating item.
Confirm ggml_rope_ext section-mode support on OpenCL (Qwen3.5 M-RoPE uses sections [11,11,10,0]; plain NEOX is fine for the draft and verified working).

NPU drafting on Qwen3 — why Qwen3 (not Qwen3.5) for this phase

Two sub-questions people conflate:

NPU porting difficulty. Pure-attention (Qwen3 small variants) maps cleanly onto QNN's shipped op library (matmul + rmsnorm + rope
- swiglu). Hybrid variants (Qwen3.5 with gated_delta_net + ssm_conv) require persistent-state handling on NPU that isn't in the sample ops. Hybrid is the blocker, not Qwen3 vs Qwen3.5. Pure-attention variants of either family work equivalently.
Draft/target pairing for standard speculative decoding (not DFlash): only needs tokenizer compatibility. Qwen3 and Qwen3.5 share the same tokenizer (Qwen family policy), so a tokenizer- compatible cross-family pairing is valid. But for apples-to-apples baselines it's cleaner to stay one-family-per-experiment.

So the Phase 5 NPU plan is: bring-up on Qwen3-0.6B-Q8_0 draft + Qwen3-8B-Q4_K_M target (same pair as Phase 2). Once it works, swap to Qwen3.5-dense-small draft + Qwen3.5-dense target for the production-target experiment. We don't attempt NPU + hybrid (Qwen3.5-27B-hybrid) together yet -- layering two unknowns. That combination lands after both Phase 4 (DFlash on hybrid) and Phase 5 (NPU on pure-attention) have baselines.

Caveats carried into Phase 5

Scoping doc is canonical: docs/npu_scoping.md (session 5, 2026-04-20) has the 10-step bring-up plan, toolchain pins (QAIRT 2.45.40, ORT-QNN 1.24.4), known-failure-mode catalog, and prior-art review. The bullets below are the short version; the doc is the single source of truth for Phase 5 execution.
Hedge docs already absorbed into the scoping doc: voice_project/current_status.md + trident's postmortem.md / npu_path_back.md / npu_current_status.md / npu_optimizations_thoughts.md.
Pin the NPU bring-up draft to Qwen3-0.6B-Q8_0 (not 1.7B). Smaller compile iterations + fewer custom-op surprises on the first pass.
Keep target side identical to Phase 2 winning config: Qwen3-8B- Q4_K_M on CPU at 18 threads (25.91 t/s TG baseline, 40.2 t/s CPU-spec at k=3). That way the first NPU-spec number is directly comparable to the CPU-spec result; we can say cleanly whether NPU-as-draft wins, loses, or ties vs CPU-as-draft.
Hexagon arch target is v81 on X2E -- proven by voice_project's working AI Hub compile (dspArch: 81, socModel: 88). Session 6 step 1 still does a QAIRT device-enum sanity check before any code, but v81 is the pin.
Primary path = AI Hub cloud compile + ORT-QNN EP 1.24.4 runtime. Not raw QnnContext_createFromBinary. voice_project hit three driver-signing walls on the raw path and only escaped via ORT's bundled signed QAIRT stack. Our Phase 5 target/draft integration needs an external-drafter sidecar (no QNN backend in llama.cpp), tracked as blocking in npu_scoping.md §8.
Prior-art review integrated from npu_thoughts_previous_examples.md (sd.npu, Mirror-SD, HeteroLLM, Dovetail, OpenPangu). Structural plan unchanged; key post-bring-up lever is sd.npu's pad/recycle trick for <8-token drafts (our k=3 is deep in that regime). Full analysis in npu_scoping.md §10.

Stashed -- now reframed as "Qwen3 close-out" candidates

After the session-5 re-sequencing, these items fall into the Qwen3 window that closes with Phase 5 NPU. Anything we don't do here gets orphaned at graduation -- pick deliberately. Priority marks: [keep] = worth doing before graduation, [drop] = fine to skip, [carry] = trivially portable to Qwen3.5 so not time- pressured.

[keep] Tighten --draft-p-min. Default 0.75 over-drafts on low-confidence streaks (the flatten pathology). Try 0.80, 0.85, 0.90 at k=3. Cheap data point; improves our CPU-spec baseline before we compare it to NPU-draft. Actively useful inside the Qwen3 window.
[carry] 14B target + 0.6B draft (or 1.7B draft). Draft/target compute ratio drops ~2×; break-even k shifts higher. Needs Qwen3-14B-Q4_K_M.gguf download. Not time-pressured -- we'll do the equivalent exercise on Qwen3.5 dense variants after graduation, so skipping on Qwen3 loses no learning. Do only if convenient alongside Phase 5 NPU.
[keep] Prose + multi-turn workloads. prompts/prose_longform.jsonl, prompts/chat_multiturn.jsonl stubs still empty. Worth filling before NPU-draft data collection so the first NPU-spec numbers already cover the full workload matrix we'd want for a writeup.
[keep] Draftless ngram spec (--spec-type ngram-*). Memory- free; should fly on JSON. Quick A/B, closes a gap in the Qwen3 spec-decode story and sets a useful floor for "dumbest possible draft" that NPU-draft and DFlash both need to beat.
[keep] Negative-result contribution upstream. The Adreno- OpenCL spec-decode story (no win at any k or placement) is worth a llama.cpp docs/discussion post. Write after Phase 5 NPU so we can contribute "here's what does work on heterogeneous X2" alongside. High visibility, low effort once the NPU number is in.
[drop] EAGLE-3 viability probe. Was Phase 3 anchor, demoted already. Lucebox paper showed chain-over-tree gives +15% on Q4_K_M (quantization flattens draft softmax); EAGLE-3 alone won't break our 1.6× ceiling. Not worth Qwen3-window time. Revisit on Qwen3.5 only if DFlash underperforms.

Phase 4 (DFlash + DDTree on Qwen3.5) -- queued behind Phase 5 NPU

Reference impl to mine: C:\Users\hotschmoe\Documents\GitHub\lucebox-hub/dflash/src/ (sibling checkout pulled 2026-04-20). See docs/reference-projects.md for file-level guidance on what to read first.

Session 5 (2026-04-20) scoping findings

Read: dflash_graph.h, qwen3_dflash_graph.cpp (168 LOC, draft graph), safetensors_draft.cpp (407 LOC, draft weights), internal.h (288 LOC, shared state/cache schema), qwen35_target_graph.cpp (806 LOC, hybrid target), delta_net_chunked.h (chunked delta-net entry), gguf_target_loader.cpp (386 LOC, non-libllama GGUF loader), RESULTS.md.

Kernel coverage on current ggml-opencl backend (llama.cpp/ggml/src/ggml-opencl/ggml-opencl.cpp, HEAD e365e658f…):

GGML_OP_FLASH_ATTN_EXT -- IMPLEMENTED (ggml_cl_flash_attn at line 9265, dispatch at 14033). Supports_op gate at 4166-4200: supported {dk,dv} pairs include {128,128} which matches our draft head_dim; supported dtypes F32/F32, F16/F16, F32+F16KV. BF16 input not supported so we convert at load, same trick the lucebox CUDA port uses for norms.
GGML_OP_SSM_CONV -- IMPLEMENTED (4085, 13934). Hybrid delta-net 1D causal conv already runs on OpenCL.
GGML_OP_GATED_DELTA_NET -- NOT implemented on OpenCL. This is the Qwen3.5 SSM recurrence kernel. Options: (i) use lucebox's delta_net_chunked.cpp path, which re-expresses the recurrence in already-supported primitives (237 LOC); (ii) CPU fallback for delta-net layers only; (iii) write an OpenCL kernel ourselves. (i) is the only tractable short-term path -- if its ops are all ggml-opencl-backed, we get hybrid on OpenCL for free.
M-RoPE with rope_sections [11,11,10,0] -- still need to verify OpenCL supports section-mode ggml_rope_ext (Qwen3.5 needs it; pure Qwen3 uses plain NEOX which works).

Non-kernel blockers

gguf_target_loader.cpp is POSIX (mmap / open / fstat / munmap). Same ~30-LOC Windows swap as safetensors_draft.cpp (CreateFileMapping/MapViewOfFile). Do both at once.
Target arch string is qwen35 (not qwen3); loader validates this. GGUF we download must match.
Token embedding stays on CPU (CUDA port notes CUDA get_rows can't handle k-quants; OpenCL likely same). CpuEmbedder in internal.h already handles this -- portable as-is.
Target graph uses capture_layers mode to sink 5 specific layer hiddens into a 4096-slot target_feat ring (qwen35_target_graph.cpp:694 — CAPTURE_LAYERS[]). The draft reads this as its "5*hidden" input. Keep as-is; no llama.cpp patching needed since we use the non-libllama loader.
DDTree adds per-delta-layer ssm_intermediate (F16, [S_v, S_v, H_v, max_verify_tokens]) and conv_input_cache. Hybrid-only overhead. Pure-attention targets pay ZERO per-node memory tax (RESULTS.md, "Memory ceiling notes") — the published DFlash paper runs budgets up to 1024 on pure-attention Qwen3-8B/30B. This is the single biggest argument for eventually doing our own pure-attention drafter (option B below).

Draft graph port cost (option A, using z-lab 27B drafter): ~near zero. Every op (ggml_mul_mat, rms_norm, mul, reshape, concat, rope_ext NEOX, permute, cont, flash_attn_ext, silu, add) already on ggml-opencl. Weights loader is pure I/O + a hand-rolled safetensors JSON parser. BF16-on-disk is fine; norms get converted to F32 at load (already in the code).

Path decision: A (today) → B (later). Documented below.

lucebox-hub's reference impl is glued to Qwen3.5-27B-hybrid. z-lab only publishes DFlash drafter weights for 27B. We pick A first to de-risk (reference impl exists, working numbers exist, no training needed) and carry B in the backlog.

Option A — Qwen3.5-27B hybrid target + z-lab 27B DFlash drafter (chosen for now)

Plan:

Download assets:
- unsloth/Qwen3.5-27B-GGUF (Q4_K_M, ~16 GB) into models/.
- z-lab/Qwen3.5-27B-DFlash (safetensors, BF16, ~3.5 GB).
Port safetensors_draft.cpp + gguf_target_loader.cpp mmap→Win32 mapping. ~1 day.
Standalone drafter smoke: port qwen3_dflash_graph.cpp against ggml-opencl backend. No target yet, just prove the drafter forward runs and dims match.
Target forward with full-attention layers only (16 of 64, every 4th). Confirm logits plumbing; garbage output is expected because delta-net layers are stubbed.
Delta-net layers via delta_net_chunked primitives path. If every op has OpenCL coverage → hybrid on OpenCL directly. Otherwise CPU-partition delta-net layers.
Chain verify (q_len=16) end-to-end. Expected AL 6-7 (lucebox Math500/GSM8K numbers).
DDTree verify. Expected AL 8+, ~3× target.
Perf-tune: f16 intermediate, gated_delta_net_tree_persist equivalent, D2D target_feat replacement (Snapdragon shared- LPDDR analogue: avoid cache flush pairs).

Upsides: reference impl exists, weights exist, results are published (3.43× HumanEval on 3090). Strong alignment with stated production target (see user memory: project_target_model.md).

Downsides: 27B Q4_K_M (~16 GB) + drafter (~3.5 GB BF16) + KV + intermediates leaves comfortable margin in 48 GB LPDDR5X -- less than Qwen3-8B (~4.5 GB weights) but nowhere near tight. Long-context 128K via Q4_0 KV would add ~8 GB and still fit. Hybrid port is more unknown than pure-attention. Per-node DDTree memory tax caps budget ~22-26 on hybrid only (a pure-attention target would be uncapped).

Option B — Train our own DFlash drafter for a pure-attention Qwen (revisit later)

Plan:

Pick target: Qwen3-8B (scaffolding) OR Qwen3.5-8B-dense (closer to production, no hybrid).
Build training infra: 5-layer DFlash drafter per the paper (docs/reference-projects.md cites Qwen3-8B/30B-MoE numbers from the original DFlash paper at 4-5× on HumanEval).
Run distillation against target hiddens on a cloud GPU (BF16 training -- not feasible locally on the X2). A few hundred GPU-hours ballpark.
Port the drafter weights + dims into our (by-then-working) DFlash pipeline. Pure-attention target skips the delta-net port entirely. Per-node memory tax is zero -> budgets up to 1024 per the paper.
Retest DDTree at large budgets; upper bound on speedup likely higher than 27B-hybrid because verify-batch memory isn't hybrid-capped.

Upsides: pure-attention path removes the deltanet port entirely; matches any Qwen3.5-dense production target; higher budget ceiling. Weights are ours (no z-lab license considerations).

Downsides: training cost; no existing reference impl at our exact dims; gap between "has a drafter" and "has a good drafter" (distillation quality drives AL).

When to pivot A→B: after Option A lands end-to-end and we have a clean hybrid DDTree baseline on Snapdragon. At that point we know the per-round overhead floor on our hardware and can quantify what training our own drafter buys. Until then, option A is cheaper information.

Dependency queue (session 5 → session 6)

Before writing any port code:

Download unsloth/Qwen3.5-27B-GGUF Q4_K_M (~16 GB) into models/. Bandwidth budget.
Download z-lab/Qwen3.5-27B-DFlash safetensors into models/ (or a sibling dir; it's not a GGUF).
Skim delta_net_chunked.cpp (237 LOC) and list every ggml op it calls; cross-check each against ggml-opencl supports_op. This is the gating item for whether hybrid-on-OpenCL is tractable without writing a new kernel.
Confirm ggml_rope_ext section-mode support on OpenCL (qwen3.5 M-RoPE). If unsupported, plan a CPU shim for rope only.
Decide on build layout: separate phase4/ source tree (lucebox-shaped standalone, non-libllama) vs. integrate into the llama.cpp fork. Lean standalone -- inherits lucebox's structure 1:1 and sidesteps llama.cpp graph-capture patching.

High-level plan (as a checklist, unchanged structure)

Scoping pass (no code). Done this session. ✓
Drafter-weight pipeline: port safetensors_draft.cpp mmap→Win32.
Target-weight pipeline: port gguf_target_loader.cpp mmap→Win32 and confirm arch=qwen35 handling.
Minimum-viable DFlash (chain-verify, no DDTree). Expected AL 6-7, comparable to session-4 CPU chain-spec in rate but higher AL.
Add DDTree verify. Target: clear 2× end-to-end.
Port any delta-net ops that aren't already on OpenCL (see delta_net_chunked.cpp survey above).
OpenCL perf tuning (f16 intermediate, target_feat copy, tree-persist kernel).

Phase 6 -- Standalone lite harness (flagged for way later)

If Phase 5 closes with llama.cpp's backend model still leaving perf on the table (specifically the OpenCL buffer model -- no zero-copy on shared LPDDR5X -- and uncoalesced kernel dispatch), spin a narrow lucebox-shaped harness (~2000 LOC, no libllama link) rather than graduating to a full runtime like trident. See README Phase 6 for scope. Lever: CL_MEM_USE_HOST_PTR / clSVMAlloc / cl_qcom_ion_host_ptr as allocation invariants, plus fused kernels and direct QAIRT↔OpenCL ION-buffer handoff for the NPU draft path.

Phase 7+ -- MoE targets (Qwen3.6-35B-A3B), explicitly deferred

Holding until the fundamentals land on dense/hybrid. Rationale:

We're doing fundamental exploration on niche hardware (X2 Elite Extreme is an almost unstudied platform for spec decode). Stacking architectural novelty (MoE expert routing) on top of platform novelty multiplies the unknowns without adding information.
MoE expert routing is a new op class on NPU. Qualcomm's shipped sample ops don't cover it; custom-op development would be an unrelated rabbit hole that delays the platform-characterization results we're actually trying to get.
Spec-decode research on MoE is still unsettled in the literature -- expert-miss rate adds a second acceptance axis beyond token-accept, and the interactions with tree-verify/DFlash aren't well-characterized.
Our 48 GB LPDDR5X has room for Qwen3.6-35B-A3B Q5_K_M (reference numbers in gguf_models/LOCAL_LLM_NOTES.md), so the only thing deferring Phase 7 costs us is research novelty -- not hardware reach.

What triggers unblocking Phase 7:

DFlash+DDTree on Qwen3.5 hybrid target has a stable baseline (Phase 4 closed).
NPU drafting has a stable zero-copy pipeline on pure-attention (Phase 5 closed, regardless of whether it wins vs CPU-spec).
At least one of (a) Phase 6 lite harness lands, or (b) we can articulate a concrete bandwidth-budget model that predicts MoE expert routing's cost on LPDDR5X.

Then Phase 7 is additive: reuse the existing DFlash target-loader path (qwen3.6 GGUF → extend gguf_target_loader.cpp with MoE tensor naming), add expert-selection to the target graph, and treat it as an incremental A/B vs Phase 4's 27B result.

Deferred / answered

~~Mixed-device CPU-draft + OpenCL-target.~~ Answered, negative. Sync overhead dominates; monotone decode improvement with k tops out at 16.1 t/s, under CPU-alone TG. Not worth more scanning.
~~Narrow k sweep + JSON workload.~~ Done; k=3 is the optimum.

Backlog (non-Phase-2)

Fill out the OpenCL perf matrix beyond 0.6B Q8_0. Run sweep_baseline.ps1 for 1.7B + 8B on OpenCL at -p 1024, 2048 context shapes.
Optional tuning: patch get_adreno_gpu_gen to recognize X2-90 (currently falls to ADRENO_UNKNOWN); opt into cl_qcom_large_buffer via GGML_OPENCL_ADRENO_USE_LARGE_BUFFER=1.
SME2 / KleidiAI retry. -Preset cpu-kleidiai already builds; see docs/SME_investigation.md. Deferred until Phase 2 settles.
Vulkan. Deferred, not abandoned. If a Qualcomm driver update lands, re-run the session-2 correctness matrix (B0-B7).
Upstream-issue search for X2-90 OpenCL to see if anyone else is ahead of us. Low priority now that the backend works.

Session 28 (2026-05-21): cold-start bootstrap on RTX Pro 6000 Blackwell — AIMET sm_120 blocker

Pod: RunPod RTX PRO 6000 Blackwell Server Edition, 96 GB VRAM, ~2 TB host RAM (cgroup limit ~233 GB), 128 vCPU. Driver 580.126.16 (CUDA 13 capable). Network volume /workspace healthy. Following end-to-end/COLD_START.md to recreate the Qwen3-4B w4a16 NPU bundle.

Bootstrap done (COLD_START steps 3-7, token-free path)

Repo already on /workspace/specula. uv installed.
venv /workspace/venvs/aimet-2.26-cu121-py310 created (py3.10).
Installed: qai-hub-models[qwen3-4b] 0.54.0, aimet_onnx 2.26.0+cu121, onnxruntime-gpu 1.23.2, transformers 4.51.0, torch 2.4.1+cu121, plus optimum 2.1.0 + optimum-onnx 0.1.0.
Doc drift found: COLD_START step 4 assumes qai-hub-models pulls optimum — it does NOT. optimum + optimum-onnx must be installed explicitly (optimum 2.x split the export onnx subcommand into the optimum-onnx package; the [exporters] extra is gone). Also qai-hub-models has no qwen3-0_6b extra (only qwen3-4b).
QAIRT 2.45.40.260406 SDK at /workspace/sdks/qairt-2.45.40.260406 (zip extracts to qairt/<ver>/; relocated to the qairt-<ver> path the scripts hard-code).
Qwen3-0.6B (1.5 GB) + Qwen3-4B (7.6 GB) downloaded via direct curl (public HF repos — no token needed).
Tokens: operator confirmed HF + AI Hub not needed (direct download; AI Hub configure step skipped). gh already authed.

Blackwell (sm_120) compat — diagnosed, then RESOLVED

Initial finding: torch 2.4.1+cu121 cannot run on this GPU — a real matmul fails with no kernel image is available; torch.cuda.get_arch_list() caps at sm_90 and ships no PTX (compute_*). RTX Pro 6000 Blackwell is sm_120.

A research pass claimed AIMET would also be a hard blocker (its CUDA custom-op lib libaimet_onnxrt_ops.so ships cubins/PTX only to sm_90/compute_90). That conclusion was wrong and we validated it empirically rather than assuming:

torch: swapped to torch 2.7.1+cu128 — arch list now includes sm_120 + compute_120; fp32 + bf16 matmul on the GPU verified.
onnxruntime-gpu 1.23.2: CUDA EP runs a real MatMul on Blackwell fine — no version bump needed (research's #26177 concern N/A here).
aimet_onnx 2.26.0+cu121: the gamble. Smoke test (/workspace/runs/aimet_smoke.py) — QuantizationSimModel with CUDAExecutionProvider, compute_encodings, and apply_seq_mse all run on sm_120. AIMET's compute_90 PTX does JIT-forward- compile onto Blackwell (PTX forward-compat spans major versions — that is exactly what PTX is for). No +cpu wheel needed; GPU AIMET is live.

Net change vs COLD_START spec: only torch 2.4.1+cu121 → 2.7.1+cu128. onnxruntime-gpu 1.23.2 and aimet_onnx 2.26.0+cu121 unchanged. Venv path name kept (aimet-2.26-cu121-py310) so hard-coded scripts still resolve. AdaScale (torch + onnx2torch, GPU) not yet isolated-tested — will be exercised by the COLD_START step 9 0.6B run, which is the real gate.

Toolchain bring-up — more COLD_START doc drift fixed

Bringing the toolchain up surfaced four more gaps vs the doc (all 2026-05-21, on Ubuntu 24.04 / py3.10):

optimum not pulled by qai-hub-models — installed optimum 2.1.0 + optimum-onnx 0.1.0 explicitly (optimum 2.x moved the export onnx subcommand into optimum-onnx).
qairt-converter needs libc++ — its native bindings (libDlModelToolsPy.so) link libc++.so.1, absent on the base image. Fix: sudo apt-get install -y libc++1 libc++abi1.
torchvision/torchaudio version skew — after the torch 2.7.1 cu128 swap, the stale torchvision 0.19.1 broke optimum's import (torchvision::nms does not exist). Re-pin the whole family from the cu128 index: torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1. Installing torchvision unpinned drags torch to 2.11 — pin all three.
accelerate missing — transformers 4.51.0 references init_empty_weights unconditionally; without accelerate the export dies with NameError. Fix: uv pip install accelerate (got 1.13.0).

optimum-cli export onnx then exported Qwen3-0.6B cleanly (model.onnx + 2.4 GB model.onnx_data). qairt-converter and qnn-context-binary-generator load. dev .bashrc got the SETUP_DEV_USER §6 env block.

Run results — pipeline runs stages 1-6; two real breakages

First 0.6B w8a16 runs exposed two independent breakages, both from optimum 2.x / transformers 4.51 toolchain drift (the pipeline last passed end-to-end in session 11 on an older optimum).

Breakage A — AIMET quant quality. SEQ_MSE-only 0.6B w8a16 probe: cos(fp,q) = 0.535 (gate ≥ 0.99; argmax FP ' Paris' vs Q '-'). Identical 0.535 at opset 17 and 18 → opset does not affect quant quality. README's "cos 0.07 → 0.99" framing means AdaScale is load-bearing, not optional. So AdaScale must work.

AdaScale — diagnosed AND fixed (this session).

It crashed in onnx2torch.convert: Got unexpected input value type (ValueType.UNKNOWN). Diagnosed via runs/debug_adascale.py: AIMET's per-block onnx extraction is correct size (~one block, ~440 nodes) but the optimum/pathb graph routes one shared preamble (embedding, RoPE/mask/shape prep) into every block's extracted subgraph. Its leaf inputs — input_ids (embedding Gather) and past_key_values.0.* (a /model/Shape reads seq-len off layer-0 KV) — were undeclared → dangling → UNKNOWN.
Fix 1 (committed, lib/aimet.py): declare input_ids + past_key_values.0.* as shared block inputs in the patched apply_adascale_pathb. Extraction is then dangling-free.
Fix 2 (committed, lib/stages.py): export at --opset 17. onnx2torch in aimet_onnx 2.26 has no ScatterND-18 / ReduceMean-18 converter; opset 17 emits ScatterND-16 / ReduceMean-13 which it handles. Verified: with both fixes, per-block onnx2torch.convert() succeeds for blocks 0/5/27 — first time AdaScale block conversion works in this repo. An opset-17 + AdaScale 0.6B run is in flight (runs/qwen3_0p6b_w8a16_op17_ada) to confirm the full AdaScale optimisation loop runs and lifts cos.

Breakage B — qairt-converter (stage 7), still OPEN. Node .../input_layernorm/ReduceMean: canonicalizeOp: Reduce param axis must be >= 0 ... get:<garbage> (value differs each run → uninitialised read). Not opset-related — fails identically at opset 17 (ReduceMean-13, axes attribute [-1]) and opset 18 (ReduceMean-18, axes input), and also with axes rewritten to a positive [2]. qairt-converter 2.45's ONNX ReduceMean importer simply never populates the reduce axis for this RMSNorm ReduceMean. The rewrite_qwen3_* scripts have no RMSNorm/ReduceMean handling, so the op reaches qairt-converter raw. This blocks producing any .bin bundle and must be solved — likely via a new graph-rewrite pass that fuses or reshapes the decomposed RMSNorm (Pow→ReduceMean→Add→Sqrt→Div→Mul) into a form qairt-converter 2.45 lowers cleanly.

Open items / next steps

qairt-converter ReduceMean (Breakage B) — the hard blocker for shipping any bundle. Options to try: (a) a graph-rewrite that fuses RMSNorm to a single op qairt handles; (b) replace ReduceMean with ReduceSum+Mul(1/N) or GlobalAveragePool- style ops; (c) check whether a newer QAIRT SDK fixes the importer.
Confirm AdaScale lifts cos — watch runs/qwen3_0p6b_w8a16_op17_ada; expect cos ≫ 0.535 toward 0.99.
opset ≥ 21 check — AIMET int16/int4 QDQ export at opset 17: verify sim.export encodings are still well-formed.
Only after 0.6B w8a16 clears cos ≥ 0.99 and a bundle builds → 0.6B w4a16 → 4B w4a16.

Environment delta (works, for WARM_START)

torch 2.7.1+cu128, torchvision 0.22.1, torchaudio 2.7.1; onnxruntime-gpu 1.23.2; aimet_onnx 2.26.0+cu121; optimum 2.1.0 + optimum-onnx 0.1.0; accelerate 1.13.0; transformers 4.51.0. System: libc++1 libc++abi1 apt-installed for qairt-converter.

BREAKTHROUGH (2026-05-21 ~07:40) — all three blockers root-caused & fixed

qairt-converter (Breakage B) — ROOT CAUSE: numpy 2.x ABI mismatch. Not opset, not version, not ReduceMean. QAIRT 2.45's bin/check-python-dependency declares numpy: 1.26.4 — its compiled bindings (ir_graph / libDlModelToolsPy) are built against the numpy 1.x C ABI. The AIMET venv has numpy 2.2.6. Every IrStaticTensor(numpy_array,…) handoff (Reduce axes, Reshape shapes, …) then reads garbage in C++ → the "Reduce param axis … get:" abort. Instrumented op_adapter.ReduceOp proved Python passes axes=[2] correctly; C++ reads garbage. Confirmed by version sweep (2.42/2.44/2.45 all fail) and op sweep (ReduceSum, MatMul/Reshape all fail) — systemic, not op-specific. Fix: created /workspace/venvs/qairt-py310 with numpy==1.26.4

onnx. With it, qairt-converter converts the real Qwen3-0.6B pathb w8a16 ONNX cleanly → INFO_CONVERSION_SUCCESS, 732 MB DLC. Only qairt-converter (a Python script) needs this venv; qnn-context-binary-generator is a compiled binary (no numpy).

AdaScale (Breakage A) — FIXED & validated. The opset-17 + aimet.py shared-preamble-input fix works: the opset-17 + AdaScale 0.6B run optimised all 28 decoder blocks end-to-end (no ValueType.UNKNOWN crash) — first time AdaScale has run in this repo.

Blackwell — fixed earlier (torch 2.7.1+cu128).

Remaining work

Patch lib/qairt.py to run qairt-converter with the qairt-py310 (numpy-1.x) venv python.
Re-run stage 7-9 on the AdaScale'd 0.6B output (--force-stage 7) → first complete bundle.
Full clean 0.6B w8a16 run → check cos ≥ 0.99 gate.
Then 0.6B w4a16, 4B w4a16, compare to Qualcomm reference.

Persistent venvs (for WARM_START)

/workspace/venvs/aimet-2.26-cu121-py310 — AIMET/AdaScale + pathb (numpy 2.2.6, torch 2.7.1+cu128). Stages 1-6.
/workspace/venvs/qairt-py310 — qairt-converter only (numpy 1.26.4). Stage 7.

MILESTONE (2026-05-21 ~07:50) — first complete end-to-end bundle

The full 9-stage pipeline ran end-to-end on Qwen3-0.6B w8a16: runs/qwen3_0p6b_w8a16_op17_ada/09_bundle_w8a16/qwen3-0p6b-w8a16-pathb-ctx512-x2e.tar (0.81 GB). Stages 7-9 (qairt-converter → 732 MB DLC → qnn-context-binary-generator → 728 MB .bin → bundle tar) all pass with the numpy-1.x venv fix. The pipeline is structurally unblocked — optimum/pathb/AIMET+AdaScale/qairt/qnn all work on the Blackwell pod.

Open: quantization quality. Probe cos(fp,q) = 0.557 (gate 0.99). AdaScale ran 28/28 blocks but only nudged cos 0.535→0.557. runs/cos_diag.py is sweeping quant schemes / bitwidths to localise the cause (min_max-forced-by-SEQ_MSE vs tf_enhanced vs structural).

cos diagnostic (2026-05-21 ~07:55) — the quality gap is STRUCTURAL

runs/cos_diag.py swept QuantSim configs on the 0.6B pathb graph (probe cos vs FP, 64 cal samples):

config	cos
w8a16 min_max	0.5416
w8a16 tf_enhanced	0.6568
w8a16 percentile	0.6548
w16a16 tf_enhanced	0.6545
w8a8 tf_enhanced	0.0000

w16a16 (near-lossless 16-bit weights) still caps at cos 0.65 — so the quality gap is NOT quantization precision. min_max only adds a small extra loss (0.65→0.54). w8a8 confirms activations need ≥16-bit (8-bit acts fully collapse). There is a structural ~0.65 ceiling independent of bitwidth. Next test: QuantSim CPU vs CUDA (cos_cpu.py) — is the AIMET CUDA custom op JIT'd onto Blackwell numerically wrong? (It was verified to run, not to be correct.)

4B w4a16 production run launched (runs/qwen3_4b_w4a16) — stages 1-5 (~1h) run while the cos root cause is chased; if a stage-6 fix lands before 4B reaches AIMET, 4B restarts --force-stage 6.

cos: CPU == CUDA (2026-05-21 ~08:20)

QuantSim w8a16 tf_enhanced run fully on CPU → cos 0.6548; CUDA → 0.6568. Identical → the AIMET CUDA custom op on Blackwell is NOT the cause. The structural ~0.65 ceiling is real on both. quant_ablation.py running to bisect (disable param/activation/input quantizer groups). 4B w4a16 run is in stage 6 (SEQ_MSE).

cos: ROOT CAUSE isolated — activation quantization (2026-05-21 ~08:30)

runs/quant_ablation.py (QuantSim w16a16, disable quantizer groups):

ablation	cos
baseline (all on)	0.6545
ALL quantizers disabled	1.0000
graph-INPUT disabled	0.6586
PARAM (weight) disabled	0.6540
ACTIVATION disabled	0.9921

ALL-off → cos 1.0 proves the QuantSim graph transform + probe are correct. Disabling activation quantizers recovers cos 0.99 → activation quantization of intermediate tensors is the whole problem, even at int16. Hypothesis under test (cos_maskval.py): the additive attention mask sentinel -65504.0 (fp16 −inf) blows up the post-mask score-tensor quantizer range so normal scores get ~1.0 int16 granularity. Sweeping the sentinel; if cos jumps as |sentinel| shrinks, the fix is a quantization-friendly mask value.

cos ROOT CAUSE confirmed (2026-05-21 ~08:45) — RMSNorm internals + mask constants

Mask-sentinel swept (-65504..-30): cos flat at 0.6559 → not the input mask. The quantizer-range dump found the real offenders:

/model/Constant_27_output_0, /model/ConstantOfShape_output_0 — range ~4.2e37 (internal mask "−inf" constants).
every layers.N/{input,post_attention}_layernorm/Pow_output_0 — range ~4.8e6 (RMSNorm x²; the residual stream has massive activations so x² hits millions).

int16 activation quantizers on tensors with range 5e6–4e37 → catastrophic granularity → model destroyed. RMSNorm internals and the mask constant must not be activation-quantized — transformers 4.51 exports RMSNorm decomposed (Pow/ReduceMean/Add/Sqrt/Div/Mul) so AIMET quantizes every intermediate. cos_threshold.py validating the fix (disable activation quantizers above a range threshold). The fix applies post-compute_encodings; AdaScale disables activation quantizers internally so the 4B AdaScale work is unaffected.

cos FIX implemented + validated (2026-05-21 ~09:40)

cos_final.py confirmed the fix (build int16 QuantSim, compute_encodings, then reconfigure activations):

config	cos
w8 + fp16-mixed activations	0.9994
w8 + activations off (fp32)	0.9994
w4 + fp16-mixed activations	0.9300
w4 + activations off (fp32)	0.9302

The 'a16' in w8a16/w4a16 is fp16, not int16. lib/aimet.py now (stage 7b, post-compute_encodings): every fp16-safe activation quantizer → float16; the ~53 fp16-overflow ones (RMSNorm x², mask constants, |val|>60000) → disabled (fp32); weights stay int8/int4. 0.6B w8a16 probe cos 0.65 → 0.9994. w4 basic 0.93 — SEQ_MSE + AdaScale (+ V/O pin) in the full pipeline should lift it.

Runs launched: 4B w4a16 --force-stage 6 (fixed) + fresh 0.6B w8a16 full validation (runs/qwen3_0p6b_w8a16_fix).

Known bug (logged): _bump_vo_to_w8 in aimet.py detected 72 V/O weights but bumped 0/72 — the w4a16 V/O-collapse mitigation is a no-op (quantizer-name lookup mismatch). To fix.

Full fixed pipeline — runs relaunched (2026-05-21 ~09:55)

lib/aimet.py now carries both fixes:

7b activation precision: int16→fp16 (fp16-safe) / fp32 (overflow).
4b V/O w8 pin: detect V/O weights on the clean graph (pre-QuantSim mutation) and bump to w8 before SEQ_MSE (so w8 scales are searched at the pinned bitwidth). Previously bumped 0/N — name mismatch.

Relaunched:

runs/qwen3_0p6b_w8a16_fix — 0.6B w8a16 full validation (cos fix; no V/O pin — off for w8a16).
runs/qwen3_4b_w4a16 — 4B w4a16 production, --force-stage 6, complete fixed recipe (cos fix + working V/O pin). ~2.5-3 h.

Both monitored. Expect 0.6B w8a16 probe cos ~0.99; 4B w4a16 cos to be the Lever-C datapoint.

Disk-full incident + recovery (2026-05-21 ~10:15)

3 concurrent runs + ~50 GB of orphaned AdaScale tempdirs (from earlier diagnostic scripts / killed runs) filled the 100 GB ephemeral overlay (/tmp) — AdaScale writes a copy.deepcopy fp32 model there (~16 GB for 4B). The 4B run died with OSError: No space left on device; cleanup then deleted live 0.6B tempdirs too. Recovery: purged orphaned /tmp/tmp* (overlay → 78 GB free), restarted 4B w4a16 + 0.6B w8a16 --force-stage 6 (stages 1-5 cached on /workspace, unaffected — /workspace never filled). Skipped the 3rd concurrent run; added a disk watchdog. 0.6B w4a16 (Lever C) deferred — cos_final.py already gave the w4 0.6B datapoint (basic cos 0.93). Op note: AdaScale tempdirs are ephemeral-overlay-bound; don't run many AIMET jobs concurrently without watching /tmp.

Activation precision: fp16 vs int16 — resolved to int16 (2026-05-21 ~11:00)

The fp16 activation fix (cos 0.65→0.9994) is HTP-incompatible: qnn-context-binary-generator (stage 8) aborts — no properties registered for q::QNN_Convert — every fp16 quantizer becomes an fp32→fp16 QNN_Convert the HTP graph-prep cannot create (the source ONNX is fp32; fp16 anywhere needs a convert). 0.6B w8a16 reached stage 8 and failed there.

Inspected Qualcomm's reference bundle: its w4a16 HTP .bins use integer activations — metadata.json boundary tensors are 1440× uint8 + 160× uint16 + 10× int32, each with per-tensor scales. The HTP wants integer activations; fp16 is out.

So lib/aimet.py 7b reverted — activations stay int16 (the QuantSim build type). int16 bundles DO compile on HTP (op17_ada produced a complete 728 MB .bin). The int16 probe-cos collapse (~0.55-0.65) is now the single open quality gap:

It is NOT precision (w16a16 also 0.65), NOT one op-type, NOT the pathological-range tensors, NOT the AIMET CUDA op (CPU==CUDA), NOT the mask sentinel. Disabling ALL activation quantizers → 0.99.
Qualcomm achieves good quality with int16 activations, so the gap is a calibration / quant-config difference — likely candidates for next session: per-channel activation quant, a non-min_max activation observer that survives SEQ_MSE, KV-cache precision handling, or AIMET's quant config (op exclusions) matching Qualcomm's recipe.

Runs restarted with int16: 4B w4a16 + 0.6B w8a16 (--force-stage 6). They will produce complete, HTP-compilable bundles (structural end-to-end recreation); probe cos is expected low pending the int16 calibration fix.

0.6B w8a16 int16 — complete bundle (2026-05-21 ~11:46)

runs/qwen3_0p6b_w8a16_fix/09_bundle_w8a16/qwen3-0p6b-w8a16-pathb-ctx512-x2e.tar (806 MB): .bin (763 MB HTP context binary) + .encodings (32 MB) + bin_info.json + metadata.json + config/tokenizer. Confirms the fully-fixed pipeline runs all 9 stages end-to-end and produces a structurally-correct, HTP-compilable bundle. Probe cos 0.557 (int16 activation gap — documented open item). Launching 0.6B w4a16 (--force-stage 6) alongside the 4B run.

V/O pin ↔ AdaScale conflict (2026-05-21 ~12:25)

0.6B w4a16 (V/O pin + AdaScale) failed stage 7: qairt-converter weights: For Per-channel, offset must be 0 or -2^(bw-1). Cause: AdaScale uses a single global ADASCALE_PARAM_BW (=4 for w4a16) to write weight encodings; the V/O-pinned w8 weights then get a bw-4 offset (−8) in a bw-8 slot → invalid. AdaScale doesn't compose with mixed-bitwidth weights. (The repo never hit this — V/O pin bumped 0/72 before the lookup fix.)

Resolution: run w4a16 with --no-use-ada-scale. SEQ_MSE computes per-tensor encodings at each quantizer's own bitwidth (w8 V/O, w4 rest) → valid offsets → qairt-converter passes. AdaScale was not moving the probe cos anyway (the cos gap is int16 activations, a weight technique can't fix it), and it's the ~2 h bottleneck — so dropping it for w4a16 is the right call. Both w4a16 runs restarted --no-use-ada-scale. (Proper fix for later: make AdaScale honor per-quantizer bitwidth, or recompute V/O encodings post-pin.)

w4a16 path UNBLOCKED — embed fix works (2026-05-21 ~13:15)

0.6B w4a16 produced a complete bundle (940 MB) — embed pin ([aimet 4c] 1/1 → int16 per-tensor) + V/O pin (56/56) + --no-use-ada-scale. All stages 1-9 pass. The full w4a16 recipe is: SEQ_MSE (no AdaScale) + V/O-w8 pin + embed-w16 pin. 4B w8a16 is finishing; 4B w4a16 to run next (reusing pathb stages).

Session 28 SUMMARY (2026-05-21) — Blackwell cold-start, pipeline brought up

Outcome: the AIMET→NPU pipeline was completely broken on the new RTX Pro 6000 Blackwell pod; it now runs end-to-end and produces complete HTP-compilable bundles for ≤1B models. Qwen3-4B is quantized through AIMET; the multi-part packaging is the last blocker.

Delivered (committed to master)

Complete bundles on /workspace/runs/:

qwen3_0p6b_w8a16_fix/09_bundle_w8a16/…tar — 0.6B w8a16, 806 MB, 763 MB .bin. All 9 stages.
qwen3_0p6b_w4a16_fix/09_bundle_w4a16/…tar — 0.6B w4a16, 940 MB, 898 MB .bin. All 9 stages.
qwen3_4b_w4a16/06_aimet_w8a16/ + 07_dlc_w8a16/ — 4B w8a16 quantized + 4.4 GB DLC (multi-part split pending).
4B w4a16 AIMET stage 6 running at session end.

Eight root-caused fixes (all committed)

Blackwell sm_120 — torch 2.4.1+cu121 has no sm_120 kernels; swapped to torch 2.7.1+cu128. (onnxruntime-gpu 1.23.2 + AIMET 2.26 cu121 both run on Blackwell as-is.)
COLD_START doc drift — optimum+optimum-onnx+accelerate not pulled by qai-hub-models; libc++ needed for qairt-converter; QAIRT zip extract path. (COLD_START.md updated.)
AdaScale — opset-18 ReduceMean/ScatterND crashed onnx2torch (→ export --opset 17); block extraction missed shared-preamble inputs (→ declare input_ids+pkv0 in lib/aimet.py). First time AdaScale runs in this repo.
qairt-converter — QAIRT 2.45 bindings are numpy-1.x ABI; the numpy-2.x venv made every IrStaticTensor read garbage. Added /workspace/venvs/qairt-py310 (numpy 1.26.4); lib/qairt.py runs the converter under it.
V/O w8 pin — detected post-QuantSim (mutated names) → bumped 0/N; now detect on the clean graph, bump pre-SEQ_MSE.
Embedding pin — w4a16 int4 embed table vs int16 Gather output → QNN rejects (Gather needs in==out encoding). Pin embed table to int16 per-tensor (lib/aimet.py stage 4c).
V/O-pin ↔ AdaScale conflict — AdaScale's global ADASCALE_ PARAM_BW writes bw-4 offsets into bw-8 V/O slots → qairt rejects. w4a16 runs use --no-use-ada-scale (AdaScale wasn't moving cos).
fp16 vs int16 activations — see "cos" below.

Two OPEN items (next session)

A. int16 activation cos gap. Probe cos: 0.6B w8a16 0.557, 0.6B w4a16 0.61, 4B w8a16 0.44 (gate 0.99). Root-caused: int16 fixed-point can't span LLM activation dynamic range. fp16 activations → cos 0.9994 BUT HTP-incompatible (fp32→fp16 QNN_Convert rejected). Qualcomm's reference uses integer activations (uint8/uint16) at good quality → the gap is a calibration/quant-config difference, not precision. Candidates: per-channel activation quant, KV-cache precision, AIMET quant-config op exclusions matching Qualcomm.

B. 4B multi-part split. A 4B model's single .bin exceeds the HTP 3.5 GB serializer limit → must split (Qualcomm ships 4 parts). compile_split_bundle.py/lib/split.py extracts parts but parts ≥3 fail. Precise diagnosis: exactly ONE shared tensor — /model/ScatterND_output_0 (the internal attention mask, produced by /model/ScatterND, consumed by every layer's self_attn/Slice_4, all 36 layers) — is a real cross-part dependency that build_part_specs does not declare. Parts 1-2 absorb the preamble that produces it; parts 3-4 consume it but their PartSpec only declares attention_bias — which is DEAD in this graph (part2 warned "declared inputs not reached: ['attention_bias']"). So extract_part for part3 walks the undeclared mask back to the embed Gather → input_ids → error. Fix recipe (next session): in lib/split.py build_part_specs, thread /model/ScatterND_output_0 as cross-part I/O — part2 (and any middle part) adds it to outputs, parts 3..N add it to inputs (drop the dead attention_bias); needs the tensor's shape via onnx.shape_inference (not in value_info) and verification that genie routes a part-2 output to parts 3-4 (X2E-side, untestable on the cloud pod). 4B is fully quantized (06_aimet_*); this packaging step is the only thing between here and a complete 4B bundle.

Environment (persistent on /workspace, for WARM_START)

venvs/aimet-2.26-cu121-py310 — AIMET stages 1-6. torch 2.7.1+cu128, numpy 2.2.6, aimet_onnx 2.26.0+cu121, onnxruntime-gpu 1.23.2, transformers 4.51.0, optimum 2.1.0.
venvs/qairt-py310 — qairt-converter only. numpy 1.26.4.
sdks/qairt-2.45.40.260406; models/Qwen3-0.6B, Qwen3-4B; reference/qwen3_4b_qualcomm/ (Qualcomm w4a16 X2E reference).

Cleanup (2026-05-21 ~14:30) — /workspace quota is 200 GB

/workspace hit ~178 GB (quota 200 GB). Cleaned to 59 GB:

removed extra QAIRT SDKs from the version sweep (2.42, 2.44 — kept only 2.45.40.260406);
4B run: kept only qwen3_4b_w4a16/06_aimet_w8a16/ (16 GB — the quantized 4B w8a16 AIMET output, the input for the split-fix in task #13); dropped regeneratable stages 1-5/7/8;
0.6B runs: kept only the final 09_bundle_* tars; dropped stage intermediates. Kept: 2 complete 0.6B bundles, the 4B w8a16 AIMET output, venvs, QAIRT 2.45 SDK, models, Qualcomm reference. The 4B w4a16 run was stopped (no bundle — blocked by task #13). All monitors + background shells removed. Op note: keep /workspace lean — 200 GB cap; AIMET stage 1-5 intermediates are ~16 GB each for 4B, prune them.

Session 29 (2026-05-21) — 4B split fixed; first complete 4B bundle

Outcome: the two open items from Session 28's task #13 are resolved. lib/split.py now splits the 4B pathb graph into 4 weight-sharing parts that compile all the way through qairt-converter + qnn-context-binary-generator, and the first complete 4B bundle (w8a16, 4-part genie, 4.53 GB) is on disk. The 4B w4a16 AIMET run that was interrupted mid-SEQ_MSE has been re-run to completion.

split.py — root cause + fix (2026-05-21 ~14:40)

Two bugs, both now fixed in lib/split.py:

Cross-part attention mask. transformers exports the causal mask as an internal ScatterND; the pathb fold-pathbmask rewrite folds it in and leaves the original attention_bias graph input dead (0 consumers — confirmed). The folded mask /model/ScatterND_output_0 is built once in a 173-node preamble and consumed by a Slice in all 36 layers' self_attn. After a 4-way split the preamble lands wholly inside part2, so parts 3-4 had no producer for the mask and extract_part's back-walk fell through to input_ids → the RuntimeError. Fix: detect_shared_attn_mask() finds the tensor structurally (fed to a Slice in >=2 distinct layers). build_part_specs threads it as cross-part I/O — part2 exports it, the last part imports it, middle parts pass it through (import + re-export, a bare input==output graph edge). The dead attention_bias is dropped from every decoder part. Mask shape [1,1,1,ctx] FLOAT (== the old attention_bias shape; verified by running the mask subgraph). This matches Qualcomm's reference topology — every decoder part takes the mask per-part (their genie feeds it; ours threads the folded one).
Encodings schema. split_encodings only handled the legacy name-keyed-dict schema; AIMET 2.26 emits the 1.0.0 list schema (activation_encodings/param_encodings are lists of {"name": ...} objects). Added a format-aware subsetter (_subset_encoding_section) handling both.

Validated end-to-end on the 4B w8a16 AIMET output: all 4 parts extract (part3 = 3112 nodes, previously crashed), onnx.checker passes on every part (incl. part3's mask passthrough), all 4 parts clear qairt-converter (INFO_CONVERSION_SUCCESS) and qnn-context-binary-generator. Untested (X2E-side): whether the genie runtime routes part2's mask output through to parts 3-4 — the pass-through design covers a strict consecutive-wiring runtime.

First complete 4B bundle — w8a16 (2026-05-21 ~14:59)

compile_split_bundle.py --precision w8a16 --num-parts 4 --dsp-arch v81 ran clean: split → qairt ×4 → qnn ×4 → genie bundle. runs/qwen3_4b_w4a16/09b_bundle_w8a16/qwen3-4b_w8a16_pathb_ctx512_x2e_v81.tar (4.53 GB). Per-part .bin: 389 MB / 1221 MB / 1221 MB / 1613 MB — all under the 3.5 GB HTP serializer ceiling that blocked the single-bin path. Bundle carries genie_config.json, htp_backend_ext_config.json, metadata.json, bin_info/, encodings/, tokenizer — matches the Qualcomm reference layout. Probe cos 0.44 (the int16-activation quality gap, still open).

4B w4a16 AIMET re-run — complete (2026-05-21 ~14:53)

The Session-28 4B w4a16 run (interrupted mid-SEQ_MSE) was re-run. Stages 1-5 regenerated (they had been pruned in cleanup), then AIMET stage 6 with the full fixed recipe: SEQ_MSE, --no-use-ada-scale, V/O-w8 pin bumped 72/72, embed-w16 pin 1/1. Output at runs/qwen3_4b_w4a16/06_aimet_w4a16/. Probe cos 0.51 (int16 gap). The single-bin stages 7-9 were skipped (not valid for 4B); the multi-part compile_split_bundle.py --precision w4a16 is the path.

4B w4a16 bundle — complete (2026-05-21 ~15:12)

compile_split_bundle.py --precision w4a16 ran clean through the fixed split. 09b_bundle_w4a16/qwen3-4b_w4a16_pathb_ctx512_x2e_v81.tar (3.72 GB). Per-part .bin: 778 / 694 / 694 / 1475 MB — part1 (embed) is ~2× the w8a16 389 MB because w4a16 pins the embed table to int16. Both 4B bundles are now on disk; the Session-28 4B packaging blocker (task #13) is closed.

Cleanup: pruned the split/DLC/bin intermediates + the w4a16 stage-5 ONNX (~50 GB). Kept on runs/qwen3_4b_w4a16/: both 06_aimet_* AIMET outputs (16 GB each), both 09b_bundle_* bundles (tar + dir). /workspace at 84 GB.

Quantization quality — P0 fix found and landed (2026-05-21 ~16:30)

Stood up docs/e2e_optimizations.md (the quality plan) and ran a parallel agent sweep. Track 4 (qai-hub-models diff, docs/qai_hub_recipe.md) found the root cause: we built QuantizationSimModel with config_file=None → AIMET's default default_config_per_channel.json (supergroup_pass_list ["MatmulAdd"], no RMSNorm pass), so int16 activation quantizers landed on every decomposed-RMSNorm intermediate (the x² tensor @ range ~5e6 etc.) → signal annihilation → cos ~0.55.

P0 fix (committed): vendor Qualcomm's default_config_llama.json as end-to-end/lib/aimet_config_llama.json and pass it as config_file. Its supergroup_pass_list ["LayerNormalization","RMSNormalization"] triggers aimet_onnx's RMSNorm graph pass, which disables the norm-internal output quantizers (HTP float fallback). Also tie Concat quantizers + ignore Slice/Constant outputs (mirrors qai-hub-models _build_quantsim).

0.6B w8a16 probe cos 0.557 → 0.9984, argmax now matches FP. The int16 gap was never precision — it was the missing QuantSim config. This collapses old Tracks 1/2/3 into one config change; the harness (end-to-end/eval_quality.py) and the P1-P4 refinements (int8-tied KV + 16x8 matmuls, mask clip to [-100,0], AdaScale block count) are tracked in docs/e2e_optimizations.md. 4B w8a16/w4a16 re-validation with P0 is in flight.

4B P0 validated; P1 landed (2026-05-21 ~17:30)

P0 confirmed on the Qwen3-4B replication anchor:

4B w8a16: cos 0.44 → 0.9962, argmax matches FP — gate cleared.
4B w4a16: cos 0.51 → 0.9751, argmax matches FP — P0 closes most of the gap but misses the 0.99 gate by ~1.5%.

P1 (int8-symmetric in/out-tied KV cache + 16x8 attention matmuls + int8 per-channel lm_head, imported from qai-hub-models — lib/aimet.py stage 4d, w4a16-only) merged to master (7d12df5). It is the w4a16 precision config expected to close the residual gap (and structurally replaces the _bump_vo_to_w8 workaround; matches Qualcomm's int8 lm_head, which also fixes the part4 +38% size delta). 4B w4a16 P0+P1 re-validation in flight (runs/qwen3_4b_p0p1).

Disk hygiene: pruned superseded pre-P0 4B AIMET outputs + bundles + the spent 0.6B P0 validation run (~66 GB freed). /workspace 85 GB / 200, local / 38 GB / 100.

w4a16 ablation campaign — complete (2026-05-21 ~21:00)

Exhaustively swept every recipe lever to close the 4B w4a16 gap (docs/w4a16_ablation.md). Verdict: P0 is the ceiling — nothing beats it. 6 runs on cached pathb stages:

run	recipe	cos
A0	P0	0.9751
A6	P0 − V/O pin	0.9758
A1	P0 + P2 mask-clip	0.9751
A2	P0 + AdaScale (−V/O pin)	0.9649
A4	P0 + scoped-P1	0.9501

Findings: AdaScale hurts (−0.011, isolated A6→A2 — regresses the probe, doesn't lift it); the V/O-w8 pin is neutral dead weight (A0≈A6 — recommend defaulting --no-vo-pin-w8 for w4a16 under P0, for a smaller model at zero cos cost); P2 mask-clip is inert; all precision-reduction levers (P1/scoped-P1/16x8/int8-KV) regress. The 0.975→0.99 gap is intrinsic int4-weight error — no qai-hub recipe knob closes it. w8a16 (int8 weights) clears the gate (0.996); w4a16 floors at ~0.975 with argmax matching FP on every ablation.

P1 reverted earlier (a3f7416) as a validated regression. New flags landed for the campaign: --mask-clip-min, --scoped-p1. P0 4B bundles (w8a16 + w4a16) built and transferred to the X2EE laptop.

Still open

w4a16 0.975 — accepted as the recipe ceiling. The 1.5% to the 0.99 gate is int4-weight precision, not a recipe miss. Final arbiter: on-device comparison vs Qualcomm's reference bundle on the X2 Elite (both bundles now on the device).
Genie part-to-part mask routing — needs X2E hardware to confirm the threaded mask reaches parts 3-4. Note Qualcomm's reference bundle instead feeds attention_mask to every decoder part from the runtime (genie computes it, like RoPE) — if our genie wiring turns out to need that, the alternative is to expose the mask as a per-part input named attention_mask rather than threading the pathb-folded one. Decided to thread (preserves pathb's self-contained-mask design); revisit if hardware says otherwise.

Session 36 (2026-06-15) — A1 + C1 closed; FA-prefill characterized

Worked the top of docs/2026-06-14_contribution_opportunities.md. Three milestone commits: fd23540 (A1), d44925f (C1), c6e0e33 (FA).

A1 — the "dense-prefill regression" is not real (RESOLVED)

git bisect over the 489-commit window 856c3adac..e37abd6b5 (new harness: scripts/bisect_prefill.ps1 + build_llama_cpp.ps1 -NoGit/-Targets; threshold 257 = midpoint of calibrated CPU endpoints 331.9/181.7) landed on aa46bda89 "Support -fa auto in llama-bench (#23714)". It only flips llama-bench's default flash-attn OFF→AUTO. Each build ships its own llama-bench, so session 35's A/B compared -fa off vs -fa auto, not two model builds. Control on the SAME new build with -fa 0: pp512 = 370 (CPU) / 569 (OpenCL) — meets/ beats every old record. No compute regression. Retire build-opencl-old; e37abd6b5 is a clean upgrade. Memory reference_llamacpp_prefill_regression_e37abd6b5 corrected (was wrong).

C1 — Q4_0 vs Q4_K_M perplexity (RESOLVED)

wikitext-2-raw, 584 chunks @ n_ctx=512, both unsloth GGUFs, -fa 0: Q4_0 14.6395 ± 0.134 vs Q4_K_M 14.7910 ± 0.138 — a tie within error, Q4_0 nominally lower AND smaller. Q4_0's speed win carries no quality penalty → validated production default for Qwen3-4B. Data: results/csv/qwen3_4b_perplexity_q4_0_vs_q4km.csv. Next: repeat for Qwen3.6 sizes before the production cutover.

FA-prefill characterization (A1 spin-off → A2/E4)

Full -fa 0/-fa 1 sweep (docs/2026-06-15_flash_attn_prefill_slowdown.md, raw results/csv/fa_sweep_2026-06-15.md): FA always loses on prefill (up to 2.18× on dense-4B OpenCL; 1.15–1.39× on the 35B MoE), while decode is small+mixed (FA on helps CPU +11% / 35B ngl0 +9%, nothing on GPU-offload). Cleanest upstream bug: Adreno OpenCL FA prefill ~2.2× slower with zero decode benefit. Includes a workload×backend -fa recommendation matrix.

Parked TODOs — two upstream issues drafted, not yet filed

Both are ready to post under the user's GH identity; reproduction caveats satisfied. These are the project's first concrete OSS contributions.

FA-prefill slowdown (Adreno OpenCL). Doc: docs/2026-06-15_flash_attn_prefill_slowdown.md + raw results/csv/fa_sweep*_2026-06-15.md. Headline: Adreno OpenCL FA prefill up to ~3.2× slower than non-FA with no decode benefit, across 5 models / 2 families / 2 quants. Repro: llama-bench -m Qwen3-0.6B-Q8_0.gguf -p 512 -n 128 -ngl 99 -ub 512 -t 16 -fa 0,1 → pp512 2767 (fa0) vs 859 (fa1).
Vulkan prefill collapse (Adreno X2-90). Ready-to-post draft: docs/upstream_drafts/vulkan_adreno_prefill_collapse.md; analysis docs/2026-06-15_vulkan_prefill_broken.md + raw results/csv/vulkan_prefill_repro_2026-06-15.md. Headline: dense prefill collapses ~18× with prompt length (pp8 118 → pp512 6.4 t/s), decode fine; 59–92× slower than CPU/OpenCL. Not -fa, not ubatch, not model-specific, coopmat ruled out → general large-M mul_mat path. Repro: llama-bench -m Qwen3-4B-Q4_0.gguf -p 8,32,64,128,256,512 -n 0 -ngl 99 -t 16 -fa 0.

Next session

A2 — characterized + draft ready (above). Optional pre-file control: GGML_VK_DISABLE_F16=1 to confirm the F16 path, then file.
A3 — -ngl 0 coprocessor profiler trace.
E1/E3 characterization matrix + energy/thermal (background).
File the parked FA upstream issue (above).
E1/E3 characterization matrix + energy/thermal (background).

FilesExpand file tree

current_status.md

Latest commit

History