PK2 wave9: ADR-011 perf re-gate -- G-ADR11-HYBRID-PERF tok/s curve (K0 23.84 / K4 5.99 / K8 3.25), sync-bound slope analysis + flatten-the-curve options; scoreboard row updated

Author: nihilistau

papers/PPT-LAT-ADR-011-CPU-LAYER-OFFLOAD.md

@@ -5,8 +5,8 @@ description: "ADR-010 proved the model (~10.9 GB) fills the 12 GB card and PCIe
 tags: [design, adr, cpu-offload, vram, layer-placement, pcie, avx2, crt, bit-exact, hybrid, whole-machine]
 timestamp: 2026-07-08T00:00:00Z
 resource: shannon-prime-lattice/papers/PPT-LAT-ADR-011-CPU-LAYER-OFFLOAD.md
-sp_status: "STAGE-2 REALIZED — CPU FFN offload live+coherent, ~0.96 GB freed @ K=8; speed pending the AVX2/OpenMP perf-lib link. VRAM lever + coherence PROVEN."
-sp_gate: "G-ADR11-MEASURE (VRAM curve) + G-ADR11-HYBRID (live: coherent 'Paris' + VRAM 11201→10221 MiB) — engine tests/perf"
+sp_status: "STAGE-2 REALIZED + PERF-GATED — CPU FFN offload live+coherent, ~0.96 GB freed @ K=8. Perf libs (AVX2/OpenMP) make it usable (K=4 ~6 tok/s / K=8 ~3.25 vs 23.84 baseline); curve is STEEP (per-FFN GPU↔CPU sync-bound, not compute)."
+sp_gate: "G-ADR11-MEASURE + G-ADR11-HYBRID (coherent+VRAM) + G-ADR11-HYBRID-PERF (tok/s curve: K0 23.84 / K4 5.99 / K8 3.25) — engine tests/perf"
 sp_commit: "builds on ADR-010 (the model fills the card) + core/forward/gemma4.c (CPU forward, reused)"
 sp_repro: "SP_G4_CPU_TAIL_MEASURE=1 in gemma4_kv_open → the per-layer + tail curve in the daemon log"
 ---
@@ -108,3 +108,28 @@ Remaining (next): (a) link the AVX2/OpenMP CPU libs into the daemon so the offlo
 (b) parallelize the tail FFNs across the 8 cores; (c) a `SP_BYTEEXACT` CRT bit-parity check
 (coherence is proven; bit-identity is the auditable stronger claim). The measurement instrument
 `SP_G4_CPU_TAIL_MEASURE` remains for sizing K.
+
+## 7. Perf re-gate (G-ADR11-HYBRID-PERF) — the AVX2/OpenMP libs, and the real slope
+
+Linked the `build-cpu-perf` math-core (clang-cl `/arch:AVX2` + OpenMP; `gemma4_ffn_block_cpu`
+vectorized) into the daemon via `SP_SYSTEM_BUILD_DIR=build-cpu-perf` + LLVM `libomp`, target
+`target-wirecuda-perf` — the same pattern the qwen36 lane uses. Measured curve (PMAX=4096, decode
+tok/s, coherent all K):
+
+| K (tail FFNs on CPU) | tok/s | VRAM | freed |
+|---|---|---|---|
+| 0 (all-GPU) | **23.84** | ~11201 MiB | — |
+| 4 | **5.99** | 10737 MiB | ~0.46 GB |
+| 8 | **3.25** | 10221 MiB | ~0.96 GB |
+
+**The perf-lib win is real:** K=8 went **0.5 → 3.25 tok/s (~6.5×)** vs the scalar `build-cpu` gate;
+K=4 is an interactive **~6 tok/s**. **But the curve is far steeper than the §3 memory-bound ideal**
+(K=8 predicted ~9.4, measured 3.25). Per-token: K0 42 ms → K4 167 ms → K8 308 ms = **~33 ms per
+offloaded FFN per token**, vs the ~4 ms memory-bound compute. The extra ~29 ms is the **per-FFN
+GPU↔CPU round-trip**: FFN-only offload *interleaves* with GPU attention, so each offloaded layer
+forces `D2H + 2× cudaStreamSynchronize + H2D` — ~2K stream syncs/token. **The sync, not the CPU
+compute, is the wall.** The honest revised verdict: a VRAM-for-latency lever with a *steep*
+sync-bound slope on this interleaved architecture — usable where VRAM is the hard constraint
+(K=4 frees ~0.5 GB at interactive ~6 tok/s). To flatten it: a cross-token decode pipeline (overlap
+token t's CPU FFN with token t's later GPU layers) or pinned-async copies — the next real speed
+project. Default-off = the 23.84 tok/s all-GPU null floor.

papers/VERIFIED-SCOREBOARD.md

@@ -55,7 +55,7 @@ Method: a read-only fleet checked each claim against (a) a commit that resolves
 | **recall∘L5 composition** (the two recall authorities, both armed) | **FREE composition REFUTED → one-authority guard ENFORCED, LIVE-GREEN 6/6** | harness (wave 5) | `G-PK2-RECALL-L5-COMPOSE-FREE.log` (the honest negative, 4/6) + **`G-PK2-RECALL-L5-COMPOSE.log` (enforced, 6/6 live)** | VERIFIED — Phase A (both armed): L5's systemecho CROSS-PICKED color-adjacent queries from the counterfact corpus and overrode the harness note ("favorite color?" → "Human blood is green"; "sky?" → "Green") — L5 selection cross-picks are a known daemon residual; composition surfaces them user-visibly. Phase B: the gateway auto-disarms spine recall whenever the request arms L5 (`auto_recall=true`), with an `{"authority":"L5"}` receipt event; guard held, L5 authority intact ("Lyon"), harness authority faithful when L5 off ("Teal."). Plus `auto_recall` body passthrough (default false). The one-authority rule is now STRUCTURAL, not operator discipline |
 | **ADR-009 batched prefill under the SWA ring** (`SP_KV_PREFILL_BATCH`, the ring-off precondition lifted) | **LIVE-GREEN 3/3 — bounded ~7× win + graceful fallback; default-off** | engine (wave 6) | `tests/perf/G-PK2-BATCHRING.log` (3/3) | VERIFIED — `PPT-LAT-ADR-009-PREFILL-SPEED.md`. The batched cold prefill was BLOCKED under the ring (so the ring config never used it); wave 6 lifts it (ring-layout `k_ring_sink` for SWA owners + retained contiguous shared-owner K/V for sharers). Live @ PMAX=4096: **n≈837 → 6.7s coherent (~7× vs ~39s per-token)**; n≈1765 → VRAM guard DECLINES → graceful per-token fallback 99s coherent (null floor, live-proven). Two guards: VRAM (`SP_KV_BATCH_VRAM_MARGIN_MB`) + persist (declines under `SP_PERSIST_KV`, since a batched-ring cache isn't a valid persist-continuation base). Verdict across 3 waves: **batching is a bounded cold-prefill win, not the general lever** — the f32 activation scratch caps it; cublasGemmEx-int8 would hit the same ceiling (not pursued). Default-off; production per-token path unaffected |
 | **ADR-010 whole-machine balance** (measured VRAM breakdown + KV auto-fit; the CPU-offload verdict) | **ANALYSIS (measured) + auto-fit realized default-off** | engine (wave 7) | `tests/perf/G-PK2-AUTOFIT.log` | VERIFIED (measurement) — `PPT-LAT-ADR-010-WHOLE-MACHINE-BALANCE.md`. Measured live: **model ~10.9 GB resident FILLS the 12 GB card; KV cache only ~0.78 GB (shared-KV — NOT the 2.5 GB I first assumed); ~0.57 GB free.** ★HONEST CORRECTION: the original ">1000-tok stall" root was co-resident LM Studio (~6.4 GB oversubscription), and the ADR-009 batch thrash was the transient scratch eating the thin headroom — the KV was never the movable mass, THE MODEL IS. Every weight-offload refuted by PCIe x8 (6.2 GB/s forbids per-token weight streaming); the ONE lever that frees real VRAM = **CPU-resident layer offload** (weights in 40 GB/s DRAM, compute on AVX2, exchange only activations; CRT keeps CPU+GPU legs bit-identical) — the ADR-011 candidate. `SP_G4_KV_AUTOFIT` (default-off) clamps Pmax to free VRAM for the co-tenant case; dedicated card serves coherent ("Paris" 12.4s), null floor intact |
-| **ADR-011 CPU-resident layer offload** (FFN-tail hybrid — REALIZED live) | **STAGE-2 REALIZED — coherent + ~0.96 GB freed @ K=8; speed pending perf-lib** | math-core+engine (wave 8) | `tests/perf/G-ADR11-MEASURE.log` + **`G-ADR11-HYBRID.log`** | VERIFIED — `PPT-LAT-ADR-011-CPU-LAYER-OFFLOAD.md`. MEASURE (dry): 48 layers, 7.86 GB matmul weights, ~167.8 MB/layer. HYBRID (live): `gemma4_ffn_block_cpu` (math-core, reuses `sp_matmul` on OK_Q4B) + `g4_kv_step` routes the last `SP_G4_CPU_TAIL` FFNs to the CPU + `build_weights` skips those FFN uploads. **`SP_G4_CPU_TAIL=8` → VRAM 11201→10221 MiB (freed ~0.96 GB) + coherent "Paris"** on the served 12B. ★DESIGN PIVOT: shared-KV couples tail attention to early GPU KV owners, so offload the **FFN only** (~90% of layer weight, STATELESS) — clean sever, one E-float crossing. HONEST perf caveat: daemon links non-AVX2 `build-cpu` libs → CPU FFN scalar+slow (tiny prompt 46.7 s; 299-tok prefill minutes) — the VRAM lever + coherence are PROVEN, the **speed follow-on = link `build-cpu-perf` (AVX2/OpenMP)**. Default-off = null floor; the one real VRAM lever, realized |
+| **ADR-011 CPU-resident layer offload** (FFN-tail hybrid — REALIZED + perf-gated) | **STAGE-2 REALIZED + PERF-GATED — coherent, frees VRAM, usable at ~6 tok/s (K=4); steep sync-bound slope** | math-core+engine (waves 8–9) | `tests/perf/G-ADR11-{MEASURE,HYBRID,HYBRID-PERF}.log` | VERIFIED — `PPT-LAT-ADR-011-CPU-LAYER-OFFLOAD.md`. `gemma4_ffn_block_cpu` (math-core, `sp_matmul` on OK_Q4B) + `g4_kv_step` routes last `SP_G4_CPU_TAIL` FFNs to CPU + `build_weights` skips those uploads. ★DESIGN PIVOT: shared-KV → offload the **FFN only** (~90% of layer weight, stateless). PERF re-gate on the AVX2/OpenMP daemon (`build-cpu-perf`+libomp, target-wirecuda-perf), PMAX=4096: **K=0 23.84 tok/s → K=4 5.99 tok/s (freed ~0.46 GB) → K=8 3.25 tok/s (freed ~0.96 GB), coherent all K.** Perf libs made it usable (K=8 0.5→3.25 tok/s ~6.5×). HONEST slope: ~33 ms/offloaded-FFN/token = the per-FFN GPU↔CPU round-trip (D2H+2×sync+H2D, ~2K syncs/token) — **sync-bound, not compute-bound**, far steeper than the ~8.4× memory-bound ideal. VRAM-for-latency lever; usable where VRAM is the hard constraint. Default-off = 23.84 tok/s null floor. Flatten-the-curve (next) = cross-token decode pipeline |
 
 ## The open frontier (what we actually skipped)