gfx906 (Radeon VII, wave64): Gated Delta Net models (Qwen3-Next/3.8) run at ~0.12 t/s while MoE & dense are fine

[philmcneely]

Summary

On gfx906 (Radeon VII / Vega 20, wave64), Gated Delta Net models (the Qwen3-Next / "Qwen3.8" family) run at ~0.12 tok/s generation and ~0.16 tok/s prompt — effectively unusable — while MoE and standard models on the same card are fast and the same GDN model on an RDNA2 (wave32) card is fine. So this looks specific to the GDN / linear-attention path on wave64, not a general gfx906 or bandwidth problem.

I have a Radeon VII on a Mac and am happy to run experimental builds and report back (see offer at the bottom).

Environment

• GPU: AMD Radeon VII (gfx906, Vega 20, wave64, 16 GB HBM2), eGPU over Thunderbolt 3
• Host: Intel Mac mini 2018 (Macmini8,1), macOS 15.7.9
• ToshLLM: v0.85.x (Aug 2026 build) — engine build 10495, llama.cpp base commit 3dc7285b4
• Device init banner:

  wave64 mode (SIMD width 64): GPU prefill matmul, CPU decode/reductions for correct output
  wave64 decode ON: quantized/f16/bf16 mat-vec on GPU (see the allowlist in ggml_metal_library_get_pipeline_mul_mv)
  simdgroup reduction   = false
  simdgroup matrix mul. = false
  

Measurements (all via llama-server, one small /completion each, --load-mode none)

Model	Card	Config	Prompt t/s	Gen t/s
Qwen3.8-27B IQ4_XS (GDN)	Radeon VII (gfx906)	-fa 1 -ctk q8_0 -ctv q8_0	0.16	0.12
Qwen3.8-27B IQ4_XS (GDN)	RX 6800 (gfx1030, wave32)	-fa 1 -ctk q8_0 -ctv q8_0	~39	22.6
gemma-4-26B-A4B IQ4_XS (MoE)	Radeon VII (gfx906)	-fa 1 -ctk q8_0 -ctv q8_0	81	38.5

So on the same Radeon VII, a MoE model gets 38.5 tok/s but the GDN model gets 0.12 tok/s (~300× slower); and the same GDN model on an RDNA2 card gets 22.6 tok/s. On gfx906 both prefill and decode collapse together for the GDN model (~0.12–0.16 t/s), which suggests something in the GDN graph is falling to CPU (or a wave64 GDN/reduction kernel is pathologically slow) rather than a decode-only issue.

Second, possibly-related bug: -fa 0 crashes on gfx906

Running the same GDN model with flash attention disabled (-fa 0, f16 KV) crashes the Metal backend on gfx906 immediately on first decode:

ggml_metal_synchronize: error: command buffer 0 failed with status 5
error: Internal Error (e00002bd:Internal Error)
ggml_metal_graph_compute: backend is in error state from a previous command buffer failure - recreate the backend to recover
llama_decode: failed to decode, ret = -3

(With -fa 1 there is no crash — it just runs at 0.12 t/s.) This may be a separate wave64 bug in the non-flash-attention attention path; noting it in case it's the same root cause.

What I've ruled out

• Not the card / not general gfx906 — MoE (gemma-4-26B-A4B) runs at 38.5 tok/s on the same card.
• Not the model / not the quant — the same Qwen3.8-27B IQ4_XS file runs at 22.6 tok/s on an RX 6800 (wave32) with identical flags.
• Not mmap — --load-mode none is used throughout (the fleet servers run this way; llama-bench without it hits the known mmap slowdown, so all numbers above are from llama-server with --load-mode none).

The GDN wave64 kernels do exist in the tree (GGML_OP_GATED_DELTA_NET / SSM_SCAN / SSM_CONV are enabled for wave64_decode, and there is a kernel_gated_delta_net_impl using gdn_sum32), so this reads as a perf/correctness bug in the wave64 GDN path rather than a missing kernel.

Offer

I have this Radeon VII on a Mac available as a gfx906 test rig and can build from source (scripts/build-engines.sh) and run any experimental branch, capture ISA (ISA_MCPU=gfx906), or grab profiles/logs you'd like. Happy to help validate a fix — just point me at a branch or tell me what to instrument.

[engeldlgado]

Thanks, this is the gfx906 report I was missing.

Two things about the comparison first. has_simdgroup_mm is false on both cards, so without TOSH_FA_AMD=1 attention runs on the CPU on the RX 6800 as well. If the 22.6 t/s came from the app and the 0.12 t/s from a shell, the two runs are not the same path. The Radeon VII being an eGPU also matters: Metal defaults external devices to shared buffers, so the weights stream over Thunderbolt on every op. The app disables that, the CLI does not.

What would settle it:

1. The same command on both cards, with and without TOSH_FA_AMD=1.
2. GGML_SCHED_DEBUG=2 on the slow run. It prints which node lands on which backend, and tells us whether something in the GDN chain falls to the CPU. Both prefill and generation hitting the same floor looks like a fixed per-pass cost, not slow math.
3. GGML_METAL_SHARED_BUFFERS_DISABLE=1 on the Radeon VII.
4. For the -fa 0 crash: GGML_METAL_WAVE64_SAFEMODE=1. If it goes away it is a wave64 kernel, and I have a candidate: the wave64 decode gate for MUL_MAT does not check ggml_is_transposed, and without flash attention the second attention matmul has a transposed src0.

The scheduler dump is the one I need most.

[philmcneely]

Checkpoint on your suggestions. One caveat up front: the Radeon VII (gfx906) box is temporarily offline, so I ran what I could on a sibling RX 6800 (wave32) machine — same ToshLLM build, same eGPU-over-Thunderbolt setup, 64 GB host. The gfx906 numbers you actually want (the 0.12 t/s speed case, the scheduler dump, -fa 0) will follow once it's back tonight. Reporting the shared-buffers result now since it's clean.

Shared buffers (GGML_METAL_SHARED_BUFFERS_DISABLE=1) — measured on RX 6800

Qwen3.8-27B IQ4_XS, -ngl 999 --load-mode none -fa 1 -ctk q8_0 -ctv q8_0, one eGPU, 1 Hz host-RAM sampling through load → settle → one completion:

	default (shared)	SHARED_BUFFERS_DISABLE=1
host RSS (steady)	~14.5 GB	~1 GB (1.6 GB peak)
model load	34 s	15 s
generation	~22 t/s	18.6 t/s
completion reloads model?	no (RSS +150 MB)	no

So on the RX 6800 the shell default does put the full weights in host RAM (~14.5 GB), and the flag moves them to VRAM (~1 GB host) as you described — confirming the storage-mode behavior. The interesting part for the gfx906 question: on RX 6800 the shared default is a memory cost, not a speed cliff — it still generates at ~22 t/s with weights in host RAM. Your theory is that on gfx906 the same shared default is what produces the 0.12 t/s (weights streamed over Thunderbolt every op). That contrast is exactly what I'll measure on the Radeon VII next.

To follow on the gfx906 box tonight

1. Same table on the Radeon VII, with and without SHARED_BUFFERS_DISABLE — does it turn 0.12 t/s into real speed (your point Can only connect on 127.0.0.1 #3).
2. GGML_SCHED_DEBUG=2 dump on the slow run. (Heads up: under llama-server the placement print is at GGML_LOG_DEBUG, which the server's log level drops — I have a local build that promotes ggml_backend_sched_print_assignments to GGML_LOG_INFO so it shows. Happy to share the one-line diff if useful, or I can capture via llama-cli if that surfaces it for you.)
3. TOSH_FA_AMD=1 on/off, both cards.
4. The -fa 0 crash with GGML_METAL_WAVE64_SAFEMODE=1 — to confirm your ggml_is_transposed candidate.

Thanks for the fast, detailed response — the storage-mode angle wasn't on my radar and it already looks like it explains the memory side. Will report the gfx906 set tonight.

[philmcneely]

Got onto the actual gfx906 (Radeon VII, eGPU, macOS 15.7.9, 32 GB host) now. Results with GGML_METAL_SHARED_BUFFERS_DISABLE=1 + q8_0 KV on Qwen3.8-27B IQ4_XS:

Shared buffers — you were right, and it's most of it

• Host RAM: 14.5 GB → 1.6 GB (weights VRAM-resident). Confirms the storage-mode fix on gfx906 too.
• Generation: 0.12 → 6.1 t/s (prompt 0.16 → 12.5). So the TB3 shared-buffer streaming was the dominant penalty — killing it recovers ~50×.

Residual gap is one kernel: GATED_DELTA_NET at head_size=128

The card is still ~3× slower than an RX 6800 on this model. test-backend-ops perf on the Radeon VII localizes it precisely — it's head-size-specific:

GATED_DELTA_NET case	us/run	GB/s
head_count=32, head_size=128 (Qwen3.8's config)	9147	0.43
head_count=32, head_size=128, kda=1	10362	0.38
head_count=16, head_size=64	42	12.0

head_size=64 is fine; head_size=128 is ~200× slower, at 0.43 GB/s on a ~1 TB/s card (~0.04% of bandwidth) — so it's compute/occupancy, not bandwidth. (SSM_SCAN is also low: 14951 us/run, 1.02 GB/s.)

Why I think it's occupancy/register pressure on wave64

kernel_gated_delta_net_impl splits S_v across tx (fixed 32 lanes, so gdn_sum32 reduces one 32-lane half) with NSG = S_v/32 elements per thread. Dispatch is (32, nsg, 1):

• S_v=64 → (32, 2) = 64 threads = one wave64 wavefront, float ls[2]/thread.
• S_v=128 → (32, 4) = 128 threads = two wavefronts, float ls[4]/thread + the full q/k/v/g/b recurrence.

At NSG=4 the per-thread live state (ls[4] + accumulators + pointers) looks like it pushes VGPR usage past the point where gfx906 keeps decent occupancy — consistent with the 0.43 GB/s / near-idle behavior. I haven't captured the ISA to confirm the VGPR count yet.

Offer

I have this Radeon VII available as a gfx906 test rig and can build from source (scripts/build-engines.sh) — happy to capture the ISA/VGPR count for the S_v=128 GDN variant, try register-pressure reductions, and re-run test-backend-ops perf against the 9147 µs baseline. If you have a preferred direction (spill ls to threadgroup memory, different thread mapping for head_size≥128, etc.), point me at it and I'll test on real hardware.

[philmcneely]

Narrowing the gfx906 GATED_DELTA_NET head_size=128 slowdown (9.1 ms/run, 0.43 GB/s):

• Not register/occupancy-limited. The slow pipeline kernel_gated_delta_net_f32_4_ne20=128 reports maxTotalThreadsPerThreadgroup = 1024 — identical to the fast f32_2_ne20=64. If it were spilling, that would be low. So it isn't a VGPR-pressure problem.
• Not the precompiled metallib. Forcing a runtime JIT compile of the embedded shaders (bypassing default.metallib) leaves head_size=128 at ~9 ms/run — no change.
• Recap: the shared-buffers fix (GGML_METAL_SHARED_BUFFERS_DISABLE=1) recovered the streaming penalty (0.12 → 6.1 t/s) and the RAM shadow (14.5 → 1.6 GB), but this GDN kernel is the entire residual gap vs an RX 6800 (~3×).

So it points at the memory/algorithmic path of the delta-net recurrence over the S_v×S_v state at S_v=128 specifically (S_v=64 is fine at 42 µs / 12 GB/s), not occupancy. 0.43 GB/s on a ~1 TB/s card. I don't have a GPU-trace to pin the exact stall, but the register + JIT causes are ruled out. Radeon VII test rig still available if you want specific counters/experiments run.

[philmcneely]

gfx906 (Radeon VII) decode optimization — findings 2026-08-25

Model: Qwen3.8-27B (dense, qwen3-next arch: GDN linear-attn + MoE-free FFN), IQ4_XS, on a Radeon VII eGPU (Thunderbolt, 40Gb/s, 16GB), macOS 15.7.

Result: 13.8 → 16.8 t/s (+22%), correct output.

Biggest win: flash-attention was silently running on the CPU

Profiling with sample showed ggml_compute_forward_flash_attn_ext_f16_one_chunk as the top CPU consumer during decode (~480% CPU, GPU only ~76% busy). The AMD FA GPU kernel is opt-in via TOSH_FA_AMD=1. Setting it moved FA to the GPU: 14.4 → 16.8 t/s, GPU util 76% → 89%, CPU 480% → 245%. No fallthrough warnings (fa_amd_width_ok true on wave64).

Mat-vec is at its ceiling (not bandwidth-bound)

The IQ4_XS mat-vec hits ~330 GB/s absolute on BOTH the Radeon VII and an RX 6800 — i.e. it's limited by the 4-bit dequant access pattern, not memory bandwidth. Confirmed insensitive to tuning: nsg=4/nr0=4 = 16.77, nsg=2/nr0=2 (4× threadgroups) = 16.70, load-batch ILP = +2%. So the VII's 1TB/s HBM2 is not exploitable for single-stream 4-bit decode, and split-K won't help (not occupancy-bound).

Remaining gap vs RX 6800 (23.4 t/s) is wave64/GCN small-op latency

Same llama.cpp code hits 23.4 on the RX 6800 (RDNA2/wave32). The ~17ms/token of non-mat-vec work (GDN reductions, norms) is where GCN (wave64, 4-cycle SIMD16) loses to RDNA2 (wave32, 1-pass). GPU is 89% busy = kernel-bound (verified via Metal System Trace).

Hazards found

• GGML_METAL_CONCURRENCY_ENABLE=1 hard-hangs the box (reboot required) — the AMD concurrent-encoder barrier issue is real on gfx906.
• GGML_METAL_WAVE64_UNSAFE=1 produces garbage output on gfx906 (64-wide simd reduction corrupts).
• N_R0_IQ4_XS=8 hangs the pipeline at init.

Investigation continues on GCN-specific reduction-kernel tuning.

[philmcneely]

Follow-up 2026-08-25: MTP (multi-token prediction) self-speculative decode on gfx906

Tested the fork's built-in MTP support (LLAMA_CONTEXT_TYPE_MTP / common_speculative_impl_draft_mtp) on Qwen3.8-27B, which ships a nextn head (blk.64.nextn.*).

Setup: the --mtp flag was scoped to the download tool only; added LLAMA_EXAMPLE_SERVER/LLAMA_EXAMPLE_COMMON to its set_examples so it works at server startup (needed so the model loads with load_mtp=true). Ran with --mtp --spec-draft-n-max N.

It activates and is correct — creates an MTP draft context against the target model, nextn head loads, output coherent. Acceptance is good on normal prompts: mean 1.6–2.5 tokens/step.

But net slower than plain decode on a Thunderbolt eGPU:

• n_max=2 → 14.98 t/s (vs 16.8 plain)
• n_max=4/8 → erratic; a 0-acceptance segment at n_max=8 craters to 0.15 t/s (6540 ms/token — drafts 8, rejects all)

Root cause: MTP does many target↔draft context switches per step. On a Thunderbolt-attached GPU each switch pays TB round-trip latency, and that fixed overhead swamps the acceptance benefit. This is the same latency-bound theme as the rest of gfx906 decode. MTP should be a real win on a PCIe-attached gfx906 (e.g. MI50) where there's no TB latency — the acceptance is there.

Net for Twiki (Radeon VII eGPU): 16.8 t/s (TOSH_FA_AMD=1) remains the ceiling for single-stream 4-bit decode of this model.

[philmcneely]

Correction to the MTP analysis (2026-08-25)

My earlier "Thunderbolt latency" explanation for why MTP is slow was wrong — retracting it. Once weights are VRAM-resident, TB only carries command submissions + tokens, and the RX6800 is also an eGPU and does fine. Profiling gave the real answer.

MTP on the VII is GPU-bound (94-99% GPU util during MTP decode, CPU ~2%). It's doing extra GPU work, not stalling. Best config is --spec-draft-n-max 1 (natural for a single nextn head): 15.4 t/s, 65% acceptance, mean len 1.65 — still under plain 16.8.

Real root cause: the MTP verify step is a small batch (ne11=2), but IQ4_XS has no batched mul_mv_ext kernel. IQ4_XS isn't in the ext type allowlist, so batched decode falls back to per-column full weight reads. Proof from test-backend-ops perf — IQ4_XS MUL_MAT scales linearly:

• n=1 → 94 µs, n=2 → 180 µs (1.9×), n=3 → 266 µs (2.8×)

i.e. a 2-token verify reads the weights ~2×, so MTP spends ~1.8 forwards to produce ~1.65 tokens → net slower than plain decode.

Fix / path to 20+: add IQ4_XS to mul_mv_ext (batched kernel that reads weights once across ne11 columns; iq4_nl already has one — IQ4_XS just adds super-block scale handling). Then the verify batch ≈ 1 forward, and MTP at ~1.65× → ~27 t/s. Mechanism is proven via perf; the kernel fix is the remaining work. This would also speed up any small-batch IQ4_XS inference (prefill of short prompts, batched serving), not just MTP.

[engeldlgado]

Excellent, I'll check it out as soon as I can, thank you for such wonderful details.