[BlivionIaG]

Ports the LLMM1 wave32 fix from FaisalBiyari/vLLM-ALU-RDNA2@a8f8c02 (csrc/rocm/skinny_gemms.cu, author FaisalBiyari).

What changed

• LLGemm1_kernel: K/8 threads means up to 32 waves for K <= 8192 on wave32, but the second-stage reduction only read partials 0..15. For K > 4096 on gfx10/11/12 the dot product silently lost waves 16..31. The fix folds them into the first 16 lanes. wave64 (gfx9) is unchanged (compile-time WARP_SIZE == 32). Commit authored by Faisal, verbatim.
  • This is probably the "LLMM1 is numerically wrong on RDNA" result we saw earlier (gemv_f16_rdna2 comment, AGENTS.md). gfx10 decode stays on gemv_f16_rdna2; the user-visible effect is on gfx11/12 (LLMM1 is reached for n=1, m%4==0, K<=8192).
• wvSplitK: TORCH_CHECK(!on_gfx10()). gfx10 is neither __HIP__GFX9__ nor __HIP__GFX1X__, so the kernels are the assert(false) stub and a call faults the queue. With this change, direct callers get an error instead. The Python dispatcher already never routes gfx10 here. Adapted from Faisal's gfx1030 guard (Co-authored-by).
• Tests: skip the wvSplitK tests on gfx10. New LLMM1 case (1, 4104, 1024), 17 waves on wave32, the smallest K that reaches the fixed partials.

Tests run (no GPU)

• ruff check / ruff format --check on the test file: pass
• clang-format 21.1.2 --dry-run --Werror on skinny_gemms.cu: pass

GPU validation to run

• Rebuild _rocm_C (gfx1030; gfx1100 too if available).
• pytest tests/kernels/quantization/test_rocm_skinny_gemms.py -k "llmm1 or wvsplitk" on gfx1030: before the fix, LLMM1 should fail at K in {4104, 6144, 8192}; after it, LLMM1 should pass and wvSplitK should be skipped. On gfx1100: LLMM1 should pass and wvSplitK should be unchanged.
• No serving path change is expected on gfx1030, since decode stays on gemv_f16_rdna2. To confirm: Flash-Next W4A16 TP=4 F&P, 1k/512 and 16k/1k at c=1/c=8, MTP0, with prefill tok/s, decode tok/s/req, TTFT and ITL within noise of rdna_extras.

---

Re-evaluation 2026-11-11

Verdict: ready to merge. The fix is correct, cannot change wave64 code, and was exercised on a real wave32 GPU (V620, gfx1030) and by compiling skinny_gemms.cu for six architectures. One test commit was added (below). Not merged; waiting for the merge decision.

Branch state

Base	rdna_extras 1136bb952b
Head	8efea0ee37 = a635342ab0 + merge of rdna_extras (17527859a3, clean: upstream only renamed two include paths in skinny_gemms.cu) + one test commit
Net diff vs base	2 files, +39 / -1 (csrc/rocm/skinny_gemms.cu, tests/kernels/quantization/test_rocm_skinny_gemms.py)
Authorship	4d6b653133 FaisalBiyari. a635342ab0 and the two new commits are BlivionIaG (Co-authored-by Faisal on a635342ab0). No agent-authored commit, so no re-authoring is needed.
pre-commit	passes on the new commit (ruff, ruff format, typos, mypy, SPDX, test tethering, ...)

New commit 8efea0ee37 (test only). The PR's (1, 4104, 1024) case does not fail on the unfixed kernel: at K=4104 wave 16 holds 8 of the 4104 products (0.2 %), below the test's rtol=1e-2. It stays (it covers the 17-wave path) and (1, 4352, 1024) is added: the 17th wave is full (5.9 % of the sum), so the case fails without the fix. The existing (1, 6144, 1024) and (1, 8192, 2048) cases also fail without the fix on any wave32 device.

The bug is still on rdna_extras

LLGemm1_kernel launches NUM_THREADS = max(rows_per_block * 16, ceil32(K / 8)) threads: 544 threads (17 waves) at K=4104 up to 1024 (32 waves) at K=8192 on wave32. The second stage reads red_smem[row][threadid % 16], so partials 16..31 are dropped. wave64 has at most 16 waves for K <= 8192 and is not affected. The same expression is on upstream vllm-project/vllm main (csrc/rocm/skinny_gemms.cuh, acc[qwarpid] = qthreadid < num_warps ? ...), so the fix could also go upstream. Upstream has since split skinny_gemms.cu per kernel family (vllm-project#62718), so this hunk has to be re-applied when the fork next rebases onto upstream.

Who reaches LLMM1 / wvSplitK

Dispatcher: vllm/model_executor/layers/utils.py rocm_unquantized_gemm_impl. The only other callers are the _custom_ops.py wrappers and tests; nothing in tools/ or benchmarks/ calls them.

arch	wave	LLMM1 via dispatcher	wvSplitK via dispatcher	effect of this PR
gfx90a / gfx942 / gfx950	64	yes: n=1, m in {4, 8}, no bias, K <= 8192	yes: n 1..5, m == 1 or m > 8	none: WARP_SIZE == 32 is false at compile time, device code unchanged (table below)
gfx1100 / gfx1200 (RDNA3/4)	32	yes, same conditions	yes	LLMM1 returns correct results for 4096 < K <= 8192 (m in {4, 8}, e.g. small router-style GEMVs; no model measured); K <= 4096 results unchanged (the added fold is never taken); wvSplitK unchanged (on_gfx10() is false)
gfx1030 (RDNA2)	32	never	never	LLMM1 numerics fixed for direct callers and tests; direct wvSplitK calls raise instead of hitting the assert(false) stub

The description above says m%4==0 for LLMM1; the dispatcher also requires not use_wvsplitk, which leaves m in {4, 8} for n=1. On gfx10 the dispatcher routes fp16 n <= 8 to gemv_f16_rdna2, n 9..32 to rdna_gemm_smallm, everything else to torch/rocBLAS. gfx1250 does not build this file.

Evidence on a V620 (gfx1030, wave32), HIP 0, ROCm 7.14, torch 2.12.0+rocm7.14.0

_rocm_C built for gfx1030 from the exact trees (rdna_extras 1136bb952b and this branch's merge 17527859a3), same tree path and build dir so code objects are comparable. Kernel code is identical in 8efea0ee37 (test-only commit on top).

check	rdna_extras _rocm_C	this PR
LLMM1 structural sweep, 56 cases (K in {4096, 4104, 4352, 5120, 6144, 7168, 8192} x rows_per_block {2, 4, 8, 16} x fp16/bf16, exact 0/1 data)	48 fail: every wave >= 16 contributes 0 (the 8 passing cases are K=4096 = 16 waves)	0 fail
LLMM1 random sweep, 8192 cases (K = 8..8192 step 8 x 4 rows_per_block x 2 dtypes, rel. error vs fp32)	4078 fail, K = 4112..8192, worst 52 % (about half the sum lost at K=8192)	0 fail, worst 9.3e-4 (fp16) / 6.8e-3 (bf16), both at K=8 (output rounding)
LLMM1 output bits, 8192 seeded cases (K = 8..8192 step 8, M=64, 4 rows_per_block, fp16/bf16; SHA-256 of the output)	reference	K <= 4096: 4096 of 4096 cases bit-identical. K > 4096: all 4096 differ, first at K=4104 (the first K that has a wave 16)
pytest test_rocm_skinny_gemms.py -k llmm1 (80 cases)	24 failed, 56 passed: K=6144 and 8192 (existing shapes) and K=4352 (new). K=4104 passes	80 passed
whole test_rocm_skinny_gemms.py	not run (the wvSplitK tests hit the assert stub; the PR skips them on gfx10)	80 passed, 2604 skipped (2316 gfx950-only, 120 wvSplitK on gfx10, 168 FP8), 0 failed
tests/model_executor/layers/test_rocm_unquantized_gemm.py	8 failed, 26 passed, 2 skipped	8 failed, 26 passed, 2 skipped (the same 8 fail on rdna_extras: they assume a gfx9/gfx11 dispatcher and expect wvSplitK / LLMM1 to be called, which gfx10 never does)
dispatcher reachability: 3744 rocm_unquantized_gemm_impl calls (fp16/bf16, n in {1..6, 8, 9, 16, 24, 32, 64, 128}, m in {1, 4, 8, 16, 128, 4096}, k in {64, 2048, 4096, 4104, 6144, 8192}, contiguous / row-stride-padded, with and without bias)	588 gemv_f16_rdna2, 3156 torch; LLMM1 / wvSplitK / wvSplitKrc reached 0 times	identical (0 reaches)
direct ops.wvSplitK(...)	assert stub (not run on the box)	RuntimeError: wvSplitK is not built for gfx10 (RDNA2); use gemv_f16_rdna2; queue still alive: an LLMM1 call right after matches the reference (rel. error 4.5e-4)
registered _rocm_C ops	172	172, schemas identical

Binary: _rocm_C.abi3.so (gfx1030 only) 91,695,992 -> 91,699,064 bytes (+3,072).

Device code, six architectures (compile only; no RDNA3/4 or CDNA hardware on the box)

skinny_gemms.hip.o of rdna_extras vs this PR, one build tree and build dir with gfx90a;gfx942;gfx950;gfx1030;gfx1100;gfx1200. Each arch's code object is disassembled and compared per function. Two layout artefacts are tolerated and nothing else: PC-relative literals after s_getpc_b64, and the __LINE__ operand of assert(false) stubs / __assert_fail, which has to have moved by +18 or +24 lines (what this PR inserts above them). Opcodes, registers, other immediates, branch offsets and instruction counts must match exactly.

arch	functions	identical	identical modulo PC-relative / __LINE__	changed
gfx90a	478	400	78 (assert stubs)	0
gfx942	478	477	1 (__assert_fail)	0
gfx950	476	476 (also byte-identical .text, .rodata, .note)	0	0
gfx1030	478	2	468 (assert stubs)	8 LLGemm1_kernel
gfx1100	478	392	78 (assert stubs)	8 LLGemm1_kernel
gfx1200	478	455	15 (14 assert stubs, __assert_fail)	8 LLGemm1_kernel

CDNA (wave64) device code is unchanged, and on every wave32 arch only the eight LLGemm1_kernel instantiations (4 rows_per_block x fp16/bf16) differ: 4900 -> 5549 instructions on gfx1030, 4981 -> 5641 on gfx1100, 5286 -> 6060 on gfx1200 (+13 to +15 % over the eight kernels). On gfx1030, a whole-library fingerprint (64 objects) shows 63 identical and only skinny_gemms.hip.o changed.

Platform risk

• CDNA (gfx90a/942/950): none, device code unchanged. Not run on hardware.
• RDNA3/4: the only behaviour change is the intended one, correct LLMM1 results for K in (4096, 8192]. For K <= 4096 results are unchanged: bit-identical on gfx1030 (all 4096 cases above), as expected since num_warps <= 16 and the added branch is never taken. The source change is confined to the second-stage reduction tail, which runs once per block. Back-to-back LLMM1 launches on gfx1030 (m in {4, 8}, K 1024..8192, fp16/bf16, rows_per_block=4 as the dispatcher uses) are launch-bound at 8.3-8.5 us per call and identical within +-1.6 % (mean +0.05 %). No gfx1100/gfx1200 GPU was available, so the RDNA3/4 part rests on the gfx1030 run (same wave32 path, WARP_SIZE == 32) and the compile diff above; kernel time on RDNA3/4 was not measured. The gfx1100 item under "GPU validation to run" in the description above is still open.
• RDNA2: LLMM1 fixed (not routed by the dispatcher anyway); wvSplitK refuses.
• on_gfx10() caches getCurrentDeviceProperties() in a function-local static like the neighbouring on_gfx1x() helpers, so a mixed-architecture process whose first call is on a gfx10 device would cache true. Nothing in this fork calls wvSplitK on gfx10, so this is theoretical.
• The check covers wvSplitK only; wvSplitKrc and wvSplitKQ stay Python-gated.
• The serving A/B listed under "GPU validation to run" was not run: the dispatcher probe shows gfx1030 never reaches the changed kernels, and the device-code comparison shows nothing else changed.

Notes, not blocking

• docs/rdna/agent-rules.md (L33-34, "assert on the device") and the comment at vllm/model_executor/layers/utils.py:361-362 ("numerically wrong on gfx1030") describe LLMM1 on gfx10 as asserting / wrong. After this PR LLMM1 is numerically right there; it is still not routed, so the rule "never route to LLMM1 on gfx10" is unchanged. I left the wording alone.