[BlivionIaG]

Purpose

Opt-in W8A8 int8 (int8 weights x int8 activations) dense and MoE GEMMs for gfx1030 through v_dot4_i32_i8 (sdot4), the int8-weight sibling of the W4A8 sdot4 pair (w4a8_sdot4_rdna2.cu / moe_w4a8_rdna2.cu). Serves compressed-tensors W8A8 int8 checkpoints (symmetric per-channel weights, dynamic per-token activations) — the scheme curvedinf/int8-vllm runs its Qwen3.8-27B W8A8 recipe with (per-channel weights, round-to-nearest per-token activation quant, all large dense GEMMs in int8). Their GEMMs are AITER/CK MFMA kernels for MI100; gfx1030 has no MFMA, so this is the sdot4 equivalent. Default OFF: nothing changes unless VLLM_RDNA2_W8A8_SDOT4=1.

Attribution. The quantization scheme and the "which layers run int8" choices follow curvedinf/int8-vllm at commit c7fe970 (author curvedinf), read from its README.md, docs/recipes/README.md and INT8_AUDIT_RESULTS.md. No code was copied from that repository (its kernels are gfx908 CK/AITER and Triton). The commit carries an Inspired-by: trailer with the same pin; the kernel header and docs/rdna2/w8a8-wiring.md repeat it.

Kernels (csrc/rocm/w8a8_sdot4_rdna2.cuh, builtin-only like the W4A8 header)

• The activation side is the W4A8 one, unchanged: w4a8_act_quant_kernel (PER_GROUP, MT=8) writes per-(token, group) int8 + f32 scales.
• Weights are repacked once at load (rdna2_w8a8.pack_weight) into [K/4, N] int32: per 8-K chunk two dword rows per column in the act tile's kAPerm byte order, so a8.x pairs with row 2c, a8.y with row 2c+1, and one u32x4 load brings 4 columns x 4 K.
• mac_window<ROWS, AG>: w4a8_gemm_kernel's loop with the nibble split replaced by the second dword row; i32 accumulate per activation group, flush to f32 with the activation scale and the weight scale (fetched per group at k / w_group, so per-channel and per-group weight scales share the loop). No zero points.
• Dense w8a8_gemm_kernel<AG>: grid (ceil(N/1024), ceil(M/8), split_k); K walked in 256-wide windows staged through 2 KiB LDS (no LDS cap, no pick_split_k mirror). All M run here (no int8 fallback exists for this layout): below 256 blocks the host splits K into a zeroed fp32 scratch (fp32 atomics + one cast); prefill-sized M keeps plain fp16 stores.
• MoE moe_w8a8_gemm_kernel<BLOCK_M, AG, C_T>: moe_w4a8_gemm_kernel's grid, staging and epilogue (router weight in fp32, fp32 scratch atomics or packed fp16 CAS, output_topk fused reduce).

Ops w8a8_gemm_rdna2(a, w_packed, w_scales, a_group) -> fp16 [M, N] and moe_w8a8_gemm_rdna2(..., a_group, fp32_accum) (moe_w4a8's contract). Both TORCH_CHECK on ineligible shapes rather than falling back.

Wiring

• Dense: RDNA2Int8ScaledMMLinearKernel (scaled_mm/rdna2_w8a8_int8.py), first in _POSSIBLE_INT8_KERNELS[ROCM]; is_supported = env + gfx1030 + op built, can_implement declines static input scales and asymmetric activations. Packs the weight and expands per-tensor scales to per-channel (fused modules) at load.
• MoE: CompressedTensorsW8A8Int8RDNA2MoEMethod, selected in get_moe_method via rocm_moe_rdna.is_supported_w8a8_int8 before the generic int8 MoE method. Reuses the parent's checkpoint-layout create_weights, packs in process_weights_after_loading, apply mirrors the W8A16 RDNA2 method.
• docs/rdna2/w8a8-wiring.md (contract, design, validate steps), a tuning.md row, tools/rdna/bench_moe_w8a8.py (W8A8 vs W8A16 int8 MoE micro A/B).

Not a duplicate. The fork's int8 work is weight-only: VLLM_RDNA_DENSE_INT8 shadows + gemv_i8_rdna2 (#56, W8A16 decode GEMV), moe_w8a16_gemm_rdna2 (W8A16 MoE), and #71/#73 are int8 KV cache. No open or closed PR has an int8 x int8 GEMM; #9 (closed, W4A8 explore) listed "W8A8 INT8 sdot4" as a dependency that did not exist. No PR touches _POSSIBLE_INT8_KERNELS or the W8A8 int8 MoE method selection.

AI assistance. Written with AI assistance (Claude Code). A human submitter must review every changed line and run the gfx1030 steps below before this leaves draft.

Test Plan

Done here (no GPU, no hipcc in this sandbox):

# Host syntax check of the kernel header with a HIP-builtin shim (clang 18, -std=c++17 -fsyntax-only),
# explicit instantiations of w8a8_gemm_kernel<32|64|128>, moe_w8a8_gemm_kernel<8,128,float>,
# <1,32,f16>, <4,64,f16> and w4a8_act_quant_kernel<256,8,true>.
# NumPy emulation of the data path: pack_weight byte order x the W4A8 act tile (A_PERM) x sdot4 over
# dword rows, vs a plain per-(token, group) quantized reference, 4 shapes incl. a K tail and w_group == K.
ruff check / ruff format --check (0.14.0) on the new and edited Python files
clang-format 21.1.2 --dry-run on the new C++ files; ops.h / torch_bindings.cpp violation counts unchanged vs base
pre-commit run mypy-3.12 --hook-stage manual --files <new python files>

On gfx1030 (V620), to do before merge:

1. Build with PYTORCH_ROCM_ARCH=gfx1030; confirm _rocm_C::w8a8_gemm_rdna2 and _rocm_C::moe_w8a8_gemm_rdna2 in torch._C._jit_get_all_schemas().
2. pytest tests/kernels/quantization/test_rdna2_w8a8.py tests/kernels/quantization/test_rdna2_moe_w8a8.py -v (the three packing tests run on CPU with torch; the rest need the GPU).
3. ISA audit as for W4A8: v_dot4c_i32_i8 in the w8a8 symbols, sgpr_spill_count / vgpr_spill_count 0, no scratch.
4. python tools/rdna/bench_moe_w8a8.py — W8A8 vs the W8A16 int8 MoE kernel per token count, with the relative error.
5. In-model A/B on a compressed-tensors W8A8 int8 checkpoint, VLLM_RDNA2_W8A8_SDOT4=0/1: RDNA2 W8A8 sdot4 dense/MoE path active once per worker, greedy probes, PPL on/off, prefill and decode tok/s at 1k/512 and 16k/1k, c=1 and c=8.

Test Result

• Host syntax check: pass (post clang-format).
• NumPy data-path emulation: rel-L2 7e-8 … 2e-7 on all 4 shapes (m,k,n,ag,wg = 3/64/8/32/64, 9/256/12/128/256, 2/128/4/64/64, 1/96/8/32/96).
• ruff check: clean on all new/edited files; ruff format: clean. Lint/format counts on edited existing files unchanged vs rdna_extras.
• clang-format: new files clean; ops.h 40 pre-existing violations before and after, torch_bindings.cpp 2 before and after.
• mypy-3.12 hook: Passed on the new Python files.
• Not built, not run on hardware, no pytest run (torch is not installable in this sandbox: the PyTorch index is blocked). No model eval: the kernels are untested on silicon, so numbers would be fabricated. Items 1–5 above are the gate.

Design points a reviewer should weigh (all opt-in, none change a default): all-M dense serving (decode through split-K fp32 atomics instead of the T45 gemv_i8_rdna2 design), the 256-K LDS window instead of W4A8's whole-split staging, and no zero-point support (GPTQ-style int8 with qzeros stays on moe_w8a16_gemm_rdna2).

Pointers: what to explore and test once it builds

Roughly in the order the risk sits. Everything here is a hypothesis until the V620 says otherwise.

Correctness risks specific to this design

1. fp32 atomic lowering. The dense split-K path and the MoE fp32 scratch use __hip_atomic_fetch_add(float*). On gfx1030 this should lower to global_atomic_add_f32; if the ISA dump shows a CAS loop instead, decode will crawl. Check the .s (or llvm-objdump -d on _rocm_C.so) for global_atomic_add_f32 in the w8a8 symbols; the fix is -munsafe-fp-atomics on these two TUs or atomicAddNoRet, as moe_accum_rdna2.cuh relies on.
2. Graph capture at decode. Unlike W4A8 (prefill-only, never captured), the dense op runs under FULL cudagraph capture and allocates per call (at::empty for the act workspace, at::zeros for the fp32 scratch). PyTorch pools these inside the graph, but docs/rdna2/w4a8-moe-wiring.md records a suspected graph-aliasing issue from exactly this pattern. Test: FULL_AND_PIECEWISE serve, greedy output identical across replays and across batch sizes 1/2/4/8. If it misbehaves, move to a persistent per-(M, K, AG, device) workspace like moe_fp32_scratch.
3. K tails. The window loop handles K % 256 != 0 (tail window is a whole number of activation groups), but the pytest shapes are all multiples of 256. Add a case like K=1120 (a_group falls to 32) and K=2304 with a_group=128 (9 windows); the NumPy emulation covered a tail but the kernel's chunks < 32 staging branch has not run on silicon.
4. Strided activations. a.stride(0) % 8 == 0 is required (16-byte u32x4 loads in the act quant). The scaled_mm path reshapes x without .contiguous(); a sliced input with an odd row stride would TORCH_CHECK rather than silently miscompute — confirm which callers reach it (QKV splits, lm_head slices) and whether a .contiguous() is cheaper than the check firing.
5. MoE edge cases that the tests do not cover: EP with expert_map (expert_id == -1 early-out), apply_router_weight_on_input=True, non-gated activations (is_gated=False), top_k=1, and num_tokens just above the 16-token decode buffers.
6. fp16 output range. Group flushes are fp32 and the epilogue rounds once, so overflow can only come from the model itself (int8-vllm hit the fp16 residual stream at long context). Worth one long-context greedy run with VLLM_LAYER_NAN_DEBUG.

Performance questions (measure before tuning)

1. Decode: this vs gemv_i8_rdna2 (T45). The all-M design runs decode through the 8-row tile with 7 padding rows and split-K atomics; the fork's int8 decode GEMV (VLLM_RDNA_DENSE_INT8) streams rows wave-per-row with fp16 activations. At the Flash-Next dense shapes (GDN in/out proj, QSA qkv/o, shared expert, lm_head) time both at M = 1, 4, 8. If the GEMV wins, the options are routing M <= 8 to a W8A16 path (needs the [N, K] layout: a second copy, or an unpack-on-the-fly), or a dedicated decode kernel over the packed layout with a wave-level K split and LDS reduce (no atomics, deterministic).
2. Split-K heuristic. kTargetBlocks = 256, windows of 256 K. At M=1, N=5120, K=5120 the grid is 5 x 1 x 20. Sweep the target (128/256/512) and the window (128/256/512); the split only exists for occupancy, so look at achieved bandwidth vs the 26 MB weight read.
3. Prefill vs rocBLAS fp16 and vs W4A8. The W4A8 dense path gave +24–34 % prefill on 27B AWQ; W8A8 moves twice the weight bytes per sdot4, so expect a smaller margin. Find the M where W8A8 crosses the TunableOp rocBLAS row at the recipe shapes (M = 33 … 2048, 1k/16k chunks). Below the crossover the kernel class should hand M to rocBLAS on a dequantized weight, or the registry should keep Triton.
4. VGPR budget and occupancy. wx[4][4] + wy[4][4] are 16 VGPRs more than W4A8's wv[4][4]; the W4A8 MoE instantiations sat at 57–168 VGPRs. Read the counts from the ISA audit; if BLOCK_M = 8 lands above 128 (fewer than 4 waves/SIMD at 256 threads), try K_STEP = 16 (DW = 2) or NPT = 2 variants.
5. LDS window vs whole-split staging. Two barriers per 256 K (20 pairs at K = 5120) versus W4A8's single stage of the whole split. Try a 512-K window (4 KiB + scales) or double-buffered staging if the barrier cost shows in the profile.
6. The act quant kernel at M = 1 is one block of 256 threads walking every group of one row — latency-bound. A dedicated 1–8 row quant, or fusing the quant into the preceding RMSNorm epilogue (int8-vllm's fused AR + RMSNorm + int8-quant idea, their commit ec90fc933), removes a launch per layer at decode.
7. MoE tile thresholds. block_size_m is 1 / 4 / 8 at 4 / 512 tokens; reuse the VLLM_RDNA2_MOE_BSM_LARGE / _BSM_MIN_TOKENS knobs rather than the constants, and compare against both moe_w8a16_gemm_rdna2 (tools/rdna/bench_moe_w8a8.py) and the W4A8 MoE at equal token counts.

Accuracy

1. Activation group size. act_group(K) takes the largest of 128/64/32 dividing K. Smaller groups cost one more flush per group but localize outlier channels (llama.cpp Q8_1); int8-vllm quantizes per whole token. Measure PPL at AG = 32/64/128 against fp16 and against a per-token reference to see whether the extra scales buy anything.
2. Which layers stay fp16. int8-vllm keeps GDN in_proj_a/b, norms, conv, attention softmax/PV and the drafter fc in float and runs lm_head int8 behind a flag. Replicate with the compressed-tensors ignore list on a Flash-Next W8A8 export and A/B each exclusion; their one-shot GPTQ-int8 lost KLD 0.011 vs 0.0069 after retraining, so expect the checkpoint recipe to matter as much as the kernel.
3. A small checkpoint first. An llm-compressor W8A8 export of Qwen3-0.6B/1.7B quantizes in minutes and exercises both the dense kernel and (on a small MoE) the MoE method before the 27B-class run.
4. Determinism. VLLM_RDNA2_MOE_FP32_ACCUM (fp32 scratch) vs the packed-fp16 CAS epilogue for run-to-run stability, as documented for the other MoE kernels.

Follow-ups this PR deliberately leaves out

1. Zero points. GPTQ-int8 checkpoints (int8-vllm's own W8A8-GS128 exports are GPTQ) store uint8 + qzeros. The W4A8 zero fold (acc = -z * asum; asum is already produced by the act quant and currently ignored) carries over unchanged, plus a loader from the W8A16 MoE layout. The kernel already takes a runtime weight group, so GS128 scales need no kernel change.
2. Static input scales (compressed-tensors static W8A8): skip the absmax pass and take the scale as input; can_implement currently declines them.
3. int8 collectives. int8-vllm's custom_all_reduce_int8_gs32.hip / decode_int8_intermediates.hip (int8 payload all-reduce fused with residual + RMSNorm at TP4) is a separate exploration for rdna_ar ([backlog] perf(rdna): raise the one-shot rdna_ar gate 64 -> 128-160 KiB #60, [backlog] perf(rdna): reduce-scatter + allgather custom all-reduce (rdna_ars) #61).
4. int8 KV ([rdna2] INT8 KV cache for the RDNA2 backend: map the path, the measured read path, and int8 + FP16 block scales #71) pairs naturally with this once both exist.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results (host-only results above; gfx1030 results pending)
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. (docs/rdna2/w8a8-wiring.md, tuning.md)

🤖 Generated with Claude Code

https://claude.ai/code/session_0147zxyqwsKMmBLRn8xgBaHL