Skip to content

Commit f27bef0

Browse files
sangeeta0201claude
sangeeta0201
and
claude
committed
Add the GPU optimization skills used for the 1.93 -> 1.84 ms decode work
.claude/skills/: optimize-megakernel the evidence-driven loop itself -- freeze baseline, localize, read final ISA, write one falsifiable hypothesis, assembly gate, correctness gate, A/B. Its references/gpt-oss-gfx950.md is rewritten for this tree: source map, the JIT-compile consequence, flag conventions (_opt vs opt-in, _BS1_ONLY_OPTS, MPK_FORWARD_VARS), the codegen gate, the genco + unbundle + objdump recipe, the s_nop-between-MFMA histogram, and the perplexity gate. cdna4-expert gfx950 hardware overview, memory/coherency/cache-flag reference with worked HIP+asm examples, and the full CDNA4 ISA manual under .claude/references/. dpp-row-operations replacing ds_bpermute shuffles with DPP/permlane cross-lane VALU ops -- this is what MPK_ROUTER_DUAL_REDUCE is. phase-profiling localizing a stage inside a fused kernel. pc-sampling finding the limiting stall when phase timing says where but not why. thread-buffer-audit auditing threadIdx-indexed buffers smaller than the block. gpu-fault-debugging rocgdb to the faulting source line before theorizing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent ff46bd4 commit f27bef0

25 files changed

Lines changed: 31403 additions & 0 deletions

File tree

‎.claude/references/gpu-architectures/cdna4/hardware-overview.md‎

Lines changed: 266 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,266 @@
1+
# AMD CDNA4 (gfx950 / MI350 Series) Architecture Reference
2+
3+
*Source: CDNA4 Instruction Set Architecture Reference Guide (5-August-2025), and AMD product documentation (CU counts, HBM, clocks).*
4+
5+
For cross-architecture comparison tables (CDNA3 vs CDNA4 vs RDNA3 vs RDNA4) see [`comparison.md`](../comparison.md).
6+
7+
---
8+
9+
## Registers
10+
11+
| Register Type | Count | Width | Notes |
12+
|---|---|---|---|
13+
| VGPR (V0–V255) | 256 | 32-bit | Allocated in groups of 8 Dwords |
14+
| AGPR (AV0–AV255) | 256 | 32-bit | Exclusive to matrix core unit |
15+
| VGPR + AGPR combined | 512 max | 32-bit | Flexible split |
16+
| SGPR (S0–S103) | 104 | 32-bit | Allocated 16–102 per wavefront, in units of 16 |
17+
| VCC | 2 SGPRs | 64-bit | Physically SGPR 106–107 |
18+
| TTMP0–TTMP15 | 16 | 32-bit | Trap temporary SGPRs (with trap handler) |
19+
20+
## Memory
21+
22+
| Resource | Size | Organization |
23+
|---|---|---|
24+
| LDS per CU | **160 KB** | 64 banks × 640 entries (4 bytes each) |
25+
| LDS atomic units | 32 integer | Built-in fast unordered atomics |
26+
| LDS allocation granularity | 1280-byte blocks | 1280-byte aligned |
27+
28+
## Execution
29+
30+
| Parameter | Value |
31+
|---|---|
32+
| Wavefront size | 64 work-items |
33+
| EXEC mask | 64-bit |
34+
| Max wavefronts per workgroup | 16 |
35+
| Max work-items per workgroup | 1024 |
36+
37+
## CU Count (product-level, not in ISA manual)
38+
39+
<!-- AUTOGEN:gpu-table id=cdna4_cu_count -->
40+
| Product | CUs | HBM |
41+
|---|---:|---|
42+
| AMD Instinct MI350X | 256 | 288 GB HBM3e |
43+
| AMD Instinct MI355X | 256 | 288 GB HBM3e |
44+
<!-- /AUTOGEN -->
45+
46+
*MI350X / MI355X both use 8 XCDs with 32 active CUs/XCD (physical 36, 4/XCD disabled for yield) = 256 active CUs total. Per AMD product pages and the MI350X / MI355X GPU brochures. MI355X differs only in clock (2.4 GHz vs 2.2 GHz) and TBP (1400W vs 1000W); compute peaks scale ~ +8% accordingly.*
47+
48+
## In-flight Instruction Counters (S_WAITCNT)
49+
50+
| Counter | Bits | Max in-flight | Tracks |
51+
|---|---|---|---|
52+
| VMCNT | 6 | 63 | Vector memory (global/buffer) loads and stores |
53+
| LGKMCNT | 4 | 15 | LDS, scalar memory (SMEM), messages (no GDS in CDNA4) |
54+
| EXPCNT | 3 | 7 | Unused on CDNA4 |
55+
56+
`S_WAITCNT` stalls the wavefront until all tracked counters reach **≤** the specified value.
57+
58+
## Cache Line Size
59+
60+
Scalar L0 data cache uses **64-byte cache lines** (per ISA manual). Vector/global memory cache line assumed 64 bytes.
61+
62+
## MFMA + VALU Concurrency
63+
64+
MFMA instructions are long-latency but the pipeline allows issuing VALU instructions while an MFMA is in-flight. Key rules:
65+
66+
- **XDLOPs** (CDNA4 term for matrix math on I8/F16/BF16) support back-to-back SrcC (accumulator) forwarding with **0 wait states** when the same opcode is chained.
67+
- **Non-XDLOP VALU writing a VGPR** read by subsequent MFMA SrcA/B requires **2 wait cycles**.
68+
- Reading an MFMA output VGPR with a VMEM/LDS instruction overlapped with the first MFMA output Dword requires **4 wait cycles**.
69+
70+
## MFMA Instructions
71+
72+
**Dense MFMA:**
73+
74+
| Instruction | Dimensions | Blocks | Cycles | FLOPs/wave-inst | Input Type |
75+
|---|---|---|---|---:|---|
76+
| V_MFMA_F32_32x32x1_2B | 32×32×1 | 2 | 64 | 4096 | F32 |
77+
| V_MFMA_F32_16x16x1_4B | 16×16×1 | 4 | 32 | 2048 | F32 |
78+
| V_MFMA_F32_4x4x1_16B | 4×4×1 | 16 | 8 | 512 | F32 |
79+
| V_MFMA_F32_32x32x2 | 32×32×2 | 1 | 64 | 4096 | F32 |
80+
| V_MFMA_F32_16x16x4 | 16×16×4 | 1 | 32 | 2048 | F32 |
81+
| V_MFMA_F32_32x32x4_2B | 32×32×4 | 2 | 64 | 16384 | F16 |
82+
| V_MFMA_F32_16x16x4_4B | 16×16×4 | 4 | 32 | 8192 | F16 |
83+
| V_MFMA_F32_4x4x4_16B | 4×4×4 | 16 | 8 | 2048 | F16 |
84+
| V_MFMA_F32_32x32x8 | 32×32×8 | 1 | 32 | 16384 | F16 |
85+
| V_MFMA_F32_16x16x16 | 16×16×16 | 1 | 16 | 8192 | F16 |
86+
| V_MFMA_F32_32x32x4_2B | 32×32×4 | 2 | 64 | 16384 | BF16 |
87+
| V_MFMA_F32_16x16x4_4B | 16×16×4 | 4 | 32 | 8192 | BF16 |
88+
| V_MFMA_F32_4x4x4_16B | 4×4×4 | 16 | 8 | 2048 | BF16 |
89+
| V_MFMA_F32_32x32x8 | 32×32×8 | 1 | 32 | 16384 | BF16 |
90+
| V_MFMA_F32_16x16x16 | 16×16×16 | 1 | 16 | 8192 | BF16 |
91+
| V_MFMA_I32_32x32x4_2B | 32×32×4 | 2 | 64 | 16384 | I8 |
92+
| V_MFMA_I32_16x16x4_4B | 16×16×4 | 4 | 32 | 8192 | I8 |
93+
| V_MFMA_I32_4x4x4_16B | 4×4×4 | 16 | 8 | 2048 | I8 |
94+
| V_MFMA_I32_32x32x16 | 32×32×16 | 1 | 32 | 32768 | I8 |
95+
| V_MFMA_I32_16x16x32 | 16×16×32 | 1 | 16 | 16384 | I8 |
96+
| V_MFMA_F64_16x16x4 | 16×16×4 | 1 | **64** | 2048 | F64 |
97+
| V_MFMA_F64_4x4x4_4B | 4×4×4 | 4 | **32** | 512 | F64 |
98+
| V_MFMA_F32_16x16x32 | 16×16×32 | 1 | 16 | 16384 | BF8/FP8 |
99+
| V_MFMA_F32_32x32x16 | 32×32×16 | 1 | 32 | 32768 | BF8/FP8 |
100+
| V_MFMA_F32_16x16x32_BF16 | 16×16×32 | 1 | 16 | 16384 | BF16 *(CDNA4-new)* |
101+
| V_MFMA_F32_32x32x16_F16 | 32×32×16 | 1 | 32 | 32768 | F16 *(CDNA4-new)* |
102+
| V_MFMA_I32_16x16x64_I8 | 16×16×64 | 1 | 16 | 32768 | I8 *(CDNA4-new)* |
103+
| V_MFMA_I32_32x32x32_I8 | 32×32×32 | 1 | 32 | 65536 | I8 *(CDNA4-new)* |
104+
| V_MFMA_F32_16x16x128_F8F6F4 | 16×16×128 | 1 | 16 (FP4/FP6) / 32 (FP8) | 65536 | FP4/FP6/FP8 mixed |
105+
| V_MFMA_F32_32x32x64_F8F6F4 | 32×32×64 | 1 | 32 (FP4/FP6) / 64 (FP8) | 131072 | FP4/FP6/FP8 mixed |
106+
| V_MFMA_SCALE_F32_16x16x128 | 16×16×128 | 1 | 16 (FP4/FP6) / 32 (FP8) | 65536 | F4/F6/F8 with block scaling |
107+
| V_MFMA_SCALE_F32_32x32x64 | 32×32×64 | 1 | 32 (FP4/FP6) / 64 (FP8) | 131072 | F4/F6/F8 with block scaling |
108+
109+
Note: F8F6F4 instructions take the longer cycle count when any operand is FP8; the shorter count otherwise.
110+
111+
**Block scaling (V_MFMA_SCALE_*) details:**
112+
- Scale factor encoding: 8-bit exponent, bias 127; valid range −127 to +127 (0xFF = NaN)
113+
- Block size: **32 elements** per scale factor
114+
- Instructions are encoded as 4-DWORD (128-bit) instructions; scale factors are loaded and applied inline with the MFMA
115+
116+
**Sparse MFMA (V_SMFMAC, 2:4 structured sparsity):**
117+
118+
*Inherited from CDNA3:*
119+
120+
| Instruction | Dimensions | Blocks | Cycles | FLOPs/wave-inst | Input Type |
121+
|---|---|---|---|---:|---|
122+
| V_SMFMAC_F32_16x16x32 | 16×16×32 | 1 | 16 | 16384 | F16 |
123+
| V_SMFMAC_F32_32x32x16 | 32×32×16 | 1 | 32 | 32768 | F16 |
124+
| V_SMFMAC_F32_16x16x32 | 16×16×32 | 1 | 16 | 16384 | BF16 |
125+
| V_SMFMAC_F32_32x32x16 | 32×32×16 | 1 | 32 | 32768 | BF16 |
126+
| V_SMFMAC_I32_16x16x64 | 16×16×64 | 1 | 16 | 32768 | I8 |
127+
| V_SMFMAC_I32_32x32x32 | 32×32×32 | 1 | 32 | 65536 | I8 |
128+
| V_SMFMAC_F32_16x16x64 | 16×16×64 | 1 | 16 | 32768 | BF8/FP8 |
129+
| V_SMFMAC_F32_32x32x32 | 32×32×32 | 1 | 32 | 65536 | BF8/FP8 |
130+
131+
*New in CDNA4 (doubled K-dimensions):*
132+
133+
| Instruction | Dimensions | Blocks | Cycles | FLOPs/wave-inst | Input Type |
134+
|---|---|---|---|---:|---|
135+
| V_SMFMAC_F32_16x16x64 | 16×16×64 | 1 | 16 | 32768 | F16 |
136+
| V_SMFMAC_F32_32x32x32 | 32×32×32 | 1 | 32 | 65536 | F16 |
137+
| V_SMFMAC_F32_16x16x64 | 16×16×64 | 1 | 16 | 32768 | BF16 |
138+
| V_SMFMAC_F32_32x32x32 | 32×32×32 | 1 | 32 | 65536 | BF16 |
139+
| V_SMFMAC_I32_16x16x128 | 16×16×128 | 1 | 16 | 65536 | I8 |
140+
| V_SMFMAC_I32_32x32x64 | 32×32×64 | 1 | 32 | 131072 | I8 |
141+
| V_SMFMAC_F32_16x16x128 | 16×16×128 | 1 | 16 | 65536 | BF8/FP8 (4 combinations) |
142+
| V_SMFMAC_F32_32x32x64 | 32×32×64 | 1 | 32 | 131072 | BF8/FP8 (4 combinations) |
143+
144+
**MFMA dependency hazards (minimum waits required):**
145+
146+
| Producing instruction | Consuming instruction | Min waits |
147+
|---|---|---|
148+
| Non-XDLOP VALU write VGPR | V_MFMA* read SrcA/B | 2 |
149+
| Same XDLOP opcode chain (SrcC) | Next XDLOP read SrcC | **0** (forwarded; **2** for 2-pass 4x4x4 XDL) |
150+
| XDLOP write SrcA/B | Next XDLOP read SrcA/B | 3 |
151+
| **XDL waits by size:** | | |
152+
| V_MFMA_*_4x4x4 (2-pass XDL) write | XDL/MFMA read SrcA/B | 5 |
153+
| V_MFMA_*_16x16x16 (4-pass XDL) write | XDL/MFMA read SrcA/B | 8 |
154+
| V_MFMA_*_32x32x8 (8-pass XDL) write | Any VGPR read (RAW+WAW) | 12 |
155+
| V_MFMA_F32_32x32x4_2B (16-pass XDL) write | Memory/VALU read VGPR | 20 |
156+
| **SGEMM (F32-input) waits by size:** | | |
157+
| V_MFMA_F32_4x4x1_16B (2-pass SGEMM) write | SGEMM/XDL read SrcA/B | 4 |
158+
| V_MFMA_F32_16x16x4 (4-pass SGEMM) write | SGEMM/XDL read SrcA/B | 6 |
159+
| V_MFMA_F32_32x32x2 (8-pass SGEMM) write | Any VGPR read (RAW+WAW) | 10 |
160+
| V_MFMA_F32_32x32x1_2B (16-pass SGEMM) write | Memory/VALU read VGPR | 18 |
161+
| **DGEMM (F64) waits:** | | |
162+
| V_MFMA_F64_16x16x4 write | MFMA read SrcA/B | 19 |
163+
| V_MFMA_F64_16x16x4 write | VM/LDS/FLAT/Export read overlapped | 20 |
164+
| V_CMPX write EXEC | V_MFMA | 4 |
165+
166+
**MFMA properties:**
167+
- Matrix core primitive: 4×1 outer product yielding 16 outputs
168+
- Ignores MODE denorm, rounding, and EXEC mask (all lanes always active) — **except** `V_MFMA_F32_*_F32` (F32 inputs), which honor MODE denorm-handling flags
169+
- Clamp supported via FP16_OVFL bit
170+
- Block scaling supported (V_MFMA_SCALE_*) for dynamic range extension
171+
- Inputs A/B from VGPR only; accumulator can be inline constant
172+
173+
---
174+
175+
## Parameters not in the ISA manual (CDNA4 only)
176+
177+
The values below are critical for kernel optimization but are defined in product/microarchitecture specs, not the ISA manual. Cross-architecture comparison tables for these parameters are in [`comparison.md`](../comparison.md).
178+
179+
### Occupancy Limits (per CU, MI350X)
180+
181+
| Parameter | Value |
182+
|---|---|
183+
| Max wavefronts per CU | 32 |
184+
| Max workgroups per CU | 8 |
185+
| Max VGPRs allocated per CU | 512 × 32 = 16384 |
186+
187+
**VGPR count → max wavefronts per CU** (32-wave hardware cap, identical to CDNA3):
188+
189+
| VGPRs per wave | Max wavefronts/CU |
190+
|---|---|
191+
| 0–16 | 32 |
192+
| 17–32 | 32 |
193+
| 33–64 | 16 |
194+
| 65–96 | 12 |
195+
| 97–128 | 8 |
196+
| 129–192 | 6 |
197+
| 193–256 | 4 |
198+
| 257–512 (with AGPRs) | 2 |
199+
200+
**LDS bytes per workgroup → max workgroups per CU** (CDNA4, **160 KB** LDS):
201+
202+
| LDS per workgroup | Max WGs/CU |
203+
|---|---|
204+
| 0 | 8 |
205+
| ≤8 KB | 8 |
206+
| ≤16 KB | 8 |
207+
| ≤32 KB | 4 |
208+
| ≤64 KB | 2 |
209+
| ≤80 KB | 2 |
210+
| ≤160 KB | 1 |
211+
212+
*Occupancy is the minimum of the wavefront limit (from VGPRs), workgroup limit (from LDS), and the hardware max. Use `rocminfo` or `hipOccupancyMaxActiveBlocksPerMultiprocessor` to compute this at runtime.*
213+
214+
### Cache Sizes
215+
216+
| Cache | MI350X / MI355X |
217+
|---|---|
218+
| L1 vector cache per CU | 32 KB |
219+
| L2 cache per XCD | 4 MB |
220+
| Total L2 (8 XCDs) | ~32 MB |
221+
| Scalar L0 cache per CU | 16 KB |
222+
| Infinity Cache (LLC, memory-side) | **256 MB** |
223+
| HBM memory | 288 GB HBM3e |
224+
| HBM bandwidth | ~8.0 TB/s |
225+
226+
*MI350X / MI355X carry the same 8-XCD topology as the MI300 series. Cross-XCD L2 access goes through Infinity Fabric.*
227+
228+
### Global Memory Latency (approximate)
229+
230+
| Level | Latency (cycles) |
231+
|---|---|
232+
| L1 vector cache hit | ~20–40 |
233+
| L2 cache hit | ~100–200 |
234+
| HBM (L2 miss) | ~400–700 |
235+
| Cross-XCD | higher than local L2 |
236+
237+
*Covering HBM latency requires ~10–22 in-flight wavefronts per CU at typical memory-bound occupancy.*
238+
239+
### VALU Throughput (per CU per clock, CDNA4)
240+
241+
| Data type | Ops/cycle/CU |
242+
|---|---|
243+
| FP32 (V_ADD_F32) | 64 |
244+
| FP32 FMA (V_FMA_F32) | 128 |
245+
| FP16 / BF16 packed (V_PK_FMA_F16) | 256 |
246+
| FP64 (V_ADD_F64) | 64 |
247+
| INT32 | 64 |
248+
| INT8 packed (V_DOT4_I32_I8) | 512 |
249+
250+
*Ops/cycle counts each multiply and each add as a separate operation. Vector throughput is unchanged from CDNA3; the doubling on CDNA4 is at the matrix engine (MFMA), not the vector engine.*
251+
252+
### Reference SKU peak summary
253+
254+
<!-- AUTOGEN:gpu-table id=cdna4_peak_summary -->
255+
| Metric | MI350X | MI355X |
256+
|---|---:|---:|
257+
| FP16 matrix peak | 2300 TF | 2510 TF |
258+
| BF16 matrix peak | 2300 TF | 2510 TF |
259+
| FP8 matrix peak | 4600 TF | 5020 TF |
260+
| INT8 matrix peak | 4600 TOPS | 5020 TOPS |
261+
| MXFP4 matrix peak | 9200 TF | 10040 TF |
262+
| Memory bandwidth | ~8.0 TB/s | ~8.0 TB/s |
263+
| FP16 ridge (peak / BW) | ~288 FLOP/B | ~314 FLOP/B |
264+
| FP8 ridge | ~575 FLOP/B | ~628 FLOP/B |
265+
| MXFP4 ridge | ~1150 FLOP/B | ~1255 FLOP/B |
266+
<!-- /AUTOGEN -->

0 commit comments

Comments
 (0)