Skip to content

Commit aa9b730

Browse files
Phil-amdPhil-amd
committed
[Feature] Add architecture-general ISA resource diff tool
Compare per-kernel register, spill, scratch, and LDS usage between two FLYDSL_DUMP_IR dump directories or JSON snapshots, exposing resource regressions that functional tests do not surface. To work outside CDNA, read register counts from the per-kernel `.set <kernel>.num_vgpr`/`.num_agpr` symbols that LLVM emits on every AMDGPU target, rather than the CDNA-only `.agpr_count` metadata field, and count LDS traffic under both the `ds_read` and the gfx11+ `ds_load` spelling. Take the processor and the feature set from the target ID, normalizing the triple's environment field, which is spelled either empty or `unknown` for one and the same target. Report each metric as a value, as not applicable, or as unparsed, and exit 0, 1, or 2 for no regression, a regression, or an untrustworthy result. Fail closed on anything that would otherwise answer from a partial comparison: an unparsed or impossible metric, a dump file that does not parse or decode, a kernel entry with no identity or a duplicated one, and two sides whose targets are not provably the same. Covered by a backend-agnostic test over all three parser axes and exposed to agents as the `isa-resource-diff` skill.
1 parent 4a6f955 commit aa9b730

6 files changed

Lines changed: 1929 additions & 0 deletions

File tree

‎.claude/skills/isa-resource-diff/SKILL.md‎

Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
---
2+
name: isa-resource-diff
3+
description: >
4+
Detect per-kernel GPU resource regressions (VGPR, SGPR, register spills, scratch,
5+
static LDS) by diffing the final ISA before and after a change, using
6+
scripts/isa_resource_table.py. Compile-only: needs no GPU and no profiler run, so
7+
it works on any target the compiler supports and runs in seconds. Use when asked
8+
whether a change increased register pressure, caused spilling, or hurt occupancy,
9+
when reviewing a kernel change for resource impact, or as a fast pre-check before
10+
spending a profiling run.
11+
Usage: /isa-resource-diff [<test-or-command>] [--arch <gfx>]
12+
allowed-tools: Read Write Bash Grep Glob
13+
---
14+
15+
# ISA Resource Diff
16+
17+
Compare per-kernel register, spill, scratch, and LDS usage between two builds to
18+
catch resource regressions that functional tests do not surface.
19+
20+
## Pick the right skill first
21+
22+
| Question | Skill |
23+
|---|---|
24+
| Did my change increase registers / cause spills / grow LDS? | **this skill** (compile-only, seconds, no GPU) |
25+
| *Why* is this kernel slow — which instructions stall, and on what? | `/kernel-trace-analysis` (needs a GPU run + rocprofv3 ATT trace) |
26+
| Which commit made it slow? | `/bisect-perf-regression` (needs a runnable benchmark) |
27+
| How do I collect a trace at all? | `/capture-kernel-trace` |
28+
29+
This skill measures **resources, not time**. A clean result here does not mean
30+
performance is unchanged — it means register/LDS/spill pressure is unchanged.
31+
A regression here is a strong, cheap signal that is usually worth acting on
32+
before profiling, because spilling and occupancy cliffs dominate most kernel
33+
slowdowns. See §7 of `docs/kernel_tuning_guide.md` for what to do about one.
34+
35+
## Arguments
36+
37+
| Argument | Required | Description |
38+
|---|---|---|
39+
| `<TEST-OR-COMMAND>` | No | What to run to produce dumps. Defaults to asking the user. Example: `pytest tests/kernels/test_softmax.py -q` |
40+
| `--arch <gfx>` | No | Target for compile-only runs, e.g. `gfx950`. Omit to use local hardware |
41+
42+
If the user already has two dump directories or two JSON snapshots, skip to Step 3.
43+
44+
## Workflow
45+
46+
### Step 1 — Capture the "before" side
47+
48+
Check out or stash to the baseline state first, then:
49+
50+
```bash
51+
FLYDSL_DUMP_IR=1 FLYDSL_DUMP_DIR=/tmp/isa-before FLYDSL_RUNTIME_ENABLE_CACHE=0 \
52+
python3 -m pytest tests/kernels/test_softmax.py -q
53+
```
54+
55+
`FLYDSL_RUNTIME_ENABLE_CACHE=0` is **required, not optional** — see Pitfalls.
56+
57+
For a target without local hardware, add `ARCH=<gfx> COMPILE_ONLY=1`:
58+
59+
```bash
60+
ARCH=gfx950 COMPILE_ONLY=1 \
61+
FLYDSL_DUMP_IR=1 FLYDSL_DUMP_DIR=/tmp/isa-before FLYDSL_RUNTIME_ENABLE_CACHE=0 \
62+
python3 -m pytest tests/kernels/test_softmax.py -q
63+
```
64+
65+
### Step 2 — Capture the "after" side
66+
67+
Apply the change, then rerun **the identical command** into a *fresh* directory
68+
(`/tmp/isa-after`). Same test, same parameters, same arch, same cache setting.
69+
70+
### Step 3 — Diff
71+
72+
```bash
73+
python3 scripts/isa_resource_table.py diff /tmp/isa-before /tmp/isa-after
74+
```
75+
76+
The tool requires **Python 3.10+**. If `python3` is older it exits 2 with a clear
77+
message; use `python3.10 scripts/isa_resource_table.py …` instead.
78+
79+
Both sides may independently be a dump directory or a `.json` snapshot, so a
80+
baseline can be captured once and reused:
81+
82+
```bash
83+
python3 scripts/isa_resource_table.py summarize /tmp/isa-before --json baseline.json
84+
python3 scripts/isa_resource_table.py diff baseline.json /tmp/isa-after
85+
```
86+
87+
## Reading the output
88+
89+
```
90+
* = regression trigger; other columns are informational.
91+
vgpr = total (arch+acc, LLVM's occupancy number); arch_vgpr/agpr are its split -- do not add them.
92+
kernel *vgpr arch_vgpr agpr ... *lds_static_bytes lds_read ...
93+
-------------------------------------------------------------------------------------------------
94+
gemm::d128_fmha_fwd_kernel_0 942->960(+18) 942->960(+18) 0 ... 212992->229376(+16384) 12
95+
96+
compared 1 of 1 kernels; 0 unchanged; 1 changed; worsened: 2; improved: 0
97+
RESULT: REGRESSION
98+
```
99+
100+
Only changed and problematic kernels are printed. The last two stdout lines are
101+
always a count line and a `RESULT:` verdict that matches the exit code exactly.
102+
103+
**Columns marked `*` are regression triggers**; the rest are context.
104+
105+
| Column | Trigger | Read from | What it is |
106+
|---|---|---|---|
107+
| `vgpr` | yes | `.vgpr_count` metadata | Total VGPRs, arch + accumulator — LLVM's own occupancy number |
108+
| `arch_vgpr` | no | `.set` symbol `num_vgpr` | The arch half of that total |
109+
| `agpr` | no | `.set` symbol `num_agpr` | The accumulator half of that total |
110+
| `sgpr` | yes | `.sgpr_count` metadata | SGPRs including the fixed extras (VCC, XNACK, FLAT_SCRATCH) |
111+
| `numbered_sgpr` | no | `.set` symbol `numbered_sgpr` | SGPRs without those extras; tells a real increase from VCC becoming live |
112+
| `vgpr_spill` | yes | `.vgpr_spill_count` metadata | VGPRs the allocator spilled |
113+
| `sgpr_spill` | yes | `.sgpr_spill_count` metadata | SGPRs the allocator spilled |
114+
| `scratch_bytes` | yes | `.private_segment_fixed_size` metadata | Private segment per work-item; exact on every target |
115+
| `lds_static_bytes` | yes | `.group_segment_fixed_size` metadata | Statically allocated LDS per work-group |
116+
| `lds_read` / `lds_write` | no | instruction count | `ds_read`/`ds_load` and `ds_write`/`ds_store` sites |
117+
| `scratch_store` / `scratch_load` | no | instruction count | `scratch_*` sites; `n/a` where spilling goes through `buffer_*` |
118+
| `matrix_ops` | no | instruction count | MFMA / WMMA sites |
119+
120+
Three things are easy to misread:
121+
122+
- **Do not add `arch_vgpr` and `agpr` to `vgpr`.** `vgpr` is already the total
123+
(arch + accumulator) and is the only VGPR-family trigger. The other two are its
124+
split, shown so you can tell *which half* moved. Moving accumulators into AGPRs
125+
— which the tuning guide recommends — deliberately does not count as a regression.
126+
- **`n/a` is not `0`.** It means the quantity does not exist on this target, e.g.
127+
`scratch_store`/`scratch_load` on a target that spills through `buffer_*`. Use
128+
`scratch_bytes`, which is exact everywhere.
129+
- **`?` means unparsed** — the tool could not read something it reports. Any `?`
130+
forces exit 2. Never treat it as unchanged.
131+
132+
### Exit codes
133+
134+
| Code | stdout | Meaning | What an agent should do |
135+
|---|---|---|---|
136+
| `0` | `RESULT: OK` | Everything comparable, no trigger increased | Proceed |
137+
| `1` | `RESULT: REGRESSION` | Everything comparable, a trigger increased | Investigate — this is a real finding |
138+
| `2` | `RESULT: NOT TRUSTWORTHY` | The tool cannot answer | **Fix the inputs and rerun. Do not report "no regression"** |
139+
140+
Exit `1` is a claim about the code under test; exit `2` is a claim about the
141+
tool's own confidence. Everything that would leave the answer partial reports `2`
142+
and never `1`: a crash, an empty dump directory, a dump file that does not parse
143+
or does not decode, a metadata entry with no kernel identity or a duplicated one,
144+
a negative resource count, a kernel present on only one side, and a target that
145+
differs — or that the tool cannot name — on either side.
146+
147+
For scripting, `-q/--quiet` prints only the verdict line, and `--json PATH`
148+
writes the full comparison as machine-readable JSON:
149+
150+
```bash
151+
python3 scripts/isa_resource_table.py diff /tmp/isa-before /tmp/isa-after -q --json report.json
152+
case $? in
153+
0) echo "no resource regression" ;;
154+
1) echo "REGRESSION — see report.json" ;;
155+
*) echo "inconclusive — inputs are bad, do not claim a clean result" ;;
156+
esac
157+
```
158+
159+
## Acting on a regression
160+
161+
Map the column that moved to a cause, then follow `docs/kernel_tuning_guide.md`:
162+
163+
| Column increased | Usual cause |
164+
|---|---|
165+
| `vgpr` | More live values; larger tiles or deeper pipelining/unrolling |
166+
| `vgpr_spill` / `sgpr_spill` / `scratch_bytes` | Register pressure crossed the budget — normally the most damaging of these signals |
167+
| `lds_static_bytes` | Bigger shared tiles or added double buffering; may cross an occupancy step |
168+
| `sgpr` alone, by 2, with `numbered_sgpr` flat | Usually just VCC becoming live — rarely meaningful |
169+
170+
If a resource regression is confirmed but the kernel is not actually slower,
171+
say so rather than "fixing" it: these are proxies for occupancy, not timings.
172+
Confirm with `/kernel-trace-analysis` before reworking a kernel.
173+
174+
## Pitfalls
175+
176+
These silently produce a *confident wrong answer* if ignored:
177+
178+
- **Dump directories are keyed by kernel name only, with no specialization key.**
179+
Every JIT specialization of one kernel writes to the same directory and
180+
overwrites the previous one, so a parametrized test leaves only its last
181+
variant. Diff one shape at a time when the answer must be exact.
182+
- **A cache hit produces no dump at all.** Hence `FLYDSL_RUNTIME_ENABLE_CACHE=0`
183+
on both sides. Without it a kernel can silently vanish from one side, which
184+
the tool reports as `ONLY IN BEFORE` and exit 2.
185+
- **`lds_static_bytes` is static LDS only.** A kernel using
186+
`SharedAllocator(static=False)` reports `0` no matter how much LDS it takes at
187+
dispatch; the tool prints a `dynamic LDS in use` note for it. LDS regressions
188+
in those kernels are invisible here.
189+
- **The stage number in `NN_final_isa.s` varies between runs.** Nothing cleans the
190+
dump directory, so reusing one can leave two files side by side. The tool uses
191+
the highest-numbered one and warns; prefer a fresh directory per run.
192+
- **Diffs across targets are refused** (exit 2). Register files and LDS banking
193+
differ between architectures, and `xnack`/`sramecc` change code generation
194+
within one, so both sides must report the same processor *and* the same target
195+
features. A target the tool cannot name is refused for the same reason. The
196+
triple's environment field is normalized, so `amdgcn-amd-amdhsa--gfx942` and
197+
`amdgcn-amd-amdhsa-unknown-gfx942` are the same target.
198+
- **Warnings on stderr never change the exit code.** They describe the input tree
199+
and are worth reading before trusting a clean result.
200+
201+
## Verifying the tool itself
202+
203+
`tests/unit/test_isa_resource_table.py` is backend-agnostic and needs no build:
204+
205+
```bash
206+
python3 -m pytest tests/unit/test_isa_resource_table.py -q
207+
```
208+
209+
It generates its ISA input in `make_isa()` rather than checking dumps in, so the
210+
test file states exactly which parts of LLVM's output the parser relies on. The
211+
shapes cover both sides of each axis that has already broken it once: whether the
212+
target emits `.agpr_count`, which LDS mnemonic spelling it uses, and whether the
213+
target ID spells the triple's environment as empty or as `unknown`. The rest of
214+
the file pins the fail-closed verdicts: a mismatched processor, a mismatched
215+
feature set, an unnameable target, an unreadable or undecodable dump file, a
216+
kernel entry with no identity or a duplicated one, and a negative count all have
217+
to reach exit 2.
218+
219+
If the tool reports `?` on a dump that looks healthy, LLVM's assembly format has
220+
probably drifted — update `make_isa()` to match the new shape rather than
221+
loosening the parser, since the failure is deliberately loud.

‎.claude/skills/kernel-trace-analysis/SKILL.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,11 @@ an optimization plan.
1818
All analysis is done programmatically via `hotspot_analyzer.py` + `code.json`.
1919
Do **not** use GUI tools.
2020

21+
> If the question is only "did my change increase register pressure, spills, or
22+
> LDS?", use `/isa-resource-diff` first — it is compile-only, needs no GPU or
23+
> profiler run, and answers in seconds. Come here when you need to know *why* a
24+
> kernel is slow rather than *what resources it uses*.
25+
2126
## Arguments
2227

2328
| Argument | Description |

‎docs/kernel_tuning_guide.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -445,6 +445,29 @@ WHERE ks.KernelName LIKE '%target_kernel%' LIMIT 5;
445445
the binding limiter, so you know whether to cut VGPR, shrink LDS per block, or
446446
reduce SGPR pressure.
447447

448+
For a lightweight before/after check without collecting a profiler trace, dump
449+
the final ISA of each build and diff the per-kernel **total VGPR (with its
450+
arch/accumulator split), SGPR, register spills, scratch bytes, and static LDS**
451+
with `scripts/isa_resource_table.py`. It is compile-only, so it needs no GPU:
452+
453+
```bash
454+
FLYDSL_DUMP_IR=1 FLYDSL_DUMP_DIR=/tmp/isa-before FLYDSL_RUNTIME_ENABLE_CACHE=0 \
455+
python3 -m pytest tests/kernels/test_softmax.py -q # repeat for /tmp/isa-after
456+
python3 scripts/isa_resource_table.py diff /tmp/isa-before /tmp/isa-after
457+
```
458+
459+
It exits `0` for no regression, `1` for one, and `2` when it cannot produce a
460+
trustworthy answer — a crash never reports as `1`. The table marks regression
461+
triggers with `*` and prints a legend above itself; `/isa-resource-diff` carries
462+
the full column reference. Set `FLYDSL_RUNTIME_ENABLE_CACHE=0` on both runs: a
463+
cache hit emits no dump at all, which would silently drop a kernel from one side.
464+
465+
Because arch VGPRs and accumulator VGPRs share one register budget on targets
466+
with a unified register file (gfx90a and later MFMA-capable parts), that
467+
tool treats the **total** as the regression signal and reports `arch_vgpr` and
468+
`agpr` alongside it for information only. Moving accumulators into AGPRs, as
469+
recommended below, therefore does **not** register as a resource regression.
470+
448471
**Do not** use `maxnreg` to force `accum_vgpr=0` — it spills MFMA results through
449472
arch_vgpr via `v_accvgpr_read` (measured ~4.5× regression).
450473

‎docs/testing_benchmarking_guide.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -395,6 +395,7 @@ bash scripts/dumpir.sh
395395
| `scripts/run_tests.sh` | Full test runner (pytest + examples + FileCheck) |
396396
| `scripts/run_benchmark.sh` | Benchmark harness with configurable shapes |
397397
| `scripts/dumpir.sh` | IR dump helper script |
398+
| `scripts/isa_resource_table.py` | Diff per-kernel ISA resource usage (compile-only) |
398399
| `tests/conftest.py` | Pytest fixtures (MLIR context, module, insert point) |
399400
| `tests/test_common.py` | `perftest()`, `checkAllclose()`, `verify_output()` |
400401
| `tests/utils.py` | `pertoken_quant()`, `shuffle_weight()` |

0 commit comments

Comments
 (0)