Skip to content

Commit 4283f11

Browse files
morrison-turnanskymorrison-turnansky
authored
[vllm-triage] Transition vllm-pytorch-ci-triage skill for automated workflow (#8539)
updated vllm-pytorch-ci-triage skill to be consistent with cron job style workflow. I generally let the prompt serve as the source of truth on this and changed the skill accordingly.
1 parent 404854f commit 4283f11

4 files changed

Lines changed: 86 additions & 380 deletions

File tree

‎.claude/skills/vllm-pytorch-ci-triage/CONFIDENCE.md‎

Lines changed: 12 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -2,30 +2,24 @@
22

33
## classification_confidence (per group)
44

5-
| Score | Meaning |
5+
| Level | Meaning |
66
|---|---|
7-
| 5 | Exact match to a cheat-sheet entry |
8-
| 4 | Strong pattern match (same exception class + framework frames) |
9-
| 3 | Reasonable inference from exception pattern |
10-
| 2 | Weak signal — exception is generic, routing based on context |
11-
| 1 | Guessing — no clear pattern match |
7+
| high | Exact match to a cheat-sheet entry, or strong pattern match (same exception class + framework frames) |
8+
| med | Reasonable inference from the exception pattern |
9+
| low | Weak or generic signal — routing based on context, or guessing with no clear pattern match |
1210

1311
## new_failure_confidence (per group)
1412

15-
| Score | Meaning |
13+
| Level | Meaning |
1614
|---|---|
17-
| 5 | Clearly a regression signature with no known variant |
18-
| 4 | Likely new — signature is distinctive |
19-
| 3 | Plausible but could be a flake or known issue variant |
20-
| 2 | Weak — generic exception that could appear in many contexts |
21-
| 1 | Likely a variant of an existing known issue |
15+
| high | Clearly a regression signature with no known variant; distinctive |
16+
| med | Plausible but could be a flake or a known-issue variant |
17+
| low | Weak, generic exception, or likely a variant of an existing known issue |
2218

2319
## shared_root_cause_confidence (per failure)
2420

25-
| Score | Meaning |
21+
| Level | Meaning |
2622
|---|---|
27-
| 5 | Identical exception signature to the group's root cause |
28-
| 4 | Same exception class + closely related message |
29-
| 3 | Same exception class, different message but likely related |
30-
| 2 | Different exception but plausibly the same underlying bug |
31-
| 1 | Grouped only by job proximity, low certainty |
23+
| high | Identical exception signature to the group, or same exception class + closely related message |
24+
| med | Same exception class, different message but likely related |
25+
| low | Different exception but plausibly the same bug, or grouped only by job proximity |

‎.claude/skills/vllm-pytorch-ci-triage/ROUTING.md‎

Lines changed: 38 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,43 @@
11
# Repo Routing Cheat-Sheet
22

3-
Match exception patterns to repo. When multiple patterns match, prefer
4-
the more specific one.
3+
Match exception patterns to a repo. When multiple patterns match, prefer the more
4+
specific one.
55

6-
If no pattern exists, give your best guess based on your repo knowledge.
6+
The **Routing** column is always exactly one of the three canonical values, matching
7+
the `routing` field the triage workflow emits:
78

9+
"pytorch/pytorch" | "vllm-project/vllm" | "infra"
810

9-
| Error pattern | Repo |
10-
|---|---|
11-
| **Import errors — route by source package** | |
12-
| `ImportError` / `ModuleNotFoundError` from `torch.*` or `torch._inductor.*` | pytorch/pytorch |
13-
| `ImportError` / `ModuleNotFoundError` from `triton.*` | pytorch/pytorch (triton) |
14-
| `ImportError` / `ModuleNotFoundError` from `vllm.*` | vllm-project/vllm |
15-
| Import errors wrapped inside `RuntimeError: Engine core initialization failed` — unwrap to find the real `ImportError` underneath before routing | *(use rules above)* |
16-
| `torch.library.Library.impl ... already a kernel registered` | pytorch/pytorch |
17-
| `MetaProxy` in `prims.*` / Inductor | pytorch/pytorch |
18-
| `PassManager::run failed` inside `triton/` frames | pytorch/pytorch (triton) |
19-
| `Pointer argument cannot be accessed from Triton` | pytorch/pytorch (triton) |
20-
| `Cannot access data pointer of Tensor (FakeTensor…)` | pytorch/pytorch (AOTAutograd) |
21-
| `_pickle.PicklingError` on triton `launcher` | pytorch/pytorch (triton + AOT cache) |
22-
| `warm_artifacts_saved: got 0`, `KeyError: None` in standalone_compile | pytorch/pytorch (Inductor cache) |
23-
| `assert 'no' == 'yes'` in `test_dynamic_shapes_compilation` | pytorch/pytorch (Dynamo), but rerun first if GPU was OOM |
24-
| `torch.compile with fullgraph=True found no compiled frames` (when `TORCH_COMPILE_DISABLE=1` is in env) | vLLM-side fix usually correct; upstream interest if behavior change is intentional |
25-
| `RayChannelTimeoutError` on tp≥2 ray | pytorch/pytorch — likely torch.compile per-worker latency exceeds Ray channel timeout |
26-
| `Nondeterministic outputs detected` (B200-only) | pytorch/pytorch — Blackwell-specific kernel drift |
27-
| `assert torch.allclose(golden_output, vllm_output)` reward/PRM | pytorch/pytorch — numerical drift from triton update |
28-
| `compare_two_settings(... cpu-offload-gb ...)` → "Results are not the same" | pytorch/pytorch (CPU↔GPU dequantize parity) |
29-
| GSM8K accuracy collapses to 0.000 (not just degrades) | pytorch/pytorch — likely worker-side crash hidden behind unpickle error |
30-
| `Generated text "X" doesn't match expected pattern "Y"` on Qwen2-VL / Qwen3-VL LoRA | pytorch/pytorch — multimodal LoRA path numerical drift |
31-
| `AssertionError: expected size N==N, stride A==B` + `torch.ops.vllm.<X>` + "incorrect fake kernel" | **vllm-project/vllm** — fake kernel returns wrong shape |
32-
| Multi-modal per-model assertions (qwen2_vl, chameleon) | vllm-project/vllm first — may be torch-side once isolated |
33-
| Responses API assertion (`'incomplete' == 'completed'`) | vllm-project/vllm |
34-
| `test_lm_eval_accuracy_v1_engine` — measured below threshold | investigate both — often numerical drift from triton update |
35-
| `ValueError: Free memory on device cuda:N (X/Y GiB) … less than desired` (tagged `test_is_infra`) | infra (GPU contention) — rerun the job, do not file; can cascade dozens of unrelated tests, so the real failures may be a subset |
36-
| CUDA OOM (`torch.OutOfMemoryError` / `CUDA out of memory`) in tp≥2 or B200 fusion tests (runner had ~4–5 GiB free at start) | infra likely — **not** tagged `test_is_infra` (runtime OOM ≠ startup free-memory check); cross-check the same job on the same-day main build. If main OOMs the same way, it's contention — skip filing |
11+
The **Notes** column carries the subsystem/area and any caveats — it is context for
12+
the write-up, never a routing value. If no pattern matches, give your best guess from
13+
your repo knowledge, still using one of the three values.
14+
15+
16+
| Error pattern | Routing | Notes |
17+
|---|---|---|
18+
| **Import errors — route by source package** | | |
19+
| `ImportError` / `ModuleNotFoundError` from `torch.*` or `torch._inductor.*` | pytorch/pytorch | |
20+
| `ImportError` / `ModuleNotFoundError` from `triton.*` | pytorch/pytorch | triton |
21+
| `ImportError` / `ModuleNotFoundError` from `vllm.*` | vllm-project/vllm | |
22+
| Import errors wrapped inside `RuntimeError: Engine core initialization failed` | *(re-route)* | Unwrap to the real `ImportError` underneath, then apply the import rules above |
23+
| `torch.library.Library.impl ... already a kernel registered` | pytorch/pytorch | |
24+
| `MetaProxy` in `prims.*` / Inductor | pytorch/pytorch | Inductor |
25+
| `PassManager::run failed` inside `triton/` frames | pytorch/pytorch | triton |
26+
| `Pointer argument cannot be accessed from Triton` | pytorch/pytorch | triton |
27+
| `Cannot access data pointer of Tensor (FakeTensor…)` | pytorch/pytorch | AOTAutograd |
28+
| `_pickle.PicklingError` on triton `launcher` | pytorch/pytorch | triton + AOT cache |
29+
| `warm_artifacts_saved: got 0`, `KeyError: None` in standalone_compile | pytorch/pytorch | Inductor cache |
30+
| `assert 'no' == 'yes'` in `test_dynamic_shapes_compilation` | pytorch/pytorch | Dynamo — but rerun first if GPU was OOM |
31+
| `torch.compile with fullgraph=True found no compiled frames` (when `TORCH_COMPILE_DISABLE=1` is in env) | vllm-project/vllm | vLLM-side fix usually correct; upstream interest only if the behavior change is intentional |
32+
| `RayChannelTimeoutError` on tp≥2 ray | pytorch/pytorch | Likely torch.compile per-worker latency exceeds Ray channel timeout |
33+
| `Nondeterministic outputs detected` (B200-only) | pytorch/pytorch | Blackwell-specific kernel drift |
34+
| `assert torch.allclose(golden_output, vllm_output)` reward/PRM | pytorch/pytorch | Numerical drift from triton update |
35+
| `compare_two_settings(... cpu-offload-gb ...)` → "Results are not the same" | pytorch/pytorch | CPU↔GPU dequantize parity |
36+
| GSM8K accuracy collapses to 0.000 (not just degrades) | pytorch/pytorch | Likely worker-side crash hidden behind unpickle error |
37+
| `Generated text "X" doesn't match expected pattern "Y"` on Qwen2-VL / Qwen3-VL LoRA | pytorch/pytorch | Multimodal LoRA path numerical drift |
38+
| `AssertionError: expected size N==N, stride A==B` + `torch.ops.vllm.<X>` + "incorrect fake kernel" | vllm-project/vllm | Fake kernel returns wrong shape |
39+
| Multi-modal per-model assertions (qwen2_vl, chameleon) | vllm-project/vllm | Check vLLM first — may be torch-side once isolated |
40+
| Responses API assertion (`'incomplete' == 'completed'`) | vllm-project/vllm | |
41+
| `test_lm_eval_accuracy_v1_engine` — measured below threshold | pytorch/pytorch | Often numerical drift from a triton update, but confirm against vLLM before filing — this one can land either side |
42+
| `ValueError: Free memory on device cuda:N (X/Y GiB) … less than desired` (tagged `test_is_infra`) | infra | GPU contention — rerun the job, do not file; can cascade dozens of unrelated tests, so the real failures may be a subset |
43+
| CUDA OOM (`torch.OutOfMemoryError` / `CUDA out of memory`) in tp≥2 or B200 fusion tests (runner had ~4–5 GiB free at start) | infra | **Not** tagged `test_is_infra` (runtime OOM ≠ startup free-memory check); cross-check the same job on the same-day main build. If main OOMs the same way, it's contention — skip filing |

0 commit comments

Comments
 (0)