Commit 35633f8
authored
Add vllm-pytorch-ci-triage skill (pytorch#8047)
## Summary
Adds a Claude Code skill that automates end-to-end triage of failing
vLLM Buildkite CI runs for PyTorch version-bump PRs.
Derived from the multi-week triage of vLLM PR vllm-project/vllm#40077
(torch 2.12.0 + triton 3.7.0), which produced umbrella issue
pytorch/pytorch#180899 with 25+ tracked sub-issues over a series of
daily runs.
## What the skill does
- Pulls the failing build's job list from Buildkite REST API and filters
to **true hard failures** (excluding `soft_failed=True`,
`waiting_failed`, and infra-aborted jobs).
- Compares each failing job against recent main `Full CI run -
nightly/daily` builds to drop **pre-existing failures**, with the caveat
that infra-killed main jobs are not a valid baseline (must be retried
first).
- Pulls and ANSI/timestamp-strips logs for the survivors and matches
them against a curated set of **root-cause signature regexes** (Inductor
MetaProxy, triton PassManager, AOT cache pickling, custom-op fake-kernel
stride mismatch, GPU contention, FP8 / quantized accuracy drift, etc.).
- Routes each root cause to the right repo: pytorch/pytorch (torch /
triton / Inductor / Dynamo / AOTAutograd) vs. vllm-project/vllm
(multimodal model assertions, custom-op fake-kernel bugs, response
APIs).
- Drafts upstream issues with reproducibility tables, environment
blocks, and tracebacks, with a strict **draft→confirm→post** protocol,
and links them under the umbrella.
- Manages umbrella checklist hygiene (mark closed, reopen on
regressions, retract on false positives like the recent #182549
retraction).
## Notable lessons baked in
- `state=failed` + `soft_failed=True` is non-blocking — always filter
both.
- `Engine core initialization failed. See root cause above.` is a red
herring — the actual exception is several lines up in the EngineCore
worker output.
- Custom-op `assert_size_stride` failures on
`torch.ops.vllm.<X>.default` are almost always **vLLM-side fake-kernel
bugs**, not torch regressions — inspect the
`direct_register_custom_op(... fake_impl=...)` registration first.
- Bulk B200 `exit_status=125` + `nvidia-container-cli: device error /
driver rpc error: timed out` is agent infra, not a regression —
recommend rerun.
- When the same B200 infra cluster wipes out *both* the test-PR and the
main-build coverage of a job, the comparison is **inconclusive** —
ALWAYS request a main rerun before filing. Filing without that baseline
produced a wrongful issue (#182549, retracted).
- Buildkite REST API rate-limit is 400/min. Token must be in a shell
variable before parallel curl in `while read` (inline `$(cat …)`
silently produces 0-byte log files).
- Title convention: `[vllm] [<sub-area tag>] <concise root cause>`.
Always include `[vllm]` from the start (post-hoc edits are noisy).
## Test plan
This skill is invoked manually by Claude Code when the user points at a
failing Buildkite build. Validation has been the actual triage of vLLM
#40077 over 16+ daily runs since 2026-04-20:
- 25+ pytorch/pytorch issues filed under umbrella #180899, with
reproducibility tables and environment blocks.
- Caught real regressions: AsyncTP correctness (#182124), Fullgraph
Smoke Test (#182125), Batch Invariance B200 (#181248, fixed by Lucas
Kabela's PR), MetaProxy in FP8 fusion (#180906), aten::bmm
double-registration (#180905), and others.
- Caught the gpt-oss MoE custom-op stride mismatch as a vLLM-side bug
(vllm-project/vllm#41645 / #41646), correctly routing it away from
pytorch.
- Caught and retracted a false positive (#182549) once the main nightly
was retried, demonstrating the infra-baseline lesson.
No automated test harness exists for Claude skills today; the skill is
exercised through the live triage workflow.
## File added
- `.claude/skills/vllm-pytorch-ci-triage/SKILL.md` (379 lines)
This follows the same convention as the four existing skills under
`.claude/skills/` (each is a single-file `SKILL.md` with YAML
frontmatter).1 parent e91070d commit 35633f8
1 file changed
Lines changed: 379 additions & 0 deletions
0 commit comments