Skip to content

Commit 35633f8

Browse files
authored
Add vllm-pytorch-ci-triage skill (pytorch#8047)
## Summary Adds a Claude Code skill that automates end-to-end triage of failing vLLM Buildkite CI runs for PyTorch version-bump PRs. Derived from the multi-week triage of vLLM PR vllm-project/vllm#40077 (torch 2.12.0 + triton 3.7.0), which produced umbrella issue pytorch/pytorch#180899 with 25+ tracked sub-issues over a series of daily runs. ## What the skill does - Pulls the failing build's job list from Buildkite REST API and filters to **true hard failures** (excluding `soft_failed=True`, `waiting_failed`, and infra-aborted jobs). - Compares each failing job against recent main `Full CI run - nightly/daily` builds to drop **pre-existing failures**, with the caveat that infra-killed main jobs are not a valid baseline (must be retried first). - Pulls and ANSI/timestamp-strips logs for the survivors and matches them against a curated set of **root-cause signature regexes** (Inductor MetaProxy, triton PassManager, AOT cache pickling, custom-op fake-kernel stride mismatch, GPU contention, FP8 / quantized accuracy drift, etc.). - Routes each root cause to the right repo: pytorch/pytorch (torch / triton / Inductor / Dynamo / AOTAutograd) vs. vllm-project/vllm (multimodal model assertions, custom-op fake-kernel bugs, response APIs). - Drafts upstream issues with reproducibility tables, environment blocks, and tracebacks, with a strict **draft→confirm→post** protocol, and links them under the umbrella. - Manages umbrella checklist hygiene (mark closed, reopen on regressions, retract on false positives like the recent #182549 retraction). ## Notable lessons baked in - `state=failed` + `soft_failed=True` is non-blocking — always filter both. - `Engine core initialization failed. See root cause above.` is a red herring — the actual exception is several lines up in the EngineCore worker output. - Custom-op `assert_size_stride` failures on `torch.ops.vllm.<X>.default` are almost always **vLLM-side fake-kernel bugs**, not torch regressions — inspect the `direct_register_custom_op(... fake_impl=...)` registration first. - Bulk B200 `exit_status=125` + `nvidia-container-cli: device error / driver rpc error: timed out` is agent infra, not a regression — recommend rerun. - When the same B200 infra cluster wipes out *both* the test-PR and the main-build coverage of a job, the comparison is **inconclusive** — ALWAYS request a main rerun before filing. Filing without that baseline produced a wrongful issue (#182549, retracted). - Buildkite REST API rate-limit is 400/min. Token must be in a shell variable before parallel curl in `while read` (inline `$(cat …)` silently produces 0-byte log files). - Title convention: `[vllm] [<sub-area tag>] <concise root cause>`. Always include `[vllm]` from the start (post-hoc edits are noisy). ## Test plan This skill is invoked manually by Claude Code when the user points at a failing Buildkite build. Validation has been the actual triage of vLLM #40077 over 16+ daily runs since 2026-04-20: - 25+ pytorch/pytorch issues filed under umbrella #180899, with reproducibility tables and environment blocks. - Caught real regressions: AsyncTP correctness (#182124), Fullgraph Smoke Test (#182125), Batch Invariance B200 (#181248, fixed by Lucas Kabela's PR), MetaProxy in FP8 fusion (#180906), aten::bmm double-registration (#180905), and others. - Caught the gpt-oss MoE custom-op stride mismatch as a vLLM-side bug (vllm-project/vllm#41645 / #41646), correctly routing it away from pytorch. - Caught and retracted a false positive (#182549) once the main nightly was retried, demonstrating the infra-baseline lesson. No automated test harness exists for Claude skills today; the skill is exercised through the live triage workflow. ## File added - `.claude/skills/vllm-pytorch-ci-triage/SKILL.md` (379 lines) This follows the same convention as the four existing skills under `.claude/skills/` (each is a single-file `SKILL.md` with YAML frontmatter).
1 parent e91070d commit 35633f8

1 file changed

Lines changed: 379 additions & 0 deletions

File tree

  • .claude/skills/vllm-pytorch-ci-triage

0 commit comments

Comments
 (0)