Add scheduled vLLM torch-nightly regression triage

[atalman]

Summary

vLLM's Buildkite pipeline runs three scheduled builds on main, all from the same HEAD:

schedule (America/Los_Angeles)	message	distinguishing env
0 23 * * 1,2,4	Full CI run torch nightly	TORCH_NIGHTLY=1
0 23 * * *	Full CI run - nightly	—
0 14 * * *	Full CI run - daily	—

On Mon/Tue/Thu the first two fire in the same second on the same commit, differing only by TORCH_NIGHTLY=1. That is a controlled A/B — a job failing in the torch-nightly build and passing in its sibling is attributable to torch, with the vLLM variable held constant. Nothing was consuming that signal.

What this adds

tools/torchci/vllm_torch_nightly_triage.py — finds the newest torch-nightly build with a same-commit baseline and:

• buckets every job into regressed / fails on both (pre-existing) / baseline-only
• clusters regressions by job-name family, collapsing shard indices and hardware qualifiers, so one root cause reads as one entry rather than N
• checks agent concentration — failures piled onto a single host usually mean a sick agent, not a torch regression

Excludes soft_failed (non-blocking by design) and retried (superseded attempts, otherwise double-counted).

.github/workflows/vllm-torch-nightly-triage.yml — runs it Tue/Wed/Fri at 12:00 UTC, after the nightly has had time to finish. When there are regressions it opens an issue with the report, ending in an @claude mention.

Why the split

ClickHouse ingests job metadata only — the tables carry log_url pointers, no log bodies (confirmed: no log/error/message columns exist in the vllm database). So this answers which jobs regressed, not why.

Rather than give the cron a Buildkite token to fetch logs on every run, the issue ends with @claude, which triggers the existing claude-code.yml on issues: opened. Root-cause analysis then runs on demand via the already-checked-in .claude/skills/vllm-pytorch-ci-triage skill, with the shortlist precomputed. Detection is automatic; the expensive half happens when someone cares.

Test plan

Ran against live ClickHouse. Latest pair, #82454 vs #82455 at commit 50c51682a18c:

28 job(s) regressed on torch nightly #82454 versus baseline #82455

- regressed (fails here, passes on baseline): 28
- fails on both (pre-existing, not torch): 6
- fails on baseline only: 4

Clusters (23)
  Multi-Modal Processor — 4 job(s), failed, exit 1
  Language Models Tests — 3 job(s), failed, exit 1
  ...
Failures span 20 agents (heaviest h200-ci-5 with 5). No single-host
concentration, consistent with real signal.

The A/B earns its keep: on the previous pair (#82195/#82196) a naive "what failed in the torch-nightly build" view showed 57 failures, of which 21 fail identically on the baseline. Six Distributed Compile* timeouts looked like a distributed-compile regression and are pre-existing — they would have been filed wrongly.

Also verified: YAML parses, both run: blocks pass bash -n, ruff clean, and the no-pair path exits 0 without writing files or opening an issue.

Known limitations

• Clustering is weak on prefix-varying families. 28 regressions produced 23 clusters. Multi-Modal Processor (CPU) 1-4 collapses correctly, but the NixlConnector family (CrossLayer KV layout Distributed NixlConnector…, DP EP Distributed NixlConnector…, Distributed FlashInfer NixlConnector…) does not, because the distinguishing token is a prefix. Keyword clustering would fix it but risks merging unrelated jobs; left conservative deliberately.
• @claude will not fire as-is. _claude-code.yml requires the issue author be OWNER/MEMBER/COLLABORATOR or pytorch-auto-revert[bot]. An issue opened by github-actions[bot] is NONE. Either add that bot to the allowlist (one line, precedent exists) or open the issue with an App token. Deliberately not changed here — it touches the shared security gate and should be a separate decision.
• Bails out entirely if the newest torch-nightly build has no same-commit baseline; reporting raw failures without a control would be misleading.
• The ci: torch-nightly label needs creating, or gh issue create will fail.

[vercel]

@atalman is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.