Add scheduled vLLM torch-nightly regression triage

[atalman]

Summary

vLLM's Buildkite pipeline runs three scheduled builds on main, all from the same HEAD:

schedule (America/Los_Angeles)	message	distinguishing env
0 23 * * 1,2,4	Full CI run torch nightly	TORCH_NIGHTLY=1
0 23 * * *	Full CI run - nightly	—
0 14 * * *	Full CI run - daily	—

The first two share the 23:00 slot, so on Mon/Tue/Thu they fire in the same second on the same commit, differing only by that one env var. That pair is a controlled A/B: a job failing in the torch-nightly build and passing in its sibling is attributable to torch, with the vLLM variable held constant. Nothing was consuming that signal.

What this adds

tools/torchci/vllm_torch_nightly_triage.py — finds the newest torch-nightly build with a same-commit baseline and:

• buckets every job into regressed / fails on both (pre-existing) / baseline-only
• excludes soft_failed (non-blocking by design) and retried (superseded attempts, otherwise double-counted)
• clusters regressions by job-name family, collapsing shard indices and hardware qualifiers
• flags agent concentration — failures piled onto one host usually mean a sick agent, not a torch regression
• optionally fetches one representative Buildkite log per cluster, ANSI/BKT-marker-stripped and tail-trimmed

.github/workflows/vllm-torch-nightly-triage.yml — two jobs:

1. triage — runs the above Tue/Wed/Fri at 12:00 UTC, after the nightly has finished. Prints to the job log and run summary, uploads report + logs as an artifact.
2. root-cause — analyses the result with the checked-in .claude/skills/vllm-pytorch-ci-triage skill.

Reports only. Files nothing anywhere.

Why the A/B earns its keep

On build 82195 a naive "what failed on torch nightly" view showed 57 failures. 21 of those fail identically on the baseline — including six Distributed Compile* timeouts that looked like a distributed-compile regression and are pre-existing. They would have been filed upstream wrongly.

Security posture of the agent job

Logs are fetched by the triage job, not the model. So:

• the read-only Buildkite token never enters the agent's tool surface
• the model runs --allowedTools "Read,Glob,Grep,Write" — no Bash, no network, no credentials
• root-cause holds no ClickHouse or Buildkite secrets, only Bedrock via OIDC
• the prompt marks all log content as untrusted data, never instructions

Follows the greenlight-pr-review.yml pattern: claude-code-action directly with a literal prompt, environment: bedrock. That works from schedule, unlike _claude-code.yml, whose gate requires an @claude mention in an issue body.

Test plan

Verified against live ClickHouse and Buildkite. Latest pair, #82454 vs #82455 at commit 50c51682a18c:

27 job(s) regressed on torch nightly #82454 versus baseline #82455

- regressed (fails here, passes on baseline): 27
- fails on both (pre-existing, not torch): 6
- fails on baseline only: 4

Failures span 20 agents (heaviest h200-ci-5 with 5). No single-host
concentration, consistent with real signal.

Log fetching returns 22 cleaned cluster logs with real failures surfacing, e.g. FAILED v1/test_tensor_ipc_queue.py::test_multiple_api_servers_to_engine - _queue.Empty.

CI-verified on a temporary pull_request trigger before it was removed: triage green, root-cause correctly skipped. Locally clean under the repo's own linter adapters (PYFMT with the pinned ruff==0.14.4/usort==1.0.8.post1, RUFF, MYPY) and shellcheck on every run: block.

Reuses the existing CLICKHOUSE_HUD_USER_* secrets — no new ClickHouse credentials.

Known limitations

This lands as a working example; a follow-up will replace the analysis core with the substantially better engine in morrison-turnansky/vllm-nightly-audit, which compares test signatures (test_id, exception_class, exception_chain) rather than job names, classifies NEW/PRE_EXISTING/NO_BASELINE against ≥3 main builds, and detects near-misses.

• Clustering is weak. 27 regressions produce ~23 clusters. Multi-Modal Processor (CPU) 1-4 collapses correctly, but the NixlConnector family does not, because its distinguishing token is a prefix. Keyword clustering would fix it and risks merging unrelated jobs, so this is deliberately conservative — and superseded by the follow-up.
• BUILDKITE_CI_READ_ONLY currently returns 401 on every log fetch. Detection is unaffected, but root-cause gates on log_count != '0' and will skip until the token is fixed (likely a trailing newline, missing read_build_logs, or the vllm org not selected).
• root-cause is unverified. It cannot run on PRs by design, so first real exercise is a workflow_dispatch after merge.
• A read-only Buildkite token cannot retry jobs, so the skill's Step 6.5 auto-restart is inert. That is intentional: a cron silently retrying vLLM CI is a side effect on someone else's infrastructure.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
torchci	Ignored	Preview	Aug 6, 2026 1:34pm

[huydhn]

Should these secrets be in an environment? I wonder why we still keep them outside

[atalman]

hi @huydhn ok will create a BE PR tomorrow to move rest of the secrets into env . Moved BUILDKITE_CI_READ_ONLY under vllm-triage env

[huydhn]

Overall LGTM! I review mainly the workflow setup. The python script as it has only simple functionalities to query CI results and pull logs, so Claude probably gets it right and we can iterate on that if needed

cc @izaitsevfb for your review too

[izaitsevfb]

findings from my agent:

1. The cron fires before the builds finish, and in-flight baseline jobs are scored as torch regressions. Cron is 12:00 UTC; both builds start 06:00 UTC and finish at 16:23 / 20:22 / +21h / +4 days — 5 of 5 recent pairs still running at 12:00. find_latest_pair() never checks build.finished_at. And compare() buckets regressed on tn_bad and in_base, where in_base only means the job exists in baseline, any state — canceled is the 2nd-commonest state (16,596/30d, ahead of failed). Fix: terminal-state gate + base_state == "passed" + cron +24h.

2. GROUP BY job_name + anyIf is nondeterministic on this data. In the PR's own test pair, 7 names have mixed states — Kernels Core Operation Test = passed,passed,failed (3 shards). Same immutable data, different verdict per run; state/url/agent each resolve from a different row. Fix: group by (name, parallel_group_index), argMaxIf(x, finished_at, …).

3. template injection: ${{ inputs.lookback_days }} spliced into run: in the step holding both secrets Unscoped Read + live AWS creds + public artifact.

details: https://www.internalfb.com/phabricator/paste/view/P2450487683

[atalman]

Thanks @izaitsevfb — the agent findings were accurate and all three are now fixed, plus a fourth bug that surfaced while verifying. Summary of everything changed since the review.

1. Cron timing

0 12 * * 2,3,5 → 0 17 * * 2,3,5 (13:00 EDT / 12:00 EST).

The original 12:00 UTC was indeed too early — measured over 11 complete pairs in the last 30 days, zero were finished at 12:00 UTC.

The +24h suggestion turned out to be more conservative than needed, because the long tail is retries rather than new signal. On 82455, all five jobs that set the build duration had been retried 2–4 times and every one still ended failed; only 15 of 323 jobs retry at all. Excluding the retry tail:

	all attempts	first attempt only
median	+11.1h	+8.4h
max	+105h	+10.6h

So 17:00 UTC is the earliest slot covering all 11 pairs (16:00 covers 8 of 11). Deliberately no terminal-state gate — gating on build completion would wait on precisely those retry tails, delaying the report by hours without changing it. A job that has already failed is enough to triage.

2. in_base was too loose

in_base only means the job exists in baseline, any state — canceled is the 2nd-commonest state

Fixed: a regression now requires base_state == "passed", not merely that the job exists in the baseline build. Jobs absent from the torch-nightly build are handled separately rather than falling through.

Effect on the current pair (82454 vs 82455): 27 → 26 regressed, 6 → 7 pre-existing, 4 → 0 baseline-only. One job was being misattributed to torch.

3. Sharded jobs were nondeterministic

GROUP BY job_name + anyIf is nondeterministic … Kernels Core Operation Test = passed,passed,failed (3 shards)

Fixed: GROUP BY (job_name, parallel_group_index) with argMaxIf(..., finished_at, ...) instead of anyIf. Latest attempt wins deterministically, and state/url/agent are guaranteed to come from the same row. Shard index is surfaced in the report when non-zero.

4. Off-schedule builds masked the newest pair (found while verifying)

find_latest_pair() took only nightlies[0] and gave up if it had no same-commit sibling. Manually-triggered torch-nightly builds land outside the 06:00 slot and have no sibling — this reproduced live with 82682 (13:21 UTC), which blanked the report while 82454/82455 sat there perfectly comparable.

Now walks torch-nightly builds newest-first and returns the first with a baseline. Note the intended consequence: a lone manual build is skipped rather than reported without a control.

5. Template injection

${{ inputs.lookback_days }} spliced into run: in the step holding both secrets

Fixed, and by removal rather than escaping: the workflow_dispatch input is gone and the value is a literal --lookback-days 14. Nobody needed the knob, and a crafted value would have been evaluated as shell in a step holding CLICKHOUSE_HUD_USER_* and BUILDKITE_CI_READ_ONLY.

No run: block in this workflow interpolates ${{ }} any more, so the whole class is closed rather than one site patched. workflow_dispatch remains as a trigger, just without the parameter — still usable to exercise root-cause after merge.

Also changed since the review

• Bedrock auth via OIDC (@huydhn's comment). The workflow had copied github_token from greenlight-pr-review.yml but omitted the configure-aws-credentials step that goes with it, so root-cause had no Bedrock credentials at all — environment: bedrock only scopes secrets and branch access. Added the OIDC step assuming role/gha_workflow_claude_code, dropped github_token entirely (the agent reads local files and writes one markdown file; it needs no GitHub API access).
• environment: vllm-triage on the triage job, since BUILDKITE_CI_READ_ONLY is an environment secret and only resolves for jobs declaring it.