Pair the torch-nightly triage by day against the pinned nightly - #8522
Merged
Conversation
|
@atalman is attempting to deploy a commit to the Meta Open Source Team on Vercel. A member of the Team first needs to authorize it. |
atalman
force-pushed
the
vllm_triage_same_day_baseline
branch
from
August 13, 2026 19:26
4873f4c to
06bd098
Compare
The triage only compared a torch-nightly build against a baseline that ran the *same commit* within 900s. That held while both fired in the same cron slot. It stopped holding when torch-nightly moved to 10:00 and the baselines stayed at 06:00/21:00 -- four hours of merges apart, they can never agree on a commit again. `find_latest_pair` walks newest-first and skips any nightly without a match, so instead of reporting nothing it silently pinned every run to the last same-commit pair, #82789/#82790 from 2026-08-07. The workflow kept succeeding and issue pytorch#8491 kept being told "still reproducing on #82789" while builds #83159, #83338 and #83539 went unexamined. The staleness was invisible: no failure, no warning, just an increasingly old build number. Two changes: - Pair on the same UTC day instead of the same commit. Over the last 21 days this pairs 12/12 torch-nightly builds versus 6/12 before, and on every day where the old rule found a pair it selects the same baseline -- so nothing regresses, the coverage gap just closes. - Prefer "Full CI run - nightly" over "Full CI run - daily", then closest in time. The plain nightly is the pinned-torch counterpart of the torch-nightly build: same pipeline, same schedule, differing only by TORCH_NIGHTLY=1. Under the previous time-first ordering 2026-08-10 would have compared against the daily (+5.8h) rather than the pinned nightly (-9.2h). The trade-off is real and deliberate: with different commits the pair is no longer a controlled A/B, and a regression may come from vLLM commits landing between the two builds rather than from torch. Reports now say which case they are. Same-commit pairs read as before; different-commit pairs get an explicit warning, the time gap, and both commits in the summary table, which now carries a commit column. Test plan: - Ran find_latest_pair against live ClickHouse build rows: now selects #83539 (2026-08-12 10:00, torch nightly) vs #83511 (2026-08-12 06:00, pinned nightly), -4.0h, same UTC day, commits differ. Before this change it returned #82789/#82790 from 2026-08-07. - Simulated the rule over 21 days of builds: 12/12 torch-nightly builds paired (was 6/12), every one against a "Full CI run - nightly" baseline, and identical to the old choice on all 6 days where the old rule found one. - Job-set sizes are comparable across the three build types (315-321 distinct job names), so the day-over-day comparison is not skewed by one build type running a materially different set. - Rendered a report both ways: different-commit pairs emit the warning, the +/-Nh gap and both commits; same-commit pairs are unchanged apart from the new commit column. - ruff check and ruff format --check clean. Authored with the assistance of Claude Code.
atalman
force-pushed
the
vllm_triage_same_day_baseline
branch
from
August 13, 2026 19:31
06bd098 to
e02619d
Compare
zxiiro
approved these changes
Aug 13, 2026
morrison-turnansky
self-requested a review
August 13, 2026 19:51
morrison-turnansky
approved these changes
Aug 13, 2026
morrison-turnansky
left a comment
Collaborator
There was a problem hiding this comment.
As long as we are good with potentially picking up manual runs instead of cron jobs we should be good. Since it will still favor same commits, I don't see that as a problem.
Contributor
Author
|
HI @morrison-turnansky yes picking up manual runs is expected. This is how we where running so far. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The triage compared a torch-nightly build only against a baseline that ran the same commit within 900s. That held while both fired in the same cron slot. It stopped holding when torch-nightly moved to 10:00 and the baselines stayed at 06:00/21:00 — four hours of merges apart, they can never agree on a commit again.
find_latest_pairwalks newest-first and skips any nightly without a match, so rather than reporting nothing it silently pinned every run to the last same-commit pair, #82789/#82790 from 2026-08-07. The workflow kept succeeding and #8491 kept being told "still reproducing on #82789" while #83159, #83338 and #83539 went unexamined. No failure, no warning — just an increasingly old build number.What changes
1. Pair on the same UTC day rather than the same commit.
Nothing regresses; the coverage gap closes.
2. Prefer
Full CI run - nightlyoverFull CI run - daily, then closest in time. The plain nightly is the pinned-torch counterpart of the torch-nightly build — same pipeline, same schedule, differing only byTORCH_NIGHTLY=1. Under a time-first ordering, 2026-08-10 would have compared against the daily (+5.8h) instead of the pinned nightly (-9.2h).Resulting pairings over 21 days — every one against a pinned
- nightly:The trade-off, made explicit
With different commits the pair is no longer a controlled A/B — a regression may come from vLLM commits landing between the two builds rather than from torch. Reports now say which case they are:
plus the
±Nhgap and both commits in the summary table (which gains a commit column). Same-commit pairs read exactly as before.Test plan
find_latest_pairagainst live ClickHouse rows now returns #83539 (torch nightly, Aug 12 10:00) vs #83511 (pinned nightly, Aug 12 06:00); before: #82789/#82790 from Aug 7.- nightly, identical to the old choice on every day the old rule worked.ruff check/ruff format --checkclean.Fixes the staleness behind #8491.
Authored with the assistance of Claude Code.