Skip to content

Pair the torch-nightly triage by day against the pinned nightly - #8522

Merged
atalman merged 1 commit into
pytorch:mainfrom
atalman:vllm_triage_same_day_baseline
Aug 13, 2026
Merged

Pair the torch-nightly triage by day against the pinned nightly#8522
atalman merged 1 commit into
pytorch:mainfrom
atalman:vllm_triage_same_day_baseline

Conversation

@atalman

@atalman atalman commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Why

The triage compared a torch-nightly build only against a baseline that ran the same commit within 900s. That held while both fired in the same cron slot. It stopped holding when torch-nightly moved to 10:00 and the baselines stayed at 06:00/21:00 — four hours of merges apart, they can never agree on a commit again.

find_latest_pair walks newest-first and skips any nightly without a match, so rather than reporting nothing it silently pinned every run to the last same-commit pair, #82789/#82790 from 2026-08-07. The workflow kept succeeding and #8491 kept being told "still reproducing on #82789" while #83159, #83338 and #83539 went unexamined. No failure, no warning — just an increasingly old build number.

83539  Aug 12 10:00  ->  no same-commit baseline   skipped
83338  Aug 11 10:00  ->  no same-commit baseline   skipped
83159  Aug 10 15:14  ->  no same-commit baseline   skipped
82789  Aug  7 06:00  ->  82790                     selected

What changes

1. Pair on the same UTC day rather than the same commit.

same-commit same-day
torch-nightly builds paired (21d) 6/12 12/12
baseline chosen on the 6 days both rules work identical

Nothing regresses; the coverage gap closes.

2. Prefer Full CI run - nightly over Full CI run - daily, then closest in time. The plain nightly is the pinned-torch counterpart of the torch-nightly build — same pipeline, same schedule, differing only by TORCH_NIGHTLY=1. Under a time-first ordering, 2026-08-10 would have compared against the daily (+5.8h) instead of the pinned nightly (-9.2h).

Resulting pairings over 21 days — every one against a pinned - nightly:

83539  2026-08-12 -> 83511 (nightly  -4.0h)
83338  2026-08-11 -> 83298 (nightly  -4.0h)
83159  2026-08-10 -> 83094 (nightly  -9.2h)
82789  2026-08-07 -> 82790 (nightly  +0.0h)
82682  2026-08-06 -> 82629 (nightly  -7.4h)
82454  2026-08-05 -> 82455 (nightly  +0.0h)
...

The trade-off, made explicit

With different commits the pair is no longer a controlled A/B — a regression may come from vLLM commits landing between the two builds rather than from torch. Reports now say which case they are:

⚠️ The two builds ran different commits (6accb779a361 vs 02ac17851dfb), so this is a same-day comparison rather than a controlled A/B. A regression here may be caused by vLLM commits landing between the builds rather than by torch nightly.

plus the ±Nh gap and both commits in the summary table (which gains a commit column). Same-commit pairs read exactly as before.

Test plan

  • find_latest_pair against live ClickHouse rows now returns #83539 (torch nightly, Aug 12 10:00) vs #83511 (pinned nightly, Aug 12 06:00); before: #82789/#82790 from Aug 7.
  • Simulated over 21 days: 12/12 paired (was 6/12), all against - nightly, identical to the old choice on every day the old rule worked.
  • Job-set sizes are comparable across the three build types (315–321 distinct job names), so the comparison is not skewed by one type running a materially different set.
  • Rendered a report both ways — the warning appears only when commits differ.
  • ruff check / ruff format --check clean.

Fixes the staleness behind #8491.

Authored with the assistance of Claude Code.

@vercel

vercel Bot commented Aug 13, 2026

Copy link
Copy Markdown

@atalman is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 13, 2026
@atalman
atalman force-pushed the vllm_triage_same_day_baseline branch from 4873f4c to 06bd098 Compare August 13, 2026 19:26
@atalman atalman changed the title Pair the torch-nightly triage by day, not by commit Pair the torch-nightly triage by day against the pinned nightly Aug 13, 2026
The triage only compared a torch-nightly build against a baseline that ran the
*same commit* within 900s. That held while both fired in the same cron slot. It
stopped holding when torch-nightly moved to 10:00 and the baselines stayed at
06:00/21:00 -- four hours of merges apart, they can never agree on a commit
again.

`find_latest_pair` walks newest-first and skips any nightly without a match, so
instead of reporting nothing it silently pinned every run to the last
same-commit pair, #82789/#82790 from 2026-08-07. The workflow kept succeeding
and issue pytorch#8491 kept being told "still reproducing on #82789" while builds
#83159, #83338 and #83539 went unexamined. The staleness was invisible: no
failure, no warning, just an increasingly old build number.

Two changes:

- Pair on the same UTC day instead of the same commit. Over the last 21 days
  this pairs 12/12 torch-nightly builds versus 6/12 before, and on every day
  where the old rule found a pair it selects the same baseline -- so nothing
  regresses, the coverage gap just closes.

- Prefer "Full CI run - nightly" over "Full CI run - daily", then closest in
  time. The plain nightly is the pinned-torch counterpart of the torch-nightly
  build: same pipeline, same schedule, differing only by TORCH_NIGHTLY=1. Under
  the previous time-first ordering 2026-08-10 would have compared against the
  daily (+5.8h) rather than the pinned nightly (-9.2h).

The trade-off is real and deliberate: with different commits the pair is no
longer a controlled A/B, and a regression may come from vLLM commits landing
between the two builds rather than from torch. Reports now say which case they
are. Same-commit pairs read as before; different-commit pairs get an explicit
warning, the time gap, and both commits in the summary table, which now carries
a commit column.

Test plan:
- Ran find_latest_pair against live ClickHouse build rows: now selects
  #83539 (2026-08-12 10:00, torch nightly) vs #83511 (2026-08-12 06:00,
  pinned nightly), -4.0h, same UTC day, commits differ. Before this change it
  returned #82789/#82790 from 2026-08-07.
- Simulated the rule over 21 days of builds: 12/12 torch-nightly builds paired
  (was 6/12), every one against a "Full CI run - nightly" baseline, and
  identical to the old choice on all 6 days where the old rule found one.
- Job-set sizes are comparable across the three build types (315-321 distinct
  job names), so the day-over-day comparison is not skewed by one build type
  running a materially different set.
- Rendered a report both ways: different-commit pairs emit the warning, the
  +/-Nh gap and both commits; same-commit pairs are unchanged apart from the
  new commit column.
- ruff check and ruff format --check clean.

Authored with the assistance of Claude Code.
@atalman
atalman force-pushed the vllm_triage_same_day_baseline branch from 06bd098 to e02619d Compare August 13, 2026 19:31

@morrison-turnansky morrison-turnansky left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as we are good with potentially picking up manual runs instead of cron jobs we should be good. Since it will still favor same commits, I don't see that as a problem.

@atalman

atalman commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

HI @morrison-turnansky yes picking up manual runs is expected. This is how we where running so far.

@atalman
atalman merged commit 5ae9e8a into pytorch:main Aug 13, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants