Pair the torch-nightly triage by day, not by commit

The triage only compared a torch-nightly build against a baseline that ran the
*same commit* within 900s. That held while both fired in the same cron slot. It
stopped holding when torch-nightly moved to 10:00 and the baselines stayed at
06:00/21:00 -- four hours of merges apart, they can never agree on a commit
again.

`find_latest_pair` walks newest-first and skips any nightly without a match, so
instead of reporting nothing it silently pinned every run to the last
same-commit pair, #82789/#82790 from 2026-08-07. The workflow kept succeeding
and issue #8491 kept being told "still reproducing on #82789" while builds
#83159, #83338 and #83539 went unexamined. The staleness was invisible: no
failure, no warning, just an increasingly old build number.

Pair on the same UTC day instead, picking the baseline closest in time (ties
still favour the plain nightly). Over the last 21 days this pairs 12/12
torch-nightly builds versus 6/12 before, and on every day where the old rule
found a pair it selects the same baseline -- so nothing regresses, the coverage
gap just closes.

The trade-off is real and deliberate: with different commits the pair is no
longer a controlled A/B, and a regression may come from vLLM commits landing
between the two builds rather than from torch. Reports now say which case they
are. Same-commit pairs read as before; different-commit pairs get an explicit
warning, the time gap, and both commits in the summary table, which now carries
a commit column.

Test plan:
- Ran find_latest_pair against live ClickHouse build rows (14-day window):
  now selects #83539 (2026-08-12 10:00) vs #83511 (2026-08-12 06:00), -4.0h,
  same UTC day, commits differ. Before this change it returned #82789/#82790
  from 2026-08-07.
- Simulated both rules over 21 days of builds:
    same-commit rule:  6/12 torch-nightly builds paired
    same-day rule:    12/12 paired, and identical to the old choice on all 6
                      days where the old rule found one
- Rendered a report both ways: different-commit pairs emit the warning, the
  +/-Nh gap and both commits; same-commit pairs are unchanged apart from the
  new commit column.
- ruff check and ruff format --check clean.

Authored with the assistance of Claude Code.

Author: atalman

tools/torchci/vllm_torch_nightly_triage.py

@@ -12,10 +12,13 @@
 ``0 14 * * *``               Full CI run - daily     (none)
 ===========================  ======================  =========================
 
-On Mon/Tue/Thu the torch-nightly build and the plain nightly fire in the *same
-second* on the *same commit*, differing only by ``TORCH_NIGHTLY=1``. That pair is
-a controlled A/B: a job failing in the former and passing in the latter is
-attributable to torch nightly, with the vLLM variable held constant.
+The baseline is the closest scheduled run on the *same UTC day*. When the two
+fire in the same cron slot they also share a commit, which makes the pair a
+controlled A/B: a job failing in the former and passing in the latter is
+attributable to torch nightly, with the vLLM variable held constant. When the
+schedules drift apart the commits differ and the comparison is day-over-day
+rather than controlled -- still useful, but a regression may then be caused by
+vLLM commits landing between the builds. Reports flag which case they are.
 
 This script finds the most recent such pair and reports the delta. It reads only
 job metadata from ClickHouse -- log *contents* are not ingested (the tables carry
@@ -40,14 +43,20 @@
 PIPELINE = "CI"
 
 TORCH_NIGHTLY_MSG = "Full CI run torch nightly"
-# The plain nightly shares the torch-nightly cron slot; the daily is the fallback
-# baseline when a same-second sibling is missing.
+# The baseline is whichever of these ran closest in time on the same UTC day; the
+# plain nightly wins ties because it shares the torch-nightly cron slot.
+#
+# Same-commit used to be required as well. That holds only while the two share a
+# slot, and stopped holding when torch-nightly moved to 10:00 while the baselines
+# stayed at 06:00/21:00: four hours of merges apart, they can never agree on a
+# commit again. Requiring it silently pinned every report to the last same-commit
+# pair (2026-08-07) while newer builds went unreported.
+#
+# The trade-off is deliberate. With different commits, a job that fails here and
+# passes on the baseline is no longer attributable to torch alone -- intervening
+# vLLM commits are also in scope -- so reports say which case they are.
 BASELINE_MSGS = ("Full CI run - nightly", "Full CI run - daily")
 
-# A build is only a valid control if it ran the same commit. Buildkite schedules
-# fire at the same instant but a commit can land between them in principle.
-SIBLING_WINDOW_SECONDS = 900
-
 BAD_STATES = ("failed", "timed_out")
 
 # Job names are frequently sharded ("Multi-Modal Processor (CPU) 1..4") or
@@ -80,9 +89,10 @@ def find_latest_pair(
 ) -> Optional[Tuple[Dict[str, Any], Dict[str, Any]]]:
     """Return (torch_nightly_build, baseline_build) for the newest complete pair.
 
-    Returns None when no torch-nightly build exists in the window, or when the
-    newest one has no same-commit baseline (in which case there is nothing to
-    compare against and reporting raw failures would be misleading).
+    The baseline is the scheduled run closest in time on the same UTC day. Returns
+    None when no torch-nightly build exists in the window, or when none of them ran
+    on a day that also had a baseline (in which case there is nothing to compare
+    against and reporting raw failures would be misleading).
     """
     builds = _rows(
         client,
@@ -114,18 +124,16 @@ def as_dict(row: Tuple) -> Dict[str, Any]:
     if not nightlies:
         return None
 
-    # Walk newest-first rather than taking only nightlies[0]. Off-schedule
-    # torch-nightly builds happen (manual triggers land outside the 06:00 slot and
-    # have no sibling); stopping at the newest would let one of those mask the most
-    # recent genuinely comparable pair.
+    # Walk newest-first rather than taking only nightlies[0]. A torch-nightly build
+    # can still land on a day with no scheduled baseline at all (manual trigger on a
+    # weekend); stopping at the newest would then report nothing rather than falling
+    # back to the most recent day that is comparable.
     for target in nightlies:
         candidates = [
             b
             for b in parsed
             if b["title"].startswith(BASELINE_MSGS)
-            and b["commit"] == target["commit"]
-            and abs((b["created_at"] - target["created_at"]).total_seconds())
-            <= SIBLING_WINDOW_SECONDS
+            and b["created_at"].date() == target["created_at"].date()
         ]
         if not candidates:
             continue
@@ -243,17 +251,39 @@ def render(
     agents = agent_concentration(regressed)
     top_agent_share = (agents[0][1] / len(regressed)) if regressed else 0.0
 
+    same_commit = tn["commit"] == base["commit"]
     out: List[str] = []
+    if same_commit:
+        out.append(
+            f"**{len(regressed)} job(s) regressed** on torch nightly "
+            f"[#{tn['number']}]({tn['url']}) versus baseline "
+            f"[#{base['number']}]({base['url']}), both at commit "
+            f"`{tn['commit'][:12]}`.\n"
+        )
+    else:
+        # Different commits: the comparison is same-day, not a controlled A/B, so
+        # a "regression" can also come from vLLM commits landing between the two.
+        gap_hours = (base["created_at"] - tn["created_at"]).total_seconds() / 3600
+        out.append(
+            f"**{len(regressed)} job(s) regressed** on torch nightly "
+            f"[#{tn['number']}]({tn['url']}) versus same-day baseline "
+            f"[#{base['number']}]({base['url']}) ({gap_hours:+.1f}h).\n\n"
+            f"> :warning: The two builds ran **different commits** "
+            f"(`{tn['commit'][:12]}` vs `{base['commit'][:12]}`), so this is a "
+            f"same-day comparison rather than a controlled A/B. A regression here "
+            f"may be caused by vLLM commits landing between the builds rather than "
+            f"by torch nightly.\n"
+        )
+    out.append("| | build | commit | outcome |")
+    out.append("|---|---|---|---|")
+    out.append(
+        f"| torch nightly | [#{tn['number']}]({tn['url']}) "
+        f"| `{tn['commit'][:12]}` | {tn['state']} |"
+    )
     out.append(
-        f"**{len(regressed)} job(s) regressed** on torch nightly "
-        f"[#{tn['number']}]({tn['url']}) versus baseline "
-        f"[#{base['number']}]({base['url']}), both at commit "
-        f"`{tn['commit'][:12]}`.\n"
+        f"| baseline | [#{base['number']}]({base['url']}) "
+        f"| `{base['commit'][:12]}` | {base['state']} |"
     )
-    out.append("| | build | outcome |")
-    out.append("|---|---|---|")
-    out.append(f"| torch nightly | [#{tn['number']}]({tn['url']}) | {tn['state']} |")
-    out.append(f"| baseline | [#{base['number']}]({base['url']}) | {base['state']} |")
     out.append(
         f"\n- regressed (fails here, passes on baseline): **{len(regressed)}**\n"
         f"- fails on both (pre-existing, not torch): {len(buckets['both'])}\n"
@@ -443,7 +473,7 @@ def main() -> int:
     pair = find_latest_pair(client, args.lookback_days)
     if pair is None:
         print(
-            f"No torch-nightly build with a same-commit baseline in the last "
+            f"No torch-nightly build with a same-day baseline in the last "
             f"{args.lookback_days} days; nothing to compare.",
             file=sys.stderr,
         )