[drci] Re-render advisor comment until the verdict concludes (alt sentinel) (#8235)

## Problem

The inline AI advisor verdict line (#8202) has two render states under
each new/unclassified failure:
- **in-progress** — a bare `AI verdict:` line + badge image, and
- **concluded** — the line wrapped in `<details>` with the reasoning.

The badge **image** flips server-side the moment the verdict lands (the
`<img>` points at `/api/drci/advisorBadge`, which camo refetches). But
the concluded `<details>` **expand is comment-body text** — it only
appears when the Dr.CI comment is re-rendered.

The 15-min cron (`updateDrciComments`) only re-renders a PR that has
workflow activity in the last `NUM_MINUTES` (30) **or** whose comment
still shows `\d Pending` jobs. A verdict can land seconds *after* the
last render and the PR's CI then go quiet — so the comment stays stuck
on the in-progress line indefinitely. The `body === comment` idempotency
check is fine; it's just never reached, because the PR drops out of the
candidate set.

Observed live on pytorch/pytorch#188479: the verdict landed **44s
after** the comment's last render, and never expanded.

## Fix

Give the advisor `<img>` an `alt` that encodes the outcome:
- in-progress → `alt="AI verdict: pending"` (a sentinel),
- concluded → `alt="AI verdict: <label>"` (`related`, `not related`,
`probably not related`, …).

Then broaden the cron candidate query (`getPRsWithPendingJobInComment` →
`getPRsNeedingCommentRefresh`) to **also** keep open PRs whose comment
still carries the pending sentinel. They re-render every tick until the
verdict concludes; the moment it does, the alt changes (no longer
`pending`) and the PR **self-clears** from the set. Same open-PR +
1-month freshness guards as the existing `\d Pending` branch, so growth
is bounded the same way.

The query matches the **full `alt="AI verdict: pending"` attribute**,
not the bare phrase: advisor summaries are HTML-escaped, so a `"` in a
summary becomes `&quot;` and can never reproduce the real double-quoted
attribute → no false-match. A single shared `ADVISOR_PENDING_ALT_ATTR`
constant drives both the emitted line and the query so they can't drift.

**Bonus:** the verdict outcome is now machine-readable text in the
comment (useful for AI agents reading the comment, or humans with images
off), not only inside the badge SVG.

## Notes

- Forward-looking: comments **already** stuck (rendered before this
change, with no `alt`) won't self-heal — they need a one-time
`@pytorchbot drci` nudge or they age out via the 1-month guard.
- Tests: `advisorBadge.test.ts` + `drci.test.ts` pass; `tsc --noEmit`
clean; lintrunner clean.

Co-authored-by: Ivan Zaitsev <izaitsevfb@meta.com>

Author: izaitsevfb

torchci/lib/advisor/advisorBadge.ts

@@ -18,6 +18,31 @@ export function drciSignalKeyForJob(fullJobName: string): string {
   return `${DRCI_SIGNAL_KEY_PREFIX}${fullJobName}`;
 }
 
+// Each advisor <img> carries an alt of the form `AI verdict: <outcome>`. This
+// does two jobs: (1) it encodes the verdict outcome as machine-readable text in
+// the comment body (an AI agent reading the comment, or a human with images
+// off, gets the verdict without fetching the badge SVG), and (2) the
+// not-yet-concluded sentinel below is what the Dr.CI cron matches on to keep
+// re-rendering a PR until its verdict lands -- the badge image flips
+// server-side via camo, but the concluded <details> expand is comment-body text
+// that only appears on a re-render, so an unconcluded PR must stay a candidate.
+export const ADVISOR_ALT_PREFIX = "AI verdict: ";
+// Sentinel alt for the in-progress line (no concluded verdict at render time).
+// No verdict label contains "pending", so a concluded line never matches this.
+export const ADVISOR_PENDING_ALT = `${ADVISOR_ALT_PREFIX}pending`;
+// The exact attribute the in-progress line emits. The cron candidate query
+// matches THIS (the full `alt="..."` form, not the bare phrase) so a
+// model-generated summary that happens to contain the words can't false-match:
+// summaries are HTML-escaped, so a literal `"` in one becomes `&quot;` and can
+// never reproduce the real double-quoted attribute.
+export const ADVISOR_PENDING_ALT_ATTR = `alt="${ADVISOR_PENDING_ALT}"`;
+
+// The alt text for a concluded verdict line: `AI verdict: <label>`, where the
+// label is the same human-readable outcome shown on the badge pill.
+export function advisorVerdictAlt(verdict: string, confidence: number): string {
+  return `${ADVISOR_ALT_PREFIX}${verdictBadge(verdict, confidence).label}`;
+}
+
 export interface AdvisorBadge {
   label: string;
   // Hex fill, e.g. "#2da44e".
@@ -156,7 +181,10 @@ export function renderInProgressLine(
 ): string {
   const badge = advisorBadgeUrl(hudBaseUrl, owner, repo, sha, jobName);
   const link = hudPrUrl(hudBaseUrl, owner, repo, prNumber, jobId);
-  return `    AI verdict: <a href="${link}"><img src="${badge}"></a>\n`;
+  // alt = the pending sentinel: it marks this PR for re-rendering until the
+  // verdict lands (see ADVISOR_PENDING_ALT_ATTR) and reads as "AI verdict:
+  // pending". Emit the shared attr constant so the cron's match stays in lockstep.
+  return `    AI verdict: <a href="${link}"><img ${ADVISOR_PENDING_ALT_ATTR} src="${badge}"></a>\n`;
 }
 
 // Concluded line: "AI verdict:" plain text toggles the expand; the badge links
@@ -169,17 +197,24 @@ export function renderVerdictLine(
   sha: string,
   jobName: string,
   jobId: number,
+  verdict: string,
+  confidence: number,
   summary: string
 ): string {
   const badge = advisorBadgeUrl(hudBaseUrl, owner, repo, sha, jobName);
   const link = hudPrUrl(hudBaseUrl, owner, repo, prNumber, jobId);
+  // alt encodes the concluded outcome (`AI verdict: related`, etc.) so the
+  // verdict is machine-readable text in the comment AND no longer matches the
+  // pending sentinel, dropping the PR from the re-render candidate set. The
+  // label comes from our own verdictBadge map, but escape defensively anyway.
+  const altText = _.escape(advisorVerdictAlt(verdict, confidence));
   // The advisor summary is model-generated from (attacker-influenceable) PR
   // content, so HTML-escape it before embedding in the comment: collapse
   // newlines (can't break the blockquote) and neutralize markup so it can't
   // close the <details>/<blockquote> or inject tags.
   const oneLine = _.escape((summary || "").replace(/\s*\n\s*/g, " ").trim());
   return (
-    `  <details><summary>AI verdict: <a href="${link}"><img src="${badge}"></a></summary><blockquote>\n\n` +
+    `  <details><summary>AI verdict: <a href="${link}"><img alt="${altText}" src="${badge}"></a></summary><blockquote>\n\n` +
     `  ${oneLine}\n\n` +
     `  <a href="${link}">Full reasoning on HUD &rarr;</a>\n` +
     `  </blockquote></details>\n`
@@ -193,6 +228,9 @@ export interface AdvisorLineJob {
   name: string;
 }
 export interface AdvisorLineVerdict {
+  // verdict + confidence drive the badge label baked into the alt text.
+  verdict: string;
+  confidence: number;
   summary: string;
 }
 
@@ -225,6 +263,8 @@ export function selectAdvisorLines(
           headSha,
           job.name,
           job.id,
+          verdict.verdict,
+          verdict.confidence,
           verdict.summary
         )
       );

torchci/lib/advisor/advisorComment.ts

@@ -59,7 +59,11 @@ export async function buildAdvisorVerdictLines(
   const verdictByKey = new Map<string, AdvisorLineVerdict>();
   for (const v of deduplicateVerdicts(verdictRows)) {
     if (v.sha === headSha) {
-      verdictByKey.set(v.signalKey, { summary: v.summary });
+      verdictByKey.set(v.signalKey, {
+        verdict: v.verdict,
+        confidence: v.confidence,
+        summary: v.summary,
+      });
     }
   }
 

torchci/lib/advisor/advisorDispatch.ts

@@ -398,7 +398,7 @@ const SKIP_DRAFT_PRS = true;
 /**
  * Fetch the minimal PR state needed to gate auto-dispatch, from the
  * default.pull_request ClickHouse mirror rather than the GitHub API. The rest
- * of the advisor path already reads from CH (and getPRsWithPendingJobInComment
+ * of the advisor path already reads from CH (and getPRsNeedingCommentRefresh
  * in drci.ts already reads pull_request.state the same way), so this keeps the
  * Dr.CI cron off the GitHub rate limit. The mirror lags GitHub by ~1 minute,
  * well within the 15-minute cron cadence. A PR not yet mirrored (no row) is

torchci/pages/api/drci/drci.ts

@@ -1,6 +1,7 @@
 import { PutObjectCommand } from "@aws-sdk/client-s3";
 import dayjs from "dayjs";
 import utc from "dayjs/plugin/utc";
+import { ADVISOR_PENDING_ALT_ATTR } from "lib/advisor/advisorBadge";
 import { buildAdvisorVerdictLines } from "lib/advisor/advisorComment";
 import { autoDispatchAdvisorForNewFailures } from "lib/advisor/advisorDispatch";
 import { fetchJSON, isTime0 } from "lib/bot/utils";
@@ -158,7 +159,7 @@ export async function updateDrciComments(
       ? []
       : fetchRecentWorkflows(
           `${owner}/${repo}`,
-          await getPRsWithPendingJobInComment(`${owner}/${repo}`),
+          await getPRsNeedingCommentRefresh(`${owner}/${repo}`),
           NUM_MINUTES + ""
         ),
   ]);
@@ -403,7 +404,20 @@ function removeFailureContext(failure: {
  * @param repo The repository to search for PRs in. E.g. "pytorch/pytorch"
  * @returns A list of PR numbers
  */
-async function getPRsWithPendingJobInComment(repo: String): Promise<number[]> {
+// PRs whose Dr.CI comment is in a transient state that warrants a re-render
+// even with no recent (< NUM_MINUTES) workflow activity. Two cases:
+//   - `\d Pending`: the comment still shows pending jobs that may resolve.
+//   - the advisor pending sentinel: a NEW/unclassified failure was dispatched
+//     to the AI advisor but its verdict had not landed at the last render, so
+//     the comment carries the in-progress `alt="AI verdict: pending"` line. The
+//     verdict can land seconds after a render and the PR's CI then go quiet,
+//     so without this clause the concluded <details> expand would never get
+//     written. Once the verdict renders, the alt becomes `AI verdict: <label>`
+//     (no "pending"), so the PR self-clears from this set. We match the full
+//     alt attribute (not the bare phrase) so an escaped model summary can't
+//     false-match.
+// Both branches stay gated by the open-PR + 1-month freshness guards below.
+async function getPRsNeedingCommentRefresh(repo: String): Promise<number[]> {
   const query = `
 select
     issue_comment.issue_url
@@ -412,12 +426,15 @@ from
     join default.pull_request on issue_comment.issue_url = pull_request.issue_url
 where
     body like '<!-- drci-comment-start -->%'
-    and match(body, '\\d Pending')
+    and (match(body, '\\d Pending') or position(body, {pendingAltAttr: String}) > 0)
     and issue_comment.updated_at > now() - interval 1 month
     and issue_url like {repo: String }
     and pull_request.state = 'open'
 `;
-  const results = await queryClickhouse(query, { repo: `%${repo}%` });
+  const results = await queryClickhouse(query, {
+    repo: `%${repo}%`,
+    pendingAltAttr: ADVISOR_PENDING_ALT_ATTR,
+  });
   return results.map((v) => parseInt(v.issue_url.split("/").pop()));
 }
 

torchci/test/advisorBadge.test.ts

@@ -1,4 +1,5 @@
 import {
+  ADVISOR_PENDING_ALT,
   advisorBadgeUrl,
   ANALYZING_BADGE,
   confidenceBucket,
@@ -100,7 +101,7 @@ describe("advisorBadgeUrl", () => {
 });
 
 describe("renderInProgressLine / renderVerdictLine", () => {
-  it("in-progress: badge only, no expand", () => {
+  it("in-progress: badge only, no expand, pending alt", () => {
     const line = renderInProgressLine(
       HUD,
       "pytorch",
@@ -114,6 +115,8 @@ describe("renderInProgressLine / renderVerdictLine", () => {
     expect(line).toContain("/api/drci/advisorBadge?");
     expect(line).not.toContain("<details>");
     expect(line).toContain("/pr/pytorch/pytorch/123#42");
+    // the pending sentinel alt is what the cron matches to keep re-rendering
+    expect(line).toContain(`alt="${ADVISOR_PENDING_ALT}"`);
   });
 
   it("concluded: AI verdict text toggles a details expand with reasoning", () => {
@@ -125,6 +128,8 @@ describe("renderInProgressLine / renderVerdictLine", () => {
       "abc",
       "trunk / x / test",
       42,
+      "related",
+      0.95,
       "Line one.\n  Line two."
     );
     expect(line).toContain("<details><summary>AI verdict:");
@@ -133,6 +138,25 @@ describe("renderInProgressLine / renderVerdictLine", () => {
     // multi-line summary collapsed to one line inside the blockquote
     expect(line).toContain("Line one. Line two.");
     expect(line).not.toContain("Line one.\n");
+    // alt encodes the concluded outcome and is NOT the pending sentinel
+    expect(line).toContain('alt="AI verdict: related"');
+    expect(line).not.toContain(ADVISOR_PENDING_ALT);
+  });
+
+  it("concluded alt carries the confidence-bucketed label", () => {
+    const line = renderVerdictLine(
+      HUD,
+      "pytorch",
+      "pytorch",
+      123,
+      "abc",
+      "trunk / x / test",
+      42,
+      "not_related",
+      0.8,
+      "summary"
+    );
+    expect(line).toContain('alt="AI verdict: probably not related"');
   });
 
   it("escapes HTML in the summary so it can't break out of the expand", () => {
@@ -144,6 +168,8 @@ describe("renderInProgressLine / renderVerdictLine", () => {
       "abc",
       "trunk / x / test",
       42,
+      "related",
+      0.95,
       "</blockquote></details><img src=x onerror=alert(1)>"
     );
     expect(line).not.toContain("</blockquote></details><img");
@@ -163,7 +189,10 @@ describe("selectAdvisorLines", () => {
 
   it("prefers a finalized verdict, else in-progress, else nothing", () => {
     const verdictByKey = new Map([
-      [drciSignalKeyForJob("trunk / a / test"), { summary: "done" }],
+      [
+        drciSignalKeyForJob("trunk / a / test"),
+        { verdict: "related", confidence: 0.95, summary: "done" },
+      ],
     ]);
     const inProgress = new Set([drciSignalKeyForJob("trunk / b / test")]);