torchci: render AI advisor verdict inline in the Dr.CI comment (#8202)

Surfaces the AI CI Advisor verdict directly in the Dr.CI PR comment
(task #132 part 3). Under each **NEW FAILURE** / **UNCLASSIFIED
FAILURE**, an `AI verdict:` line shows a colored status pill; once the
verdict is finalized, the pill becomes a collapsible expand with the
reasoning.

### What it looks like
`AI verdict:` (plain text — toggles the expand) followed by a pill
linking to HUD:

- **In-progress** (dispatched, no verdict yet): just the pill
(`analyzing`), no expand.
- **Concluded**: the `AI verdict:` text toggles a `<details>` whose
summary is the pill and whose body is the advisor's reasoning + a HUD
link.

Confidence is encoded **in the wording** (never a number) and the pill
color rides a green → yellow → red gradient, with low confidence pulling
toward yellow:

| confidence | not-related pole | related pole |
|---|---|---|
| high (≥0.89) | `not related` (green) | `related` (red) |
| med (0.70–0.89) | `probably not related` | `probably related` |
| low (≤0.70) | `not related (uncertain)` (yellow) | `related
(uncertain)` (yellow) |

`garbage` → gray; `unsure`/unknown → `inconclusive`. Rendering mock:
pytorch/ciforge#517 (final = the "Option C"
comment).

### How it works
- **New SVG endpoint** `pages/api/drci/advisorBadge.ts`, keyed on
`(owner, repo, sha, job)`. The comment's `<img>` points here, so the
pill can flip *analyzing → verdict* server-side **without rewriting the
comment** (which only happens on the ~15-min Dr.CI cron).
**State-dependent caching**: short TTL (`max-age=60`) while in-progress
so the proxy (GitHub camo) re-fetches and the badge updates soon after
the verdict lands; **immutable** long TTL once final (a verdict never
changes for a fixed `sha`+`job`) → near-zero load afterward.
- `lib/advisor/advisorBadge.ts` (pure): verdict+confidence →
label/color, flat SVG render, badge-URL + line-HTML builders, and the
per-job line selector. Fully unit-tested.
- `lib/advisor/advisorComment.ts`: reads finalized verdicts
(`advisor_verdicts_for_pr`) + in-progress dispatch state
(`advisor_dispatch_states`) and builds the per-job lines. Verdicts match
jobs by **exact signal-key equality** (`dr_ci_${jobName}`, the same key
auto-dispatch wrote) — no fuzzy matching.
- `drci.ts`: minimal additive hook — build the map (wrapped in try/catch
so an advisor/ClickHouse error can never break the comment), thread it
into the NEW FAILURE / UNCLASSIFIED sections.
- New saved query `advisor_verdict_for_job` (indexed point lookup on the
verdict table's ORDER BY prefix).

### Rollout
Gated behind a **new `DRCI_ADVISOR_COMMENT_ENABLED` flag (default off)**
so it ships dark — flip it in Vercel prod to enable. **Display-only**:
never edits merge behavior. (Note: GitHub's comment sanitizer strips
`target=_blank`, so the pill link opens in the same tab — unavoidable
without JS.)

### Test plan
- `tsc --noEmit` clean; `prettier`/`sqlfluff` clean.
- New `test/advisorBadge.test.ts` (jest, node ≥18 — runs in CI) covers
the confidence buckets, gradient mapping, SVG render, badge-URL
encoding, in-progress vs concluded line rendering, and the
verdict/in-progress/none line selection.
- Manual: enable the flag on a deployment and confirm the line renders
on a real pytorch/pytorch PR with a known verdict.

---------

Co-authored-by: Ivan <izaitsevfb@meta.com>

Author: izaitsevfb

torchci/clickhouse_queries/advisor_verdict_for_job/params.json

@@ -0,0 +1,8 @@
+{
+  "params": {
+    "repo": "String",
+    "sha": "String",
+    "signalKey": "String"
+  },
+  "tests": []
+}

torchci/clickhouse_queries/advisor_verdict_for_job/query.sql

@@ -0,0 +1,16 @@
+-- Latest AI advisor verdict for one (repo, suspect commit, signal_key). Backs
+-- the Dr.CI advisor badge endpoint. Filters the verdict table's ORDER BY prefix
+-- (repo, suspect_commit, signal_key, timestamp), so it reads only the relevant
+-- granules.
+SELECT
+    verdict,
+    confidence
+FROM
+    misc.autorevert_advisor_verdicts
+WHERE
+    repo = {repo: String}
+    AND suspect_commit = {sha: String}
+    AND signal_key = {signalKey: String}
+ORDER BY
+    timestamp DESC
+LIMIT 1

torchci/lib/advisor/advisorBadge.ts

@@ -0,0 +1,249 @@
+// Pure rendering helpers for the AI CI Advisor verdict surfaced in the Dr.CI
+// comment (no server-only imports, so this is unit-testable and importable
+// anywhere). Two outputs:
+//   - renderBadgeSvg: the colored status pill served by the advisorBadge API
+//     route (the <img> in the comment points at that route so it can flip
+//     analyzing -> verdict server-side without rewriting the comment).
+//   - selectAdvisorLines / renderVerdictLine / renderInProgressLine: the
+//     "AI verdict:" line emitted under each new failure in the comment.
+
+import _ from "lodash";
+
+// The signal_key prefix for HUD-originated (Dr.CI) advisor dispatches. Mirrors
+// signalKeyForJob() in advisorDispatch.ts; duplicated here to keep this module
+// free of any server-only imports.
+export const DRCI_SIGNAL_KEY_PREFIX = "dr_ci_";
+
+export function drciSignalKeyForJob(fullJobName: string): string {
+  return `${DRCI_SIGNAL_KEY_PREFIX}${fullJobName}`;
+}
+
+export interface AdvisorBadge {
+  label: string;
+  // Hex fill, e.g. "#2da44e".
+  color: string;
+  // Use dark text for light (yellow-ish) fills that wash out white text.
+  darkText: boolean;
+}
+
+// Dispatched but no verdict yet.
+export const ANALYZING_BADGE: AdvisorBadge = {
+  label: "analyzing",
+  color: "#9f9f9f",
+  darkText: false,
+};
+
+// No verdict and no in-progress dispatch found (race / stale comment).
+export const PENDING_BADGE: AdvisorBadge = {
+  label: "pending",
+  color: "#9f9f9f",
+  darkText: false,
+};
+
+export type ConfidenceBucket = "high" | "med" | "low";
+
+// Confidence is shown as a word baked into the label, never a number:
+//   high >= 0.89, med (0.70, 0.89), low <= 0.70.
+export function confidenceBucket(confidence: number): ConfidenceBucket {
+  if (confidence >= 0.89) return "high";
+  if (confidence > 0.7) return "med";
+  return "low";
+}
+
+// Map a verdict + confidence to a label and a color on a green -> yellow -> red
+// gradient, with "uncertain" (low confidence) pulling toward yellow. The "AI
+// verdict:" prefix lives in the comment line, so the badge carries only the
+// status text.
+export function verdictBadge(
+  verdict: string,
+  confidence: number
+): AdvisorBadge {
+  const v = (verdict || "").toLowerCase();
+
+  if (v === "garbage") {
+    return { label: "garbage", color: "#6e7781", darkText: false };
+  }
+  if (v === "unsure") {
+    return { label: "inconclusive", color: "#8b949e", darkText: false };
+  }
+
+  const bucket = confidenceBucket(confidence);
+
+  // `related` is the context-neutral successor to `revert`; treat both as the
+  // "related to this PR" pole.
+  if (v === "related" || v === "revert") {
+    if (bucket === "high") {
+      return { label: "related", color: "#d1242f", darkText: false };
+    }
+    if (bucket === "med") {
+      return { label: "probably related", color: "#e8702a", darkText: false };
+    }
+    return { label: "related (uncertain)", color: "#e0a82e", darkText: true };
+  }
+
+  if (v === "not_related") {
+    if (bucket === "high") {
+      return { label: "not related", color: "#2da44e", darkText: false };
+    }
+    if (bucket === "med") {
+      return {
+        label: "probably not related",
+        color: "#94c11f",
+        darkText: true,
+      };
+    }
+    return {
+      label: "not related (uncertain)",
+      color: "#c9b81a",
+      darkText: true,
+    };
+  }
+
+  // Unknown verdict value -> treat as inconclusive rather than guessing a pole.
+  return { label: "inconclusive", color: "#8b949e", darkText: false };
+}
+
+// A minimal flat single-segment SVG pill (shields "flat" look, no left label).
+export function renderBadgeSvg(badge: AdvisorBadge): string {
+  const text = badge.label;
+  // Approximate text width; Verdana ~6.5px/char at 11px, plus horizontal pad.
+  const width = Math.max(46, Math.round(text.length * 6.5) + 16);
+  const textColor = badge.darkText ? "#33333a" : "#ffffff";
+  const safe = _.escape(text);
+  return `<svg xmlns="http://www.w3.org/2000/svg" width="${width}" height="20" role="img" aria-label="${safe}">
+<title>${safe}</title>
+<rect width="${width}" height="20" rx="3" fill="${badge.color}"/>
+<text x="${(width / 2).toFixed(
+    1
+  )}" y="14" fill="${textColor}" font-family="Verdana,DejaVu Sans,Geneva,sans-serif" font-size="11" text-anchor="middle">${safe}</text>
+</svg>`;
+}
+
+// Absolute URL of the badge image for one (repo, sha, job). Camo fetches it
+// server-side; the route resolves the live state, so the same URL flips from
+// analyzing to the verdict without the comment changing.
+export function advisorBadgeUrl(
+  hudBaseUrl: string,
+  owner: string,
+  repo: string,
+  sha: string,
+  jobName: string
+): string {
+  const qs = new URLSearchParams({ owner, repo, sha, job: jobName });
+  return `${hudBaseUrl}/api/drci/advisorBadge?${qs.toString()}`;
+}
+
+function hudPrUrl(
+  hudBaseUrl: string,
+  owner: string,
+  repo: string,
+  prNumber: number,
+  jobId: number
+): string {
+  return `${hudBaseUrl}/pr/${owner}/${repo}/${prNumber}#${jobId}`;
+}
+
+// In-progress line: just the badge (no expand), linked to HUD. The pill flips
+// to the verdict in place once the advisor finishes.
+export function renderInProgressLine(
+  hudBaseUrl: string,
+  owner: string,
+  repo: string,
+  prNumber: number,
+  sha: string,
+  jobName: string,
+  jobId: number
+): string {
+  const badge = advisorBadgeUrl(hudBaseUrl, owner, repo, sha, jobName);
+  const link = hudPrUrl(hudBaseUrl, owner, repo, prNumber, jobId);
+  return `    AI verdict: <a href="${link}"><img src="${badge}"></a>\n`;
+}
+
+// Concluded line: "AI verdict:" plain text toggles the expand; the badge links
+// to HUD; the reasoning lives inside the expand.
+export function renderVerdictLine(
+  hudBaseUrl: string,
+  owner: string,
+  repo: string,
+  prNumber: number,
+  sha: string,
+  jobName: string,
+  jobId: number,
+  summary: string
+): string {
+  const badge = advisorBadgeUrl(hudBaseUrl, owner, repo, sha, jobName);
+  const link = hudPrUrl(hudBaseUrl, owner, repo, prNumber, jobId);
+  // The advisor summary is model-generated from (attacker-influenceable) PR
+  // content, so HTML-escape it before embedding in the comment: collapse
+  // newlines (can't break the blockquote) and neutralize markup so it can't
+  // close the <details>/<blockquote> or inject tags.
+  const oneLine = _.escape((summary || "").replace(/\s*\n\s*/g, " ").trim());
+  return (
+    `  <details><summary>AI verdict: <a href="${link}"><img src="${badge}"></a></summary><blockquote>\n\n` +
+    `  ${oneLine}\n\n` +
+    `  <a href="${link}">Full reasoning on HUD &rarr;</a>\n` +
+    `  </blockquote></details>\n`
+  );
+}
+
+// Minimal shapes the pure selector needs (subset of RecentWorkflowsData /
+// AdvisorVerdict) so this module stays import-light.
+export interface AdvisorLineJob {
+  id: number;
+  name: string;
+}
+export interface AdvisorLineVerdict {
+  summary: string;
+}
+
+// Decide the per-job line: a finalized verdict wins; otherwise an in-progress
+// dispatch ('dispatching'/'dispatched') shows the analyzing badge; otherwise no
+// line. Pure so it is unit-testable without ClickHouse. Returns job.id -> HTML.
+export function selectAdvisorLines(
+  hudBaseUrl: string,
+  owner: string,
+  repo: string,
+  prNumber: number,
+  headSha: string,
+  jobs: AdvisorLineJob[],
+  verdictByKey: Map<string, AdvisorLineVerdict>,
+  inProgressKeys: Set<string>
+): Map<number, string> {
+  const out = new Map<number, string>();
+  for (const job of jobs) {
+    if (!job.name) continue;
+    const key = drciSignalKeyForJob(job.name);
+    const verdict = verdictByKey.get(key);
+    if (verdict) {
+      out.set(
+        job.id,
+        renderVerdictLine(
+          hudBaseUrl,
+          owner,
+          repo,
+          prNumber,
+          headSha,
+          job.name,
+          job.id,
+          verdict.summary
+        )
+      );
+      continue;
+    }
+    if (inProgressKeys.has(key)) {
+      out.set(
+        job.id,
+        renderInProgressLine(
+          hudBaseUrl,
+          owner,
+          repo,
+          prNumber,
+          headSha,
+          job.name,
+          job.id
+        )
+      );
+    }
+  }
+  return out;
+}

torchci/lib/advisor/advisorComment.ts

@@ -0,0 +1,88 @@
+// Server-side glue for the inline AI advisor verdict lines in the Dr.CI comment.
+// Reads finalized verdicts + in-progress dispatch state from ClickHouse, then
+// delegates the (pure) line selection/rendering to lib/advisor/advisorBadge.
+//
+// The verdict's signal_key is exactly `dr_ci_${jobName}` (what auto-dispatch
+// wrote), so verdicts match jobs by exact signal-key equality -- no fuzzy
+// matchVerdictToJob needed for the PR side.
+
+import {
+  AdvisorLineVerdict,
+  selectAdvisorLines,
+} from "lib/advisor/advisorBadge";
+import { isAdvisorEnabled } from "lib/advisor/advisorConfig";
+import {
+  readDispatchStates,
+  signalKeyForJob,
+} from "lib/advisor/advisorDispatch";
+import {
+  AdvisorVerdictRow,
+  deduplicateVerdicts,
+} from "lib/advisorVerdictUtils";
+import { queryClickhouseSaved } from "lib/clickhouse";
+import { RecentWorkflowsData } from "lib/types";
+
+// Gate the inline verdict rendering behind its own flag so it ships dark and
+// can be enabled per deployment (Vercel env var), independently of the
+// auto-dispatch flag. Display-only, so it doesn't also require VERCEL_ENV
+// (unlike auto-dispatch, which fires real workflow_dispatches).
+export function advisorCommentEnabled(owner: string, repo: string): boolean {
+  return (
+    process.env.DRCI_ADVISOR_COMMENT_ENABLED === "true" &&
+    isAdvisorEnabled(owner, repo)
+  );
+}
+
+/**
+ * Build the per-job "AI verdict:" line for a PR's new/unclassified failures.
+ * Returns job.id -> rendered HTML (empty map when the comment flag is off, the
+ * repo isn't advisor-enabled, or there are no jobs). The caller wraps this so a
+ * ClickHouse error can never break the Dr.CI comment.
+ */
+export async function buildAdvisorVerdictLines(
+  hudBaseUrl: string,
+  owner: string,
+  repo: string,
+  prNumber: number,
+  headSha: string,
+  jobs: RecentWorkflowsData[]
+): Promise<Map<number, string>> {
+  if (!advisorCommentEnabled(owner, repo) || jobs.length === 0) {
+    return new Map();
+  }
+
+  // Finalized verdicts for this PR, keyed by signal_key for the head commit.
+  const verdictRows = (await queryClickhouseSaved("advisor_verdicts_for_pr", {
+    repo: `${owner}/${repo}`,
+    prNumber,
+  })) as AdvisorVerdictRow[];
+  const verdictByKey = new Map<string, AdvisorLineVerdict>();
+  for (const v of deduplicateVerdicts(verdictRows)) {
+    if (v.sha === headSha) {
+      verdictByKey.set(v.signalKey, { summary: v.summary });
+    }
+  }
+
+  // In-progress dispatches (dispatching/dispatched) for the head commit.
+  const signalKeys = jobs
+    .filter((j) => j.name)
+    .map((j) => signalKeyForJob(j.name));
+  const states = await readDispatchStates(owner, repo, headSha, signalKeys);
+  const inProgressKeys = new Set<string>();
+  for (const [key, st] of states) {
+    if (st.state === "dispatching" || st.state === "dispatched") {
+      inProgressKeys.add(key);
+    }
+  }
+
+  return selectAdvisorLines(
+    hudBaseUrl,
+    owner,
+    repo,
+    prNumber,
+    headSha,
+    jobs.map((j) => ({ id: j.id, name: j.name })),
+    verdictByKey,
+    inProgressKeys
+  );
+}