feat: strengthen timeline claim checks

Author: sarthakagrawal927

PROJECT_STATUS.md

@@ -21,7 +21,7 @@ CodeVetter is a local-first desktop workbench for checking agent-generated code.
 - Evidence Pattern Search has its first vertical slice: deterministic risk candidate packets are generated from changed files, sensitive paths, optional `ast-grep` structural matches, blast/history context, and verification signals; the top ranked candidates and procedure gates are injected into CLI review prompts, returned in review metadata, shown in the Review sidebar, can be resolved per review, persist candidate outcomes locally, write durable procedure events for QA/fix/test-evidence runs, suggest scored verification commands from prior pass/fail recency, package-manager-aware repo scripts, changed/finding file affinity, and artifacts, run explicit local verification commands with cancelable timeout-bounded stdout/stderr log artifacts, derive additional execution links from finding/browser evidence, show a fuller procedure event timeline, and are included in copied reviewer proof with outcomes, artifacts, gates, linked execution events, and blocked-on reasons.
 - The catch-rate benchmark harness can now compare stored review outputs with and without deterministic evidence search using `--evidence-comparison=with:without`, including catch-rate, precision/F1, false-positive, redundant-match, newly-caught, and regressed-ID deltas in JSON and Markdown reports.
 - Review Memory Graph has a repo-level and review-scoped first slice: Repo Unpacked now persists a deterministic local `repo_graph` artifact with package scripts, routes, Tauri commands, DB tables, tests, and decision markers; exports local graph JSON plus agent-context markdown sidecars for optional Graphify/Hunk-style interop without adding either dependency; imports graph JSON only through an explicit Repo Unpacked file action, validates CodeVetter or loose graph-shaped JSON, and renders imported graphs as non-mutating previews; CLI review results also carry a bounded local graph over changed files, evidence candidates, procedure gates, blast/history context, the review prompt includes that graph neighborhood, Review shows a compact graph panel plus selected-finding graph focus, copied reviewer proof includes both full graph and focused finding graph nodes/edges, and Review can copy a selected finding as a Hunk-style agent-context note with file/line, evidence status, local history, focused graph, and next verification actions.
-- Agent Verification Timeline has a first normalized spine: Review now builds a shared task/review/QA/evidence/claim-check/fix/worktree timeline contract, attaches bounded raw-session command anchors to evidence rows with transcript excerpts, adds a dedicated Claim check row for failed/stale command claims, explicit extracted agent claims, positive test/check claims contradicted by failed/stale command evidence, unchecked findings, unresolved post-fix QA, and fixes without same-flow reruns, attaches edit-origin anchors for fix changed files to worktree rows, renders timeline stages and anchors in the sidebar, exposes first-class jump targets for findings, files, QA artifacts, fix worktrees, command source anchors, and edited files, shows same-flow post-fix QA before/after deltas with artifact anchors on the QA row, can copy segment-scoped fix packets directly from Review/Evidence/QA/Fix/Worktree timeline rows, includes clicked-row timeline replay metadata and transcript snippets in those packets, and includes the same timeline plus source/event/artifact/jump/edit/transcript anchors in copied reviewer proof.
+- Agent Verification Timeline has a first normalized spine: Review now builds a shared task/review/QA/evidence/claim-check/fix/worktree timeline contract, attaches bounded raw-session command anchors to evidence rows with transcript excerpts, adds a dedicated Claim check row for failed/stale command claims, unknown verification-command outcomes, explicit extracted agent claims, positive test/check claims contradicted by failed/stale command evidence, unchecked findings, latest QA failures without post-fix comparison, unresolved post-fix QA, fixes without same-flow reruns, evidence-count-only loops with no executable proof, and clean loops with passed command/QA proof counts, attaches edit-origin anchors for fix changed files to worktree rows, renders timeline stages and anchors in the sidebar, exposes first-class jump targets for findings, files, QA artifacts, fix worktrees, command source anchors, and edited files, shows same-flow post-fix QA before/after deltas with artifact anchors on the QA row, can copy segment-scoped fix packets directly from Review/Evidence/QA/Fix/Worktree timeline rows, includes clicked-row timeline replay metadata and transcript snippets in those packets, and includes the same timeline plus source/event/artifact/jump/edit/transcript anchors in copied reviewer proof.
 - Codebase History Explainer has a first file-level slice: Review now builds bounded, cited "why this code exists" explanations from local commits, decision markers, recurring findings, agent notes, and command anchors, renders them in the sidebar, and includes them in copied reviewer proof.
 - AI Session Intelligence has a first local scorecard slice: indexed sessions produce a schema-versioned six-dimension scorecard with cited evidence refs, anti-gaming notes, recommendations, normalized Claude/Codex/Cursor adapter coverage summaries, production and scorecard adapter run metadata/parse warnings in `session_adapter_runs`, a compact `session_message_archive` for normalized adapter messages/tool calls, local backfill for previously indexed Claude/Codex sessions missing archive rows, FTS-backed local archive search over messages/tool calls, startup/periodic/manual archive update events, a shared raw parser adapter contract with Claude/Codex/Cursor fixtures, Claude/Codex/Cursor production indexing wired through that contract, a Tauri IPC contract, compact Roadmap dashboard panels for scorecard and source health, visible adapter run status, per-adapter run trends, and recent-run drilldowns.
 - Home now opens directly into a usage dashboard panel with Today/Week/Month/Year counters above error or roadmap noise. The Verification Workbench launcher and latest build banner live on the Roadmap page, so Evidence Search, Agent Timeline, Synthetic QA, Memory Graph, History Brief, AI Sessions, transcript replay, claim checks, replay packets, timeline fix packets, edit origins, and post-fix QA deltas remain visible without pushing usage down on app launch.
@@ -31,7 +31,7 @@ CodeVetter is a local-first desktop workbench for checking agent-generated code.
 ## Planned Next
 
 1. Continue the AI Session Intelligence PRD in `docs/PRD-AI-SESSION-INTELLIGENCE.md`: add direct filesystem-watch tailing for currently open transcript files if periodic indexing proves too coarse; Claude, Codex, and Cursor production session rows now use the normalized raw parser adapter contract, production index passes persist adapter run metadata/parse warnings plus compact message/tool-call archive rows, local backfill repairs older Claude/Codex sessions with missing archive rows, Roadmap shows latest source-health status with per-adapter trends and recent-run drilldowns, Roadmap can search the normalized archive locally, and startup/periodic/manual indexes emit archive update events.
-2. Continue the Agent Verification Timeline PRD in `docs/PRD-AGENT-VERIFICATION-TIMELINE.md`: add richer claim-vs-evidence discrepancy parsing beyond literal positive test/check claim contradictions, command/QA/evidence-count signals, and fuller multi-turn transcript replay; raw session command anchors with bounded transcript excerpts, explicit agent-claim anchors, dedicated claim-check rows, edit-origin anchors, timeline-specific jump targets, timeline-segment replay packets, same-flow post-fix QA deltas, archive search, and the Roadmap latest-build banner are now attached to visible Review/Roadmap actions and proof export.
+2. Continue the Agent Verification Timeline PRD in `docs/PRD-AGENT-VERIFICATION-TIMELINE.md`: add scope-drift/repeated-edit discrepancy parsing and fuller multi-turn transcript replay; raw session command anchors with bounded transcript excerpts, explicit agent-claim anchors, command/QA/evidence-count claim-check signals, edit-origin anchors, timeline-specific jump targets, timeline-segment replay packets, same-flow post-fix QA deltas, archive search, and the Roadmap latest-build banner are now attached to visible Review/Roadmap actions and proof export.
 3. Continue the Codebase History Explainer PRD in `docs/PRD-CODEBASE-HISTORY-EXPLAINER.md`: turn the persisted `history_brief` slice into a queryable local history graph; Repo Unpacked history brief integration and agent-context sidecar export are now implemented.
 4. Curate 20-30 real public agent-generated PR benchmark cases with hand-labeled ground truth before making external catch-rate claims.
 5. Add benchmark fields for unverified-fix count and time/cost impact once review artifacts capture those values consistently.

apps/desktop/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@code-reviewer/desktop",
-  "version": "1.1.48",
+  "version": "1.1.49",
   "private": true,
   "scripts": {
     "dev": "lsof -ti:1420 | xargs kill -9 2>/dev/null; vite",

apps/desktop/src-tauri/tauri.conf.json

@@ -2,7 +2,7 @@
   "$schema": "https://raw.githubusercontent.com/tauri-apps/tauri/dev/crates/tauri-utils/schema.json",
   "identifier": "com.codevetter.desktop",
   "productName": "CodeVetter",
-  "version": "1.1.48",
+  "version": "1.1.49",
   "build": {
     "beforeDevCommand": "npm run dev",
     "beforeBuildCommand": "npm run build",

apps/desktop/src/lib/review-proof.test.ts

@@ -440,6 +440,138 @@ describe("buildReviewerProofMarkdown", () => {
     assert.equal(contradictedClaim?.jump?.kind, "command_source");
   });
 
+  it("flags unknown verification command outcomes as claim-check proof gaps", () => {
+    const timeline = buildVerificationTimeline({
+      runId: "review-unknown-command",
+      review: {
+        findingsCount: 1,
+      },
+      evidenceCounts: {
+        fixed: 0,
+        reproduced: 1,
+        notReproduced: 0,
+      },
+      history: {
+        command_signals: [
+          {
+            agent: "codex",
+            date: "2026-06-12T00:00:00Z",
+            command: "npm run test:checkout",
+            status: "unknown",
+            source: "raw_session",
+            source_path: "/tmp/session.jsonl",
+            source_line: 22,
+            event_id: "cmd-unknown",
+            session_id: "session-unknown-command",
+          },
+        ],
+      },
+    });
+
+    const claimCheckStep = timeline.find((item) => item.id === "claim-check");
+    assert.equal(claimCheckStep?.status, "active");
+    assert.match(claimCheckStep?.detail ?? "", /0 blocking, 1 need proof/);
+    assert.equal(
+      claimCheckStep?.anchors?.[0]?.label,
+      "Unverified command outcome: npm run test:checkout",
+    );
+    assert.equal(claimCheckStep?.anchors?.[0]?.source, "raw_session");
+    assert.equal(claimCheckStep?.anchors?.[0]?.jump?.kind, "command_source");
+  });
+
+  it("blocks claim checks when latest QA is still failing without a comparison", () => {
+    const timeline = buildVerificationTimeline({
+      runId: "review-latest-qa-failed",
+      review: {
+        findingsCount: 1,
+      },
+      qa: {
+        latest: {
+          pass: false,
+          runnerType: "repo_playwright",
+          route: "/checkout",
+          goal: "Complete checkout",
+          durationMs: 900,
+          screenshotPath: "artifacts/latest-fail.png",
+          artifacts: ["artifacts/latest-fail.log"],
+        },
+      },
+      evidenceCounts: {
+        fixed: 1,
+        reproduced: 0,
+        notReproduced: 0,
+      },
+    });
+
+    const claimCheckStep = timeline.find((item) => item.id === "claim-check");
+    assert.equal(claimCheckStep?.status, "blocked");
+    assert.match(claimCheckStep?.detail ?? "", /1 blocking, 0 need proof/);
+    assert.equal(claimCheckStep?.anchors?.[0]?.label, "Latest QA still failing: /checkout");
+    assert.equal(claimCheckStep?.anchors?.[0]?.artifact, "artifacts/latest-fail.png");
+    assert.equal(claimCheckStep?.anchors?.[0]?.jump?.kind, "artifact");
+  });
+
+  it("flags evidence-count-only loops that lack executable proof", () => {
+    const timeline = buildVerificationTimeline({
+      runId: "review-thin-proof",
+      review: {
+        findingsCount: 2,
+      },
+      evidenceCounts: {
+        fixed: 1,
+        reproduced: 1,
+        notReproduced: 0,
+      },
+    });
+
+    const claimCheckStep = timeline.find((item) => item.id === "claim-check");
+    assert.equal(claimCheckStep?.status, "active");
+    assert.match(claimCheckStep?.detail ?? "", /0 blocking, 1 need proof/);
+    assert.equal(
+      claimCheckStep?.anchors?.[0]?.label,
+      "Executable proof missing: 2 evidence statuses for 2 findings",
+    );
+    assert.equal(claimCheckStep?.anchors?.[0]?.source, "review:evidence-strength");
+    assert.match(claimCheckStep?.anchors?.[0]?.contextExcerpt?.join("\n") ?? "", /0 passed verification commands/);
+  });
+
+  it("recognizes passed command proof when claim gaps are clean", () => {
+    const timeline = buildVerificationTimeline({
+      runId: "review-good-loop",
+      review: {
+        findingsCount: 1,
+      },
+      evidenceCounts: {
+        fixed: 0,
+        reproduced: 1,
+        notReproduced: 0,
+      },
+      history: {
+        command_signals: [
+          {
+            agent: "codex",
+            date: "2026-06-12T00:00:00Z",
+            command: "npm run test:checkout",
+            status: "passed",
+            source: "raw_session",
+            source_path: "/tmp/session.jsonl",
+            source_line: 30,
+            event_id: "cmd-passed",
+            session_id: "session-good-loop",
+          },
+        ],
+      },
+    });
+
+    const claimCheckStep = timeline.find((item) => item.id === "claim-check");
+    assert.equal(claimCheckStep?.status, "done");
+    assert.match(
+      claimCheckStep?.detail ?? "",
+      /No claim\/evidence gaps detected · 1 passed verification command/,
+    );
+    assert.equal(claimCheckStep?.anchors?.length, 0);
+  });
+
   it("copies concrete command evidence into finding handoff proof", () => {
     const history = new Map<number, HistoryFindingSummary>();
     history.set(0, {

apps/desktop/src/lib/review-proof.ts

@@ -449,6 +449,20 @@ function isPositiveVerificationClaim(claim: string): boolean {
   ].some((pattern) => pattern.test(normalized));
 }
 
+function isVerificationCommandLabel(label: string): boolean {
+  const normalized = label.trim().toLowerCase();
+  return [
+    /\b(?:npm|pnpm|yarn|bun)\s+(?:run\s+)?(?:test|lint|build|typecheck|check|e2e|playwright)\b/,
+    /\b(?:cargo\s+(?:test|clippy|build)|go\s+test|pytest|vitest|jest|tsc|eslint|playwright|cypress)\b/,
+    /\b(?:test|lint|build|typecheck|check|e2e|qa|ci)\b/,
+  ].some((pattern) => pattern.test(normalized));
+}
+
+function latestQaArtifact(run: NonNullable<VerificationTimelineInput["qa"]>["latest"]): string | null {
+  if (!run) return null;
+  return run.screenshotPath ?? run.artifacts?.[0] ?? null;
+}
+
 function buildClaimCheckTimelineAnchors(
   input: VerificationTimelineInput,
   commandAnchors: VerificationTimelineAnchor[],
@@ -459,6 +473,12 @@ function buildClaimCheckTimelineAnchors(
   const runId = input.runId?.trim() || "active-review";
   const findingsCount = Math.max(0, input.review?.findingsCount ?? 0);
   const uncheckedCount = Math.max(0, findingsCount - evidenceTotal);
+  const passedVerificationCommandCount = commandAnchors.filter(
+    (anchor) => anchor.status === "passed" && isVerificationCommandLabel(anchor.label),
+  ).length;
+  const successfulQaProofCount =
+    (input.qa?.latest?.pass ? 1 : 0) +
+    (qaComparison && qaComparisonStatusToTimelineStatus(qaComparison.status) === "done" ? 1 : 0);
 
   commandAnchors
     .filter((anchor) => anchor.status === "failed" || anchor.status === "stale")
@@ -473,6 +493,21 @@ function buildClaimCheckTimelineAnchors(
       });
     });
 
+  commandAnchors
+    .filter((anchor) => anchor.status === "unknown" && isVerificationCommandLabel(anchor.label))
+    .slice(0, 2)
+    .forEach((anchor) => {
+      anchors.push({
+        ...anchor,
+        id: `claim:unknown-command:${anchor.id}`,
+        label: `Unverified command outcome: ${anchor.label}`,
+        status: "unknown",
+        contextExcerpt: anchor.contextExcerpt?.length
+          ? anchor.contextExcerpt
+          : ["Command was observed without a pass/fail status; rerun it or attach its log before trusting the claim."],
+      });
+    });
+
   const contradictingCommand = commandAnchors.find(
     (anchor) => anchor.status === "failed" || anchor.status === "stale",
   );
@@ -513,6 +548,27 @@ function buildClaimCheckTimelineAnchors(
     });
   }
 
+  if (input.qa?.latest && !input.qa.latest.pass && !qaComparison) {
+    const artifact = latestQaArtifact(input.qa.latest);
+    anchors.push({
+      id: `${runId}:claim:latest-qa-failed`,
+      label: `Latest QA still failing: ${input.qa.latest.route ?? input.qa.latest.goal}`,
+      source: `qa:${input.qa.latest.runnerType}`,
+      status: "failed",
+      sourcePath: artifact,
+      eventId: `${runId}:claim:latest-qa-failed`,
+      sessionId: runId,
+      artifact,
+      jump: artifact
+        ? {
+          kind: "artifact",
+          label: "Open latest QA artifact",
+          path: artifact,
+        }
+        : null,
+    });
+  }
+
   if (qaComparison) {
     const status = qaComparisonStatusToTimelineStatus(qaComparison.status);
     if (status !== "done") {
@@ -558,6 +614,26 @@ function buildClaimCheckTimelineAnchors(
     });
   }
 
+  if (
+    anchors.length === 0 &&
+    findingsCount > 0 &&
+    evidenceTotal >= findingsCount &&
+    passedVerificationCommandCount + successfulQaProofCount === 0
+  ) {
+    anchors.push({
+      id: `${runId}:claim:executable-proof-missing`,
+      label: `Executable proof missing: ${evidenceTotal} evidence status${evidenceTotal === 1 ? "" : "es"} for ${findingsCount} finding${findingsCount === 1 ? "" : "s"}`,
+      source: "review:evidence-strength",
+      status: "unknown",
+      contextExcerpt: [
+        `${input.evidenceCounts.reproduced} reproduced, ${input.evidenceCounts.fixed} fixed, ${input.evidenceCounts.notReproduced} not reproduced`,
+        "0 passed verification commands, 0 passing QA proofs",
+      ],
+      eventId: `${runId}:claim:executable-proof-missing`,
+      sessionId: runId,
+    });
+  }
+
   return anchors
     .sort((a, b) => statusRank(a.status) - statusRank(b.status))
     .slice(0, 4);
@@ -722,6 +798,20 @@ export function buildVerificationTimeline(
     evidenceTotal,
   );
   const failedCommandCount = commandAnchors.filter((anchor) => anchor.status === "failed").length;
+  const passedVerificationCommandCount = commandAnchors.filter(
+    (anchor) => anchor.status === "passed" && isVerificationCommandLabel(anchor.label),
+  ).length;
+  const successfulQaProofCount =
+    (latestQa?.pass ? 1 : 0) +
+    (qaComparison && qaComparisonStatusToTimelineStatus(qaComparison.status) === "done" ? 1 : 0);
+  const proofSignalDetail = [
+    passedVerificationCommandCount > 0
+      ? `${passedVerificationCommandCount} passed verification command${passedVerificationCommandCount === 1 ? "" : "s"}`
+      : null,
+    successfulQaProofCount > 0
+      ? `${successfulQaProofCount} QA proof${successfulQaProofCount === 1 ? "" : "s"}`
+      : null,
+  ].filter(Boolean).join(", ");
   const selectedFindingIndex = input.review?.selectedFindingIndex ?? null;
   const firstFindingPath = input.review?.firstFindingPath?.trim();
   const firstFindingLine = input.review?.firstFindingLine ?? null;
@@ -796,7 +886,7 @@ export function buildVerificationTimeline(
   const claimCheckDetail = claimCheckAnchors.length > 0
     ? `${blockedClaimCount} blocking, ${pendingClaimCount} need proof`
     : claimCheckStatus === "done"
-      ? "No claim/evidence gaps detected"
+      ? `No claim/evidence gaps detected${proofSignalDetail ? ` · ${proofSignalDetail}` : ""}`
       : "No claims checked yet";
   const claimCheckJump = claimCheckAnchors.find((anchor) => anchor.jump)?.jump ?? null;