Harness should fail fast on authentication errors instead of retrying

[fullsend-ai-retro]

What happened

On PR #2052, the review agent was dispatched (run 27188931660). Claude Code CLI exited immediately (81ms) with authentication_failed and apiKeySource: "none" — zero tokens consumed. The harness treated this as a normal agent failure and retried (iteration 2), which failed identically (58ms, same auth error). After 2 failed iterations, the workflow exited with validation failed after 2 iteration(s). Total wall time wasted on sandbox setup and teardown across both iterations was ~2 minutes, plus the cognitive cost of a human seeing a 'Failure' status on the PR.

What could go better

Authentication failures are infrastructure issues (credential misconfiguration, OIDC token not forwarded into sandbox, Vertex AI setup problems). They will never self-resolve by retrying within the same workflow run. The harness currently treats all agent failures uniformly and retries up to the configured iteration limit. This wastes runner time and produces misleading failure statuses. I am confident this is a systemic issue — any transient credential delivery failure would trigger the same pointless retry. Issue #1032 proposes a pre-flight auth check which would help catch some cases, but even with pre-flight checks, auth can fail at runtime (e.g., token expiry mid-run per #750). A complementary post-failure classification is needed.

Proposed change

In the harness retry loop (the iteration logic that re-runs the agent on validation failure), add exit-reason classification before retrying. When the agent's JSONL transcript or exit output contains authentication_failed or apiKeySource: "none", classify it as a non-retryable infrastructure failure and exit immediately with a clear error message (e.g., Error: agent authentication failed — not retrying. Check credential configuration.). This should be implemented in the harness runner code that handles iteration/retry decisions. The classification could be extended over time to cover other non-retryable failures (e.g., sandbox creation failures, network policy blocks).

Validation criteria

On the next occurrence of an authentication failure in any agent run, the harness should: (1) not retry the agent, (2) exit within seconds of the first failure, and (3) include 'authentication' or 'credential' in the error message posted to the PR. Verify by searching workflow logs for runs where both iterations failed with identical auth errors — these should no longer occur.

---

Generated by retro agent from #2052

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 8:48 AM UTC · Completed 8:51 AM UTC
Commit: ba204cb · View workflow run →

[fullsend-ai-triage]

Triage Summary

Category: Feature — non-retryable failure classification in the harness retry loop
Severity: Medium
Related issues: #1032 (pre-flight Vertex AI auth check — complementary, not duplicate), #750 (mid-run token expiry)

Problem

The harness iteration loop retries agent failures uniformly without classifying the failure reason. Authentication failures — which are infrastructure issues that will never self-resolve within the same workflow run — get retried uselessly, wasting ~2 minutes of runner time and producing misleading PR statuses.

Root Cause

The retry loop in internal/cli/run.go (lines 648-783) retries whenever validation fails. The transcript parser (internal/runtime/claude_transcript.go) already extracts error Subtype and ErrorMessage from agent JSONL output, but this information is not used to inform retry decisions.

Recommended Approach

1. Add a classification function in internal/runtime/ that identifies non-retryable errors (starting with authentication_failed / apiKeySource: "none").
2. In the retry decision point (~line 773 of run.go), check transcript errors before retrying. If classified as non-retryable, break the loop with a clear error message.
3. The unused FeedbackMode field in ValidationLoop (harness.go:187) may serve as an extension point.

Proposed Test

Test: harness does not retry on authentication failures

Setup: Mock transcript JSONL with result event containing
  subtype: "authentication_failed" or error message with
  "apiKeySource: \"none\""

Assertions:
  1. Classification function returns non-retryable
  2. Iteration loop exits after first failure (iteration count = 1)
  3. Error output contains "authentication" or "credential"
  4. Non-auth failures still retry normally (control case)

---

Labels: Issue targets harness retry logic (component/harness) and is a feature request for non-retryable failure classification.

[fullsend-ai-prioritize]

RICE Priority Score: 2.4

Score breakdown

Dimension	Score	Reasoning
Reach	1.5	This affects all users of the agent harness system across orgs. Any user who encounters an authentication failure (credential misconfiguration, OIDC token issues, token expiry) hits this problem. However, auth failures are not constant — they occur during infrastructure misconfigurations or transient credential delivery issues, so not every run is affected. Score of 1.5 reflects multiple customers/orgs being impacted when it does occur.
Impact	1	Medium impact. The current behavior wastes ~2 minutes of runner time per occurrence and produces misleading 'Failure' statuses on PRs, causing cognitive overhead for developers. However, a workaround exists (recognize the auth failure from logs and re-trigger manually), and no data is lost. The improvement is noticeable but not blocking a core workflow.
Confidence	0.8	High confidence. The issue is extremely well-described with specific log output, timing data, and a concrete proposed solution. The triage comment identifies exact code locations (run.go lines 648-783, claude_transcript.go) and confirms the transcript parser already extracts the needed error information. Scope is clear and bounded.
Effort	0.5	Simple, well-scoped change. The triage analysis shows the transcript parser already extracts error subtypes — the work is adding a classification function and a check before the retry decision point (~line 773 of run.go). Estimated at a few files touched with clear test criteria. No architectural changes needed.

Formula: (1.5 x 1 x 0.8) / 0.5 = 2.4