feat: detect inline model errors (T: [resource_exhausted] / Connection stalled) and surface with pulsing session-row indicator

[heavygee]

Problem

When Cursor's upstream model fails mid-turn, cursor-agent does not exit. Instead it emits the error as a normal assistant message (Error: T: [canceled], Error: T: Connection stalled, Error: T: [resource_exhausted], etc.) and immediately sends a ready event. HAPI treats this as a clean turn completion.

If the agent's prior message claimed completion ("All done", "Committed", "Done."), the operator sees that text, sees the green ready dot, and walks away. The actual tool call never completed.

This is distinct from the quota-stderr exit-1 case (where the agent process dies with a non-zero exit code). Here the process stays alive; the error surfaces only as text in the message stream.

Observed patterns (fleet evidence, 2026-06-08 to -10)

Session name	Error text in agent/message	Provider	Mid-flight tool
android watch	Error: T: [resource_exhausted] Error	Bedrock Claude	idle
valutwarden	Error: T: [resource_exhausted] Error (x2)	Bedrock Claude	idle
enophone & DJ	Error: T: [canceled] http/2 stream closed with error code CANCEL (0x8)	Bedrock Claude	mid-reasoning
coolify	Error: T: Connection stalled	Vertex Claude	mid-Edit File after agent said "All done"

Gemini flavor (separate runner path, same UX failure):

• Gemini prompt failed: Failed to generate content: The input token count exceeds the maximum number of tokens allowed 1048576.
• Gemini prompt failed: You have exhausted your capacity on this model.

Root cause

cursor-agent (and the Gemini runner) surface gRPC/upstream errors as assistant text events rather than process exits. HAPI's current event loop has no intercept between "receive agent/message" and "persist + emit ready". There is no classification, no metadata update, and no distinct UI state for a failed turn vs a clean one.

---

Proposed fix

1. Inline message classifier

Add a classifier that runs on every incoming agent/message event before it is persisted. Recognises the following patterns:

Pattern	Kind	Transient
^Error: T: \[resource_exhausted\]	quota_exhausted	no
^Error: T: \[canceled\]	canceled	yes
^Error: T: \[deadline_exceeded\]	deadline_exceeded	yes
^Error: T: \[unavailable\]	unavailable	yes
^Error: T: Connection stalled	connection_stalled	yes
Gemini prompt failed: .*token count exceeds	context_window	no
Gemini prompt failed: .*exhausted your capacity	capacity_exhausted	no

Returns null for anything not matching (benign messages pass through unchanged).

2. Session metadata update

When a match is found, set session.metadata.lastModelError:

{
  kind: string,           // one of the kinds above
  transient: boolean,
  rawSnippet: string,     // first 400 chars of the error text
  atTs: number,
  priorAssistantClaimsDone: boolean,  // was the prior assistant message "All done" / "Committed" / "Done." etc?
  acknowledgedAt?: number,
}

3. Synthetic modelError session event

Emit a modelError event distinct from message so the web client and notification pipeline can react without parsing message text.

4. In-session banner

A persistent banner inside the session view when lastModelError is set and not acknowledged:

⚠ MODEL ERROR — connection_stalled (transient)
The agent said "All done" but its last tool call (Edit File) did not complete.
The work is likely INCOMPLETE. Verify before trusting this session's output.

[Retry last message]   [Dismiss]   [View raw error]

Copy is harsher when priorAssistantClaimsDone === true. Banner clears on Dismiss (sets acknowledgedAt).

5. Pulsing amber dot in session list row

When lastModelError is unacknowledged, the session's ready indicator switches from green to pulsing amber (CSS ping / pulse animation). This gives a fleet-level visual signal without requiring the operator to be inside the session. Stops pulsing on Dismiss.

---

What this does NOT include

• The stderr/exit-1 quota crash path (separate issue; different code path in cursorRemoteLauncher.ts)
• Proactive quota gauges (separate feature)
• Model picker label changes (separate issue)
• Auto-retry on failure (can stack on top; not required for v1)

---

Files likely touched

File	Change
cli/src/cursor/cursorAgentMessageClassifier.ts (new)	Classifier function + unit tests
cli/src/cursor/cursorRemoteLauncher.ts or ACP event handler	Call classifier before persisting; emit modelError event; mark turn degraded
shared/src/protocol.ts	Add lastModelError to SessionMetadata; add modelError event type
hub/src/sync/messageService.ts	Recognise + persist lastModelError; broadcast SSE
web/src/components/SessionView/ModelErrorBanner.tsx (new)	In-session banner
web/src/components/SessionView/SessionRow.tsx	Pulsing amber dot when lastModelError unacknowledged

Test plan

1. Unit tests: classifier covers every row of the pattern table above + benign messages return null
2. Integration: stream replay ending in Error: T: Connection stalled + ready event → lastModelError set, banner renders, dot pulses, Dismiss clears
3. Happy path: normal assistant messages unaffected

[github-actions]

Confirmed: this is not currently modeled as a failed turn in HAPI. Cursor/Gemini remote paths can send failure text as normal session/agent messages and still emit ready after the prompt block finishes when the queue is empty. I did not find an existing lastModelError metadata field, modelError sync event, or inline classifier for these strings.

A couple of file-name corrections for the current tree:

• There is no shared/src/protocol.ts; session metadata and sync event schemas are in shared/src/schemas.ts.
• The active Cursor remote path is split between cli/src/cursor/cursorRemoteLauncher.ts and cli/src/cursor/cursorAcpRemoteLauncher.ts; cursorRemoteLauncher.ts only selects ACP vs legacy transport.
• Session-list row rendering is currently in web/src/components/SessionList.tsx, not web/src/components/SessionView/SessionRow.tsx.

Relevant code:

• cli/src/cursor/cursorAcpRemoteLauncher.ts:199 - Cursor ACP streams backend messages through handleAgentMessage(...).
• cli/src/cursor/cursorAcpRemoteLauncher.ts:249 - handleAgentMessage converts backend text/error messages and sends them as normal agent messages.
• cli/src/cursor/cursorAcpRemoteLauncher.ts:202 - caught prompt failures are sent as normal type: 'message' session events.
• cli/src/cursor/cursorAcpRemoteLauncher.ts:214 - ready is emitted after the prompt finishes if the queue is empty.
• cli/src/gemini/geminiRemoteLauncher.ts:179 - Gemini prompt failures are sent as normal type: 'message' events with Gemini prompt failed: ... text.
• cli/src/gemini/geminiRemoteLauncher.ts:187 - Gemini remote also emits ready after the prompt finishes if the queue is empty.
• cli/src/agent/messageConverter.ts:39 - backend text messages become normal { type: 'message' } agent messages.
• cli/src/agent/messageConverter.ts:81 - backend error messages become { type: 'error' }, but there is no model-error classification here.
• hub/src/socket/handlers/cli/sessionHandlers.ts:78 - hub receives CLI message events.
• hub/src/socket/handlers/cli/sessionHandlers.ts:108 - hub persists the message before broadcasting, with no special model-error branch found.
• shared/src/schemas.ts:28 - MetadataSchema has no lastModelError field today.
• shared/src/schemas.ts:305 - SyncEventSchema has no modelError event today.
• shared/src/sessionSummary.ts:59 - session summaries expose selected metadata only; no model-error state is included for the sidebar today.
• web/src/components/SessionList.tsx:672 - the row currently shows spinner/attention state, not a model-error indicator.

No extra info needed from the report; the pattern examples are enough to design tests around the classifier and the ready-after-error behavior.

---

HAPI Bot

[heavygee]

Update 2026-06-12: revisiting the approach.

The initial implementation (regex classifier on agent/message text) is too brittle:

1. False positives by shape match. Anchored patterns help but cannot eliminate. Any agent message that legitimately starts with a quoted error string (e.g. an assistant explaining a fix and quoting the error verbatim) misfires. Self-test confirmed: the very assistant message describing the patterns it watches for triggered the banner.

2. Real wire format is messier than the spec. Error: T: errors from cursor-agent ACP arrive with leading whitespace (observed: "\n\nError: T: WritableIterable is closed" in session b52b9117). JS ^ matches start-of-string, not start-of-line; the patterns need trimStart on every entry. Each iteration of "spec said this, reality is that" is a sign the spec is at the wrong abstraction layer.

3. Better signals already exist on the wire. The HAPI-side ACP transport (AcpStdioTransport) already classifies stderr into typed AcpStderrError (rate_limit | model_not_found | authentication | quota_exceeded | unknown), exposes onStderrError for subscribers, fires markClosed(error) on transport close, and surfaces process exit/error. The cursor launcher just doesn't subscribe for model-error reporting.

4. The structured information exists at cursor-agent. Error: T: [resource_exhausted] is a stringified gRPC status that cursor-agent caught internally and .toString()'d into a text emit. The structure was discarded. This is an upstream design issue worth raising separately - the agent should emit typed errors over ACP, not stringify and reprint.

Revised approach

Primary signals (structural, no text matching):

• Subscribe cursorAcpRemoteLauncher to transport.onStderrError and route AcpStderrErrorType -> lastModelError.kind.
• Catch JSON-RPC error responses on session/prompt rejections.
• Hook markClosed during in-flight turns -> kind: 'transport_closed' (the WritableIterable is closed case becomes structural).
• Watch process exit during a pending prompt -> kind: 'agent_crashed'.

Fallback (text classification, last resort):

• Keep cursorAgentMessageClassifier but only fire when no structural signal already classified the turn AND the text matches an anchored, narrow pattern. Reduce false-positive surface; document explicitly as "agent stringified an error we couldn't see structurally."

Branch: feat/cursor-detect-inline-model-errors (currently 3 commits on the regex approach). Will rewrite in place; no PR was opened yet.

[heavygee]

Notes for cursor-agent team (operator will mirror to the cursor forum).

This is a writeup of what an ACP consumer (HAPI) sees when cursor-agent hits an internal model-side failure, why it makes reliable error detection on the consumer side impossible, and what would fix it cleanly.

Observed wire format (session e7d9b44b, 2026-06-13)

cursor-agent emitted a single agent/message text event with body:

Three of the four hit Codex's usage limit (#151, #153, #155) - no code review
delivered. Only #157 actually got reviewed. Let me pull the inline comments to
see Codex's specific suggestions:

Error: T: [resource_exhausted] Error

Note the shape: ~250 chars of normal narrative output, then \n\n, then a stringified gRPC status appended at the end of the same text message. The session/prompt JSON-RPC call returned success. A ready event followed immediately.

A second wire format we've seen (session b52b9117) is the same shape but for a different internal error: Error: T: WritableIterable is closed appended as agent text instead of bubbling up structurally.

What an ACP consumer can subscribe to today

• JSON-RPC error responses on session/prompt -> caught via await backend.prompt(...) rejection.
• Typed stderr lines parsed by the transport (HAPI classifies these into rate_limit / quota_exceeded / authentication / model_not_found / unknown via AcpStdioTransport.parseStderrError) -> caught via onStderrError subscription.
• Transport close mid-prompt (process exit, stdin EPIPE, ACP socket dead) -> markClosed rejects all pending RPCs, surfaces through the same prompt rejection catch.
• Process crash signals -> via systemd / process supervision.

What was on the wire for the e7d9b44b failure

• No JSON-RPC error response (prompt resolved success).
• No stderr emit.
• No transport close.
• No process exit.
• Just one agent/message text event whose last line happened to be the gRPC error stringification.

So: the structural information about the failure existed inside cursor-agent (it knew it had hit RESOURCE_EXHAUSTED), but cursor-agent caught the error in its own runtime, stringified it (looks like Go's status.Error() output -> T: [snake_case_code]), and appended the string to the in-flight assistant text instead of surfacing it through any of the structural channels ACP/JSON-RPC/stderr provides.

Why this matters to consumers

Any ACP consumer wanting to flag model failures has to fall back to substring matching on free-text agent prose. That's fragile in both directions:

• Misses real failures whose marker doesn't match the consumer's regex shape (we missed e7d9b44b at first because our regex was anchored start-of-string and the error sat after \n\n mid-message).
• False-positives on agent prose that legitimately mentions error strings (e.g. an assistant explaining a fix and quoting the error verbatim, OR a code-review comment containing a pattern that looks like the error).

The consumer can't tell "the model emitted this string because it actually failed" from "the model emitted this string because it's discussing failures" without a structural signal.

The downstream UX cost is real: an operator looking at the chat sees a long, normal-looking assistant message that ends in an error string and can't tell whether the agent's reasoning above the error was actually completed work or a half-finished turn that fell over. The session keeps its "ready" green-dot state. Operators walk away thinking their work is done when it isn't.

Concrete asks for cursor-agent

Any one of these would let consumers detect this failure class without text scraping:

1. Reject the session/prompt RPC with a JSON-RPC error response when the agent's underlying model call hit a non-recoverable error mid-turn. The standard place. Lets us catch it via the prompt-rejection path we already have.

2. Surface the failure as a typed ACP notification (e.g. an agent/error or agent/model_status event with { kind, transient, raw }). This matches the way the rest of the protocol communicates state changes.

3. Write the error to stderr in a parseable format. Lowest-friction option - HAPI's AcpStdioTransport.parseStderrError already classifies stderr lines into typed errors and we'd catch it for free. Doesn't even require an ACP protocol change.

4. Don't .toString() internal gRPC statuses into the conversational text stream at all. The Error: T: [resource_exhausted] shape is an internal serialization format leaking out to end users.

(1) or (2) are the right answers. (3) is the cheapest interim fix. (4) is the cleanup that should happen regardless.

Bonus: distinguish transient vs non-transient

If the structured signal carries a hint about whether the failure is retryable (transient: rate_limit / canceled / connection stalled / deadline_exceeded vs non-transient: quota_exhausted / context_window / capacity_exhausted), consumers can offer different UX (auto-retry vs operator decision). HAPI infers this from the kind today, but having it on the wire from cursor-agent removes another layer of guessing.

[heavygee]

Scope amendment (2026-06-20) — one feature, three legs

The original issue framed detection + surfacing. Dogfood on :3006 confirmed that works: accurate Cursor-specific error capture, no false positives in normal use, amber banner + pulsing session-list dot + emergency notifications (Push / Telegram / FCM/Wear severity=error).

Detection without recovery is half a fix. The operator still has to notice the alert and manually re-type or hope "continue" works. The complete solution to this issue is therefore one feature with three parts:

1. Detect — classify model-side turn failures (structural signals first, text classifier as fallback)
2. Surface — don't lie (lastModelError, block spurious ready, banner, notifications)
3. Bridge — for recoverable hiccups, patch the gap so the turn can actually finish

Parts 1–2 without 3 leave the operator holding the bag. Part 3 without 1–2 is blind automation. Same problem, same PR.

This supersedes the v1 exclusion "Auto-retry on failure (can stack on top; not required for v1)" below. What remains out of scope is retry policy (loops, exponential backoff, multiple automatic attempts) — not the single bridge described here.

---

6. Transient error bridge (same feature)

When a transient model error is recorded (connection_stalled, deadline_exceeded, unavailable, transport_closed, rpc_timeout, rate_limited, canceled, etc. — see classifier transient flag), HAPI may send one bridge prompt to the agent so the turn can resume.

This is not a retry loop. One bridge per distinct error event (lastModelError.atTs). If the bridge also fails, set retriedAndFailed: true on metadata and stop.

Bridge prompt shape

Do not inject the bare word "continue". Re-send the last user message from the failed turn wrapped in explicit context so the agent knows why it is seeing the message again:

[HAPI bridge — transient model error]

The previous turn failed before completing ({kind}: {short excerpt}).

Re-sending your last message below. Continue the task from where you left off.
Do not repeat work you already finished unless the error invalidated it.

---
{lastUserMessage}

When priorAssistantClaimsDone === true, add one line before the separator:

You may have reported completion before this error — verify what is actually done before proceeding.

Triggers (same CLI implementation, multiple entry points)

• Manual: banner [Bridge & retry] (issue mockup already had [Retry last message] — same intent), optional FCM/Wear/Telegram action buttons → hub RPC → CLI bridge function
• Automatic (opt-in, default off): settings toggle e.g. "Automatically bridge transient model errors" — when enabled, CLI fires one bridge immediately after recording a transient lastModelError

Gating

Condition	Bridge allowed?
transient === true	yes (manual always; auto only if opt-in)
quota_exhausted, context_window, capacity_exhausted, auth_failed, model_not_found	no — surface only; operator must fix root cause
Already bridged this atTs	no — dedupe
Bridge prompt also failed	set retriedAndFailed: true, no second bridge

Implementation locus: CLI (cursorAcpRemoteLauncher — batch.message is in scope when recordModelError fires). Hub exposes RPC/REST for manual trigger; web settings store the opt-in pref.

---

Implementation status (dogfood, 2026-06-20)

Leg	Status
Detect (structural-first + text fallback, multiline patterns)	shipped in soup
Surface (metadata, banner, amber dot, block ready)	shipped in soup
Notifications (Push, Telegram, FCM sendModelError)	shipped in soup (FCM via soup bridge layer pending upstream #803 merge)
Bridge prompt + opt-in setting + manual affordances	not yet implemented — remainder of this issue

Branch: feat/cursor-detect-inline-model-errors on heavygee/hapi (7 commits, PR to tiann/hapi pending rebase + open).

---

Updated "What this does NOT include"

• The stderr/exit-1 quota process exit path (separate code path; legacy launcher transient requeue)
• Proactive quota gauges
• Model picker label changes
• Multi-attempt retry policy (backoff loops, capped auto-requeue chains) — distinct from the single bridge above

---

Updated files likely touched (bridge leg)

File	Change
cli/src/cursor/cursorAcpRemoteLauncher.ts	buildBridgePrompt(), invoke after transient recordModelError when opt-in; handle manual bridge RPC
cli/src/api/apiSession.ts or RPC handler	register bridge trigger
hub/src/web/routes/sessions.ts	POST /sessions/:id/model-error/bridge
web/src/components/ModelErrorBanner.tsx	[Bridge & retry] button (transient-gated)
web/src/routes/settings/	opt-in toggle
shared/src/schemas.ts	optional user/namespace pref if not stored client-side only

---

Updated test plan (bridge leg)

1. Transient error recorded → manual bridge RPC → agent receives wrapped last user message (not bare "continue")
2. Opt-in on → one automatic bridge per atTs; opt-in off → no automatic bridge
3. Non-transient error → bridge button disabled / RPC rejected
4. Bridge fails → retriedAndFailed: true, no second bridge for same atTs

PR will reference this issue as the single justification for detect + surface + bridge.

[heavygee]

Forensic: session 0e04ebe7 ("teams structure") missed (2026-06-20)

Operator report: Cursor session showed Error: RetriableError: [canceled] then ready; no amber dot / notification.

Clarification: This was a Cursor session (metadata.flavor=cursor). Messages appear as content.type: "codex" in the DB because HAPI wraps ACP agent output via convertAgentMessage for the web/sync pipeline — not because the Codex runner handled the turn. Detection hooks in cursorAcpRemoteLauncher were on the correct path; the classifier pattern was wrong.

Wire format (seq 1992):

Error: RetriableError: [canceled] http/2 stream closed with error code CANCEL (0x8)

Immediately followed by ready (seq 1993). Operator manually sent continue (seq 1994).

Root cause: Classifier only matched Error: T: [canceled], not Error: RetriableError: [canceled]. Same gRPC bracket notation, different stringification prefix from cursor-agent.

Fix (same PR / branch): Extend text-fallback patterns to accept (?:T:|RetriableError: ) before the bracket status codes. Regression test added for session 0e04ebe7.

Bridge status: Spec'd in scope amendment (2026-06-20) as leg 3 of the same feature — not yet implemented in code. Detection + surface + notifications shipped; bridge prompt + opt-in auto + manual [Bridge & retry] remain open on this issue.