Proposal: expose full per-run diagnostics to local users & external agents (od run inspect + diagnostics API)

[tomsen02]

Why

When a chat run fails or runs slow/expensive, the daemon already computes a rich diagnostic picture on the run-terminal path — full failure classification (failure_category, failure_detail, failure_stage, retryable, user_action in run-failure-classification.ts), timing segments (queue / pre-spawn / spawn / first-token / generation / tool / finalize in run-analytics-observability.ts:591-807), and token/cache breakdown (input / output / cache-read / cache-creation / hit-ratio). But almost none of it reaches a local user or an external agent: it is emitted to PostHog/Langfuse telemetry and otherwise dropped.

What's already exposed vs not:

• feat(chat): surface daemon failure_detail to sharpen run-error guidance #5321 (feat(chat): surface daemon failure_detail) persisted failureCategory / failureDetail onto ChatRunStatusResponse (packages/contracts/src/api/chat.ts:469-473) and renders them as chat-UI guidance. Good — this proposal builds on it, it does not duplicate it.
• Still unreachable anywhere outside telemetry: failureStage, retryable, user_action, all timing segments, and all token/cache fields.
• od run has start / watch / cancel / list / info / result-package (cli.ts:5983-5992) but no diagnostics command — the CLI, which AGENTS.md calls the embeddability contract for external agents, exposes none of this.

The pain: a self-hoster who hits a failed run can only see a coarse status: failed / exitCode: 1 unless they happen to be in the chat UI on an AMR-guided path; they cannot ask where it failed, whether it's retryable, or where the time/tokens went. Worse, an external agent driving Open Design through od (the exact integration surface AGENTS.md's dual-track rule protects) has no way to read retryable / user_action to make its own retry or routing decision. The daemon computes the answer and throws it away for everyone not looking at telemetry.

What users will see

• od run inspect <runId> [--json] — prints the run's full diagnostics: failure classification (category, detail, stage, retryable, user_action), timing segments, and token/cache breakdown. --json for scripts/agents.
• A run diagnostics panel in the web run detail (expand from the existing error card) showing the same, so the UI reaches parity with the CLI.
• The /api/* response that backs both.

Surface area

• CLI / env var — new od run inspect subcommand
• API / contract — new/extended run-diagnostics endpoint + packages/contracts DTO
• UI — run diagnostics panel
• i18n keys — for the new UI panel labels

(This is a dual-track capability per AGENTS.md — landing API + CLI + UI in one PR.)

Scope boundaries (what this does NOT do)

• Pure exposure — no new computation. Every field already exists on the terminal path; this only surfaces it.
• No change to failure-classification logic. Deliberately stays out of the accuracy-tuning lane ([codex] Classify opaque runtime failure details #4966 / fix(daemon): classify runtime stream failures #5223 / fix(analytics): attribute BYOK OpenCode providers #5356).
• Does not duplicate feat(chat): surface daemon failure_detail to sharpen run-error guidance #5321. That surfaced category/detail as chat guidance; this adds the CLI surface + the deeper fields (stage/retryable/user_action) + timing/token, which feat(chat): surface daemon failure_detail to sharpen run-error guidance #5321 did not touch.

Open questions for maintainers (decide the shape)

1. Persistence. Timing/token analytics are currently fire-and-forget into the telemetry event, not persisted. To make them queryable after a run finishes, do we (A) persist them on the run object / SQLite — which must follow the RUNTIME_DATA_DIR data-directory contract in AGENTS.md — or (B) expose them only for runs still resident in memory (recent runs), accepting that older runs return classification-only? I lean B for a first slice (smallest, no data-dir surface), with A as a follow-up if there's demand.
2. Endpoint shape. Extend GET /api/runs/:id (wider response) or add a dedicated GET /api/runs/:id/diagnostics sub-resource (opt-in fetch)? I lean the sub-resource.
3. Phasing. Land failure-classification completion (stage / retryable / user_action) + od run inspect first, and timing/token as a second PR? That keeps the first PR small and reviewable.

Product relevance

Sits in the #3408 reliability lane, but from the exposure angle rather than the failure-rate-optimization angle (#3547). It directly serves two AGENTS.md invariants: the UI+CLI dual-track rule, and the CLI-as-embeddability-contract principle for external agents. No existing issue proposes local run-diagnostics exposure; #5321 is the nearest and is complementary (chat-guidance for category/detail), not overlapping.

I'm happy to implement this once the scope (esp. Q1/Q3) is blessed.

[lefarcen]

Hi @tomsen02! 👋

This is a strong proposal — the gap you called out is real. AGENTS.md explicitly treats UI + CLI as a dual-track capability, while the current run surface still stops at /api/runs/:id + /result-package, and ChatRunStatusResponse only carries failureCategory / failureDetail today, not failureStage / retryable / user_action or the timing/token slices.

It also looks complementary to existing work rather than duplicative: #3408 is the broader reliability/classification lane, and #5321 already proved the "daemon computes it → contracts expose it → UI consumes it" path for category/detail.

The main design seam does seem to be the one you called out around persistence for timing/token analytics. A narrow first slice of failureStage / retryable / user_action plus a dedicated diagnostics endpoint and od run inspect --json feels like the safest place to start, with the full timing/cache breakdown following once the storage story is settled.

Keeping this open for maintainer scope blessing makes sense. If you want to sketch the first-slice DTO/endpoint shape in a follow-up comment, that would make the next review step even easier.

[tomsen02]

Thanks @lefarcen — here's the first-slice DTO/endpoint sketch.

Key insight: the persistence design seam you flagged is really about the bulky timing/token analytics. The failure classification is a fixed set of 5 small scalars, and failureCategory / failureDetail are already persisted via persistRunFailureClassification (server.ts:5261). So slice 1 only needs to widen that persistence from 2/5 → 5/5 (add stage / retryable / user_action), which lets od run inspect work after the run finishes (and across restart) without touching the timing/token storage decision at all.

1) Endpoint — a dedicated sub-resource, so GET /api/runs/:id doesn't grow:

GET /api/runs/:id/diagnostics

2) DTO (packages/contracts/src/api/) — re-export the two tracking types onto the API surface, exactly like RunFailureCategory / RunFailureDetail already are:

export interface RunDiagnosticsResponse {
  runId: string;
  status: ChatRunStatus;
  // null when the run did not fail
  failure: {
    category: RunFailureCategory;      // existing
    detail: RunFailureDetail;          // existing
    stage: RunFailureStage;            // NEW — re-export of TrackingRunFailureStage
    retryable: boolean;                // NEW
    userAction: RunFailureUserAction;  // NEW — re-export of TrackingRunFailureUserAction
  } | null;
  // timing / token slices intentionally omitted here, pending the storage decision
}

Values reuse the existing enums verbatim:

• stage: preflight | spawn | session_init | model_select | prompt_send | first_token_wait | tool_execution | artifact_write | child_close | finalize
• userAction: retry | login | recharge | upgrade | switch_model | reduce_context | install_cli | fix_config | none

3) Persistence — widen persistRunFailureClassification to also store stage / retryable / user_action. My lean is a single failure_classification JSON column on the run row (carries all 5 fields, and future additions don't need a schema change), rather than three separate columns. This is bounded, small, non-fire-and-forget data — distinct from the timing/token analytics — so it doesn't touch the seam.

4) CLI — od run inspect <runId> [--json], backed by the endpoint above:

$ od run inspect run_abc
run_abc — failed
failure: auth / token_expired   (stage: first_token_wait)
retryable: no
suggested action: login

// --json
{ "runId": "run_abc", "status": "failed",
  "failure": { "category": "auth", "detail": "token_expired",
    "stage": "first_token_wait", "retryable": false, "userAction": "login" } }

5) Slice 2 (later PR): timing segments + token/cache. That is where the persistence decision lives (A: persist to SQLite under the RUNTIME_DATA_DIR contract, vs B: expose only for in-memory recent runs). Slice 1 deliberately doesn't presuppose it.

Slice-1 scope: one diagnostics endpoint + DTO (re-exporting 2 types) + classification persistence 2/5 → 5/5 + od run inspect + a read-only UI panel. Pure exposure, no change to classification logic, complementary to #5321 (which surfaces category/detail as chat guidance).

Does this shape look right to land as the first PR? If so I'll open it with the endpoint + CLI + a regression test, and follow up with the timing/token slice once the storage story is settled.

[lefarcen]

Hey @tomsen02, this refinement helps a lot.

That 2/5 → 5/5 persistence split is the key clarification — it cleanly separates the already-bounded failure classification from the much heavier timing/token storage question. The dedicated /api/runs/:id/diagnostics resource + od run inspect --json shape also reads like the right slice-1 contract, and a single failure_classification JSON blob sounds like a reasonable way to keep that persisted surface cohesive.

I'm routing this to the product/maintainer side for direction before we bless it as the first PR slice. I'll mark it for product-direction review now, and we usually get back within about 12 hours.

[lefarcen]

Quick heads-up: PR #5511 is now open as the bounded slice-1 implementation for this proposal — it carries the 2/5 → 5/5 failure-classification exposure (stage / retryable / userAction) plus GET /api/runs/:id/diagnostics and od run inspect, while still deferring the heavier timing/token persistence question. Keeping this issue open until the product/storage direction here is settled and the slice lands.