Reliability push: classify run failures, correlate PostHog with Langfuse, and add safe retries

[yinjialu]

Why

We need to drive the observed chat run failure rate from ~15% to <=8% and make every failed run explainable to both the product team and the user.

Source planning doc: https://powerformer.feishu.cn/wiki/KHqiw4QlRiOqBdk5SPAckSWSn0b

Current baseline from the doc and PostHog run_finished analysis:

• Window: 2026-05-12 to 2026-05-30, n ~= 179k run_finished events.
• Overall failure rate: 14.75%.
• Failures are systemic across engines, not isolated to one agent: claude ~14.8%, codex ~14.3%, opencode ~12%, gemini ~16.8%.
• ~92% of failures are effectively not root-cause readable. For claude / codex / opencode, about 75% of failures collapse into AGENT_EXECUTION_FAILED + AGENT_EXIT_1.
• Existing diagnostic logic such as apps/daemon/src/claude-diagnostics.ts can produce useful human text, but it is not consistently converted into structured error classes for run_finished or dashboards.

Current Signal Gap

run_finished.error_code is currently too coarse:

• AGENT_EXECUTION_FAILED mixes auth, quota/rate limit, upstream 5xx, empty output, model/service failures, tool/runtime failures, and other generic execution failures.
• AGENT_EXIT_1, AGENT_EXIT_41, AGENT_SIGNAL_SIGTERM, etc. describe process termination shape, not the user-meaningful reason.
• PostHog can show where failure volume concentrates by agent_provider_id, model_id, os, etc., but cannot reliably explain the root cause.
• Langfuse already carries per-run trace detail and natural-language error/output context. The trace id is the run id, so run_finished.properties.run_id can theoretically correlate directly to the Langfuse trace. However, Langfuse metadata does not yet carry the same structured error_code / failure category fields, and PostHog does not carry the Langfuse-readable failure reason.

Relevant current code paths:

• apps/daemon/src/server.ts: /api/runs handler emits run_created and schedules run_finished via design.runs.wait(run).
• apps/daemon/src/run-result.ts: maps run terminal status to result and derives fallback error_code values such as AGENT_EXIT_*.
• apps/daemon/src/runtimes/auth.ts: already has reusable auth/rate-limit/upstream text classifiers.
• apps/daemon/src/claude-diagnostics.ts: produces richer Claude failure diagnostics and retryability.
• apps/daemon/src/langfuse-bridge.ts / langfuse-trace.ts: sends completed run traces to Langfuse with run.id as trace id.
• apps/web/src/runtime/amr-guidance.ts: maps known failure codes to user actions such as retry, authorize, recharge, or switch to AMR.

Proposal

1. Add a canonical failure classification layer at the run terminal path

At the single child-close / run-finalization choke point, classify failures into user- and dashboard-readable categories before emitting run_finished.

Suggested fields:

failure_category:
  | 'auth'
  | 'rate_limit'
  | 'insufficient_balance'
  | 'model_unavailable'
  | 'prompt_too_large'
  | 'upstream_unavailable'
  | 'timeout'
  | 'empty_output'
  | 'tool_error'
  | 'process_exit'
  | 'user_cancel'
  | 'unknown'

failure_stage:
  | 'preflight'
  | 'spawn'
  | 'session_init'
  | 'model_select'
  | 'prompt_send'
  | 'first_token_wait'
  | 'tool_execution'
  | 'artifact_write'
  | 'child_close'
  | 'finalize'

retryable: boolean
user_action: 'retry' | 'login' | 'recharge' | 'switch_model' | 'reduce_context' | 'install_cli' | 'none'
langfuse_trace_id: run.id

Use the existing classifiers first:

• classifyAgentServiceFailure(...) for auth / rate limit / upstream.
• classifyAgentAuthFailure(...) for agent-specific auth guidance.
• classifyAmrAccountFailure(...) for AMR_AUTH_REQUIRED and AMR_INSUFFICIENT_BALANCE.
• diagnoseClaudeCliFailure(...) for Claude-specific endpoint/auth/profile problems.
• Prompt-size guards such as AGENT_PROMPT_TOO_LARGE should classify as prompt_too_large and user_action='reduce_context'.

AGENT_EXIT_* and AGENT_SIGNAL_* should remain as fallback diagnostics but not be the primary root-cause category when stderr/stdout/diagnostics imply a better class.

2. Emit the same classification to PostHog and Langfuse

PostHog run_finished should include the structured fields above. Langfuse trace metadata should include the same values so a PostHog event can be correlated to a trace by run_id == traceId.

Minimum correlation invariant:

• PostHog: run_finished.properties.run_id
• Langfuse: trace id = same run.id
• PostHog should also emit langfuse_trace_id: run.id explicitly for discoverability.

This lets us analyze aggregate failure trends in PostHog, then inspect representative traces in Langfuse without guessing.

3. Add safe automatic retry for transient failures

Automatically retry 1-2 times only for transient classes:

• rate_limit when the provider signal is retryable and not an account/quota hard stop.
• upstream_unavailable / 5xx / network errors.
• short first-token / empty-output failures before any tool call or artifact write.

Do not auto-retry:

• auth required
• AMR insufficient balance
• prompt too large / context overflow
• model unavailable
• missing CLI / agent unavailable
• user cancellation
• runs that have already performed tool writes or produced artifacts, unless the retry path is explicitly idempotent

Emit retry observability:

run_retry_attempted
run_retry_finished
retry_of_run_id
retry_attempt_index
retry_strategy
retry_result
retry_suppressed_reason

4. Add main-path segment timing

The doc calls out that P90 latency is currently ~566s for successful tasks, but the current data cannot explain where the time goes. Add stage timestamps/durations for:

• queued
• process spawn
• session init
• model selection
• prompt send
• first token
• generation
• tool execution aggregate
• artifact reconciliation/finalization

Populate existing or planned run_finished duration fields where possible, including time_to_first_token_ms and generation_duration_ms.

5. Update user-facing failure guidance

Make the chat error card and AMR guidance consume the new classification fields where available:

• auth -> login / authorize CTA.
• rate_limit -> retry when transient, or suggest switching to AMR / another model when quota-like.
• insufficient_balance on AMR -> recharge CTA.
• prompt_too_large -> explain context is too large and suggest reducing selected context / switching to stdin-capable adapter.
• model_unavailable -> choose another model.
• upstream_unavailable -> retry CTA.

Acceptance Criteria

• run_finished failed events include failure_category, failure_stage, retryable, and user_action.
• Langfuse trace metadata includes the same failure fields plus the existing run id correlation.
• AGENT_EXECUTION_FAILED + AGENT_EXIT_* no longer dominate as the only root-cause fields for known auth/rate-limit/upstream/context failures.
• Failed runs with result === 'failed' still always carry a non-empty error_code.
• Unknown/unclassified failure share is measurable and alertable.
• Safe transient failures retry automatically with bounded attempts and retry telemetry.
• Non-retryable failures produce clear user-facing guidance.
• Unit tests cover the failure classifier and the invariant that every failed run_finished has a specific category or explicit unknown fallback.
• A PostHog dashboard can show failure rate by failure_category, failure_stage, agent_provider_id, model_id, os, and retry outcome.
• A Langfuse trace can be opened from a failed PostHog run using run_id / langfuse_trace_id.

Suggested Validation

• Add daemon unit tests around classification helpers and run-result derivation.
• Replay mock CLI traces for representative failures: auth, rate limit, upstream 5xx, prompt too large, timeout/SIGTERM, empty output.
• Confirm PostHog receives classified run_finished events in local smoke with telemetry enabled.
• Confirm Langfuse trace metadata mirrors the same classification.
• Use a fixed task set as a release guard: success rate, P90, token usage, unknown failure share.

Timeline From Planning Doc

• 2026-06-01 EOD: failure classification, main-path segment telemetry, automatic retry.
• 2026-06-03 EOD: 100% failures attributable enough to produce an ROI-ranked optimization list.
• 2026-06-08 EOD: first high-ROI fixes.
• 2026-06-10 EOD: first optimization release plus second medium-ROI release.
• By 2026-06-12: validate first retry effect with new data and put regression gates in place.

Notes

The first engineering move should be to avoid adding more distributed failure emitters. The planning doc explicitly recommends keeping classification and timing at one run-terminal choke point, with a test that prevents new code paths from bypassing structured failure reporting.

[yinjialu]

PR1 Spec: failure attribution + PostHog/Langfuse correlation

Scope for the first PR:

• Add structured failure fields to failed run_finished events: failure_category, failure_stage, retryable, user_action, and langfuse_trace_id.
• Mirror the same structured fields into Langfuse trace metadata so PostHog aggregate events can drill into Langfuse traces by run_id == langfuse_trace_id == traceId.
• Keep existing error_code behavior intact: every failed run still has a non-empty error_code; AGENT_EXIT_* / AGENT_SIGNAL_* remain fallback diagnostics, not the primary root-cause category when a better class is available.
• Reuse existing local classifiers where possible: agent service auth/rate/upstream detection, AMR account failures, prompt-too-large codes, process exit/signal fallbacks.
• No automatic retry behavior in PR1. Retry policy and retry telemetry will be PR2 after we can measure categories reliably.
• No main-path timing expansion in PR1 beyond preserving existing duration fields. Segment telemetry is PR3 unless a tiny supporting field is required for classification.
• Do not send raw prompt, stderr, stack, or provider output to PostHog. Langfuse keeps the existing redaction/consent behavior.

Acceptance criteria for PR1:

• run_finished failed events include failure_category, failure_stage, retryable, user_action, and langfuse_trace_id.
• Langfuse trace metadata includes the same failure_category, failure_stage, retryable, user_action, and canonical error_code values.
• langfuse_trace_id is exactly the daemon run id, matching the Langfuse trace id convention.
• Known cases classify into stable buckets:
  • auth required -> auth / login
  • rate or quota limited -> rate_limit / retry or user guidance
  • upstream/network/service unavailable -> upstream_unavailable / retry
  • AMR insufficient balance -> insufficient_balance / recharge
  • model unavailable -> model_unavailable / switch_model
  • prompt/context too large -> prompt_too_large / reduce_context
  • empty output, timeout/signal, and process exits have explicit fallback categories instead of disappearing into unclassified data.
• Unknown remains explicit as failure_category=unknown and is dashboardable.
• Successful runs are not marked as failures and behavior is otherwise unchanged.

Validation plan for PR1:

• Add unit tests for the classifier covering auth, rate limit/session limit, upstream unavailable, AMR recharge, prompt too large, model unavailable, empty output, timeout/signal, process exit, cancel, and unknown fallback.
• Add a focused test or assertion around run-finished prop construction if the existing test harness makes that cheap.
• Run package-scoped daemon tests, contracts typecheck, plus repo pnpm guard and pnpm typecheck before marking ready.
• Manual smoke target: trigger or replay representative failed runs, then confirm PostHog can slice by failure_category and Langfuse metadata has matching fields for the same run_id.

Draft PR title: Improve run failure classification and Langfuse correlation

[yinjialu]

Follow-up: make PostHog ↔ Langfuse coverage explainable

When investigating why some PostHog run_finished.properties.run_id values cannot be found as Langfuse traceId, add explicit delivery-state fields to run_finished rather than treating the absence as sampling.

Recommended fields:

• langfuse_expected: boolean
• langfuse_drop_reason?: string | null
• langfuse_delivery_status?: "not_expected" | "queued" | "accepted" | "failed"

Important nuance: langfuse_expected does not prove that a trace exists in Langfuse. It only means the daemon believed the run was eligible for Langfuse reporting.

Suggested derivation:

langfuse_expected =
  telemetry.metrics === true &&
  telemetry.content === true &&
  hasTelemetrySink === true;

Suggested langfuse_drop_reason values:

| "metrics_consent_off"
| "content_consent_off"
| "missing_sink_config"
| "payload_too_large"
| "relay_429"
| "relay_413"
| "relay_5xx"
| "langfuse_4xx"
| "langfuse_5xx"
| "network_error"

This lets us separate three cases in PostHog:

1. langfuse_expected=false
   No Langfuse trace is expected. Example: content telemetry is off, metrics are off, or no relay/credentials are configured.

2. langfuse_expected=true && langfuse_drop_reason != null
   A trace was expected, but local delivery failed or was intentionally dropped. Example: payload too large, relay 429, relay/Langfuse 5xx, network error.

3. langfuse_expected=true && langfuse_delivery_status="accepted"
   The daemon received a successful response from the relay/Langfuse ingestion path. If Langfuse UI still cannot find the trace by run_id, that is the real investigation bucket: ingestion delay, UI search limitation, partial 207 errors, or trace-id mismatch.

Implementation notes:

• PostHog run_finished should include these fields even when Langfuse is not expected, so coverage gaps are explainable from PostHog alone.
• Langfuse trace metadata should also include error_code, failure_category, and failure_stage when a trace is successfully created.
• For relay/direct ingestion, treat 2xx/207 with no per-event errors as accepted; 207 with per-event errors should become failed with a specific drop reason if possible.
• These fields are observability state, not user-facing diagnosis. They should help answer: “Why does PostHog have this run_id but Langfuse has no trace?”

[yinjialu]

PR1 Scope Update: include latency and token segmentation

The current PR1 scope must include the timing/token observability called out in the planning doc, not only failure attribution. The previous PR1 comment said main-path timing could wait for PR3; that is now superseded by this update.

Why this belongs in PR1:

• We know successful tasks are slow, but not which segment is slow. Without segment timings, there is no reliable way to choose the highest-ROI latency fix.
• OD has a clear main path: queue -> process spawn -> first token -> generation -> tools -> finalization. Recording timestamps/durations for those segments lets the dashboard show where P90 latency is concentrated and which task types are slowest.
• Token optimization has the same problem: output is not the cost driver. The input side must be split enough to distinguish context construction from provider/cache behavior before we can find the 25% reduction opportunity.
• Failure attribution and segment telemetry can be implemented in parallel and do not block each other.

Baseline to preserve in the issue:

• Latency for successful tasks: P50 98s, P90 566s (9.4 minutes), P99 23 minutes. These are mixed across all task types, so the next dashboard must break them down by task/project/run dimensions.
• Token usage for successful tasks: average 360k tokens/run, with about 350k input and 9k output. Cost is dominated by input, so token savings should focus on context size/cache behavior rather than compressing output.

PR1 acceptance additions:

• run_finished emits segment timing fields for the main run path where the daemon can observe them now, including queue/start, spawn, first-token, generation, tool aggregate, finalize, and total duration.
• run_finished emits token-analysis fields beyond raw input/output/total, including an input-context estimate and cache-read/cache-write fields when provider usage exposes them. Unknown parts must be explicit/nullable rather than guessed.
• Langfuse trace metadata mirrors the same timing/token analysis fields so aggregate PostHog slices can drill into representative traces.
• The implementation should prefer one choke point and reusable event scanning helpers; no raw prompt/stderr/stack/provider output should be sent to PostHog.

[lefarcen]

Hey @yinjialu, I read through the issue plus the three scope comments. The latest update is clear: PR1 should cover failure attribution/correlation and the latency/token segmentation needed for the first ROI dashboard; automatic retry stays out of PR1.

A good implementation shape from the current code paths is:

• keep apps/daemon/src/run-result.ts as the place that preserves the non-empty error_code invariant, then layer the new failure_category / failure_stage / retryable / user_action fields beside it rather than replacing it;
• extend the run_finished construction in apps/daemon/src/server.ts at the existing terminal design.runs.wait(run) choke point, so we do not create parallel failure emitters;
• thread the same classified fields, Langfuse delivery state, and timing/token summary into the flat Langfuse trace metadata path in apps/daemon/src/langfuse-trace.ts / langfuse-bridge.ts;
• keep the privacy boundary explicit: PostHog gets structured buckets and delivery/timing/token counters only, while raw prompt/stderr/provider output stays out.

So for PR1, I’d treat the superseded scope as: classified failed run_finished + Langfuse trace metadata correlation + Langfuse expected/drop/delivery fields + observable segment timing/token analysis, with retry policy/telemetry reserved for PR2. Please link the implementation PR back here when it opens so the acceptance list stays anchored to this tracker.

[lefarcen]

Quick update: draft PR #3412 is now open for the PR1 slice. It focuses on the structured failed-run fields, PostHog ↔ Langfuse correlation via langfuse_trace_id, and the timing/token observability from the latest scope update above.

Keeping this issue open while that PR goes through review so the tracker still owns the broader reliability push and follow-up slices.

[yinjialu]

Progress update for PR1 (#3412):

• Addressed the latest scope feedback from this issue: PR1 now covers classified failed run_finished, PostHog/Langfuse correlation, Langfuse expected/drop/delivery fields, segment timing, and token/cache analysis. Automatic retry remains out of PR1 and should stay PR2.
• Added Langfuse delivery-state fields to the analytics contract and run_finished: langfuse_expected, langfuse_delivery_status, and langfuse_drop_reason. PostHog can now distinguish not-expected traces such as consent/config gaps from eligible traces queued for Langfuse.
• Extended the Langfuse sender to return concrete delivery outcomes: accepted for successful ingestion, failed with drop reasons for payload-too-large, relay/direct HTTP failures, 207 per-event errors, and network errors.
• Fixed the CI feedback in apps/daemon/tests/langfuse-bridge.test.ts after token metadata expanded to include provider/effective/context/cache fields.
• Verified local Claude-logged-out smoke with real PostHog + Langfuse: the failed run classified as AGENT_AUTH_REQUIRED / auth / session_init / login, and the same run id was visible in both systems.
• Created PostHog insight for failed runs with direct Langfuse trace links: https://us.posthog.com/project/420348/insights/s8PQx5fJ

Validation now passing locally:

• pnpm --filter @open-design/daemon exec vitest run -c vitest.config.ts tests/langfuse-trace.test.ts tests/langfuse-bridge.test.ts tests/run-analytics-observability.test.ts tests/run-lifecycle-analytics.test.ts tests/run-failure-classification.test.ts
• pnpm --filter @open-design/contracts typecheck
• pnpm --filter @open-design/daemon typecheck
• pnpm guard
• pnpm typecheck

[yinjialu]

Clarification on PR1 priority after re-checking the acceptance comment:

• Required for PR1: classified run_finished, non-empty error_code, PostHog/Langfuse correlation by run_id, timing segmentation, and token/cache analysis.
• Non-required for PR1: making PostHog prove the final Langfuse ingestion result (accepted / failed) for every run.
• run_finished should stay the reliable primary metric. It may include Langfuse eligibility/enqueue fields such as langfuse_expected and langfuse_delivery_status=queued/not_expected, but it must not wait on Langfuse logging or let Langfuse delivery affect metric capture.
• If we need to analyze why an expected trace never appears in Langfuse, that should be a lower-priority follow-up via a separate delivery-result event or relay-side telemetry, not a blocker for PR1.

I updated PR #3412 body to make this priority explicit so review does not treat final Langfuse delivery-result backfill as a required acceptance item.

[lefarcen]

Got it @yinjialu — the priority clarification makes sense.

I’ll treat PR1’s acceptance as: classified run_finished, non-empty error_code, run_id / langfuse_trace_id correlation, timing segmentation, and token/cache analysis. Final Langfuse ingestion proof should not block PR1; run_finished should stay the reliable PostHog metric path, with Langfuse eligibility/enqueue state only where it is available without waiting on logging.

I also checked the #3412 body and it now states that boundary clearly, so review should not require delivery-result backfill in this PR.

[lefarcen]

Hey @yinjialu — quick sync from the tracker side: #3412 is now out of draft and sitting in reviewer-responsible state, and we’re carrying the PR1 boundary you clarified yesterday into that review.

So on this issue, the acceptance line stays: classified run_finished, non-empty error_code, run_id / langfuse_trace_id correlation, timing segmentation, and token/cache analysis; retry policy plus any deeper Langfuse delivery-result telemetry can stay follow-up slices after this PR lands.

We’ll keep #3408 open while #3412 finishes review so the broader reliability push still has one parent tracker. ❤️

[lefarcen]

Hey @yinjialu — PR #3412 is now merged, so the PR1 slice from this tracker is in.

That lands the structured failed run_finished classification, run_id / langfuse_trace_id correlation, Langfuse eligibility/enqueue/drop fields, timing segmentation, and token/cache analysis we aligned on in this thread. We’re keeping #3408 open because the broader reliability push still includes the follow-up slices after PR1 — especially retry policy/telemetry and the next ROI-ranked optimization work.

I’ll keep using this issue as the parent tracker for those next steps. ❤️

[lefarcen]

Quick tracker link: #3526 is the spec/backlog follow-up for the post-PR1 reliability work after #3412 landed.

It carries the next slices we already kept open here — safe retry, retry telemetry, ROI-ranked failure fixes, latency/token optimization, fixed-task regression gates, and PostHog alerting — so we can review the broader plan in one place before implementation PRs branch off. Keeping #3408 open until those follow-up slices land.