feat: Agent phase observability — structured traces with frontend timeline

[ryaneggz]

Agent Phase Observability & Trace Visualization

Structured spans per agent phase (planner, generator, evaluator, handoff) with timing, token counts, model, cost, and error info. Frontend timeline component showing phases as visual blocks with drill-down. OpenTelemetry export for external tools (Jaeger, Honeycomb, Datadog).

Key Deliverables

• AgentSpan schema — Pydantic model + SQLAlchemy model for structured phase spans with timing, token counts, cost, error info
• PostgreSQL agent_spans table — Alembic migration with indexes on trace_id, thread_id, started_at
• Span lifecycle utilities (backend/src/utils/tracing.py) — start_span, end_span, @trace_phase context manager with contextvars-based trace propagation
• REST API endpoints — GET /threads/{id}/traces, GET /traces/{id}/spans/{id} for frontend consumption
• Frontend swimlane timeline — horizontal colored phase blocks with hover tooltips (duration, tokens, cost) and click drill-down
• OpenTelemetry OTLP exporter — optional export to Jaeger/Honeycomb/Datadog, zero overhead when disabled

Spec

See /.claude/specs/agent-observability.md for full details.

Milestone

v0.10.0 — Harness Design v2

[ryaneggz]

OpenTelemetry Integration Analysis

Research findings for structured agent observability. This analysis covers the current architecture, integration gaps, and recommendations.

---

1. Current State — What Exists Today

No OTel packages in pyproject.toml. OpenTelemetry is only present as a transitive dependency of daytona>=0.140.0 (which pulls opentelemetry-api, opentelemetry-sdk, opentelemetry-exporter-otlp-proto-http, opentelemetry-instrumentation-aiohttp-client). These are resolved in uv.lock but never imported or configured by Orchestra code.

Existing observability primitives:

• main.py HTTP middleware logs method, path, duration, status_code as JSON via loguru — plain structured logging, no trace correlation.
• cache_metrics_middleware (middleware.py:80) logs prompt-cache hit ratios after each model call — structured log line, no span attachment.
• checkpoint_resilient.py exposes a .metrics property (retry_count, fallback_count, using_fallback, is_connected) — polled from WorkerState.get_metrics().
• No contextvars usage anywhere in backend/src/ — there is no trace context propagation mechanism today.

Feature branch feat/agent-observability already has:

• Full spec at .claude/specs/agent-observability.md
• 5-phase implementation plan at .claude/plans/agent-observability.md
• No actual code yet — just the spec and plan.

---

2. OTel Packages Needed

The spec proposes these as optional dependencies (Phase 5). Recommended additions to pyproject.toml [project.optional-dependencies]:

[project.optional-dependencies]
otel = [
    "opentelemetry-api>=1.39.0",
    "opentelemetry-sdk>=1.39.0",
    "opentelemetry-exporter-otlp-proto-http>=1.39.0",
    "opentelemetry-instrumentation-fastapi>=0.60b0",  # auto-instruments HTTP spans
]

Since daytona already pulls opentelemetry-api and opentelemetry-sdk into uv.lock, those resolve without conflict. The only truly new packages would be opentelemetry-instrumentation-fastapi and the OTLP exporter (already a transitive dep). For v1, only opentelemetry-api is needed if using manual span creation.

---

3. Where Spans Should Be Created/Closed

The middleware stack in init_default_middleware() (middleware.py:322-346) assembles:

compaction_middleware -> add_ai_message_metadata -> cache_metrics_middleware
-> retry_model -> PII middleware(s) -> AutoEvictMiddleware
-> [optional AnthropicPromptCachingMiddleware]

Recommended span insertion points (top-level to leaf):

Span	Where	Hook
request	main.py HTTP middleware (line 132)	Wrap call_next() — gives total request duration with trace_id
agent_turn	stream_generator() (stream.py:323) or _execute_agent_stream() (tasks.py:371)	Wrap the agent.astream() loop — one span per turn
model_call	New @wrap_model_call tracing middleware, inserted after cache_metrics_middleware	Captures per-LLM-call duration, input/output tokens from usage_metadata
tool_call	New AgentMiddleware.awrap_tool_call tracing middleware, inserted alongside AutoEvictMiddleware	Captures tool name, duration, success/error
compaction	Inside compaction_middleware (compacting.py:178)	Optional span for summarization LLM call

The @wrap_model_call and AgentMiddleware.awrap_tool_call decorators from langchain.agents.middleware are the natural extension points. A new tracing middleware would follow the same pattern as cache_metrics_middleware and AutoEvictMiddleware.

---

4. Span Correlation Across the Streaming Pipeline

Request flow (sync mode):

HTTP request -> llm_stream route -> LLMController.llm_stream()
  -> stream_generator() -> construct_agent() -> agent.astream()
    -> [middleware stack per model call / tool call]
  -> _persist_final_state() -> extract_trajectory (TaskIQ background task)

Request flow (distributed mode):

HTTP request -> llm_stream route -> run_agent_stream.kiq() [enqueued to Redis]
  -- TaskIQ worker picks up --
  -> _execute_agent_stream() -> construct_agent() -> agent.astream()
    -> [middleware stack per model call / tool call]
  -> _stream_chunks_to_redis() -> extract_trajectory.kiq()

Correlation strategy:

• Use contextvars.ContextVar for trace_id and parent_span_id — this propagates naturally through async/await within the same process.
• For distributed workers: the trace_id must be serialized into the task_dict passed to run_agent_stream.kiq() (tasks.py:129). The worker reconstructs the trace context from the dict. This is a manual step since TaskIQ does not auto-propagate OTel context.
• The run_id already generated at the route level (llm.py:101) is a natural candidate for trace_id.

---

5. Span Propagation Through Async/Background Tasks (TaskIQ)

The spec does NOT account for this. This is a significant gap:

• extract_trajectory is dispatched fire-and-forget via extract_trajectory.kiq() from both _persist_final_state() (stream.py:308) and _execute_agent_stream() (tasks.py:690). It runs in a separate TaskIQ worker with no context carryover.
• run_agent_stream itself runs in a worker process — the trace context from the originating HTTP request does not automatically propagate.

Recommendation: Add trace_id as an explicit parameter to both run_agent_stream() and extract_trajectory() task signatures. The worker reconstructs a TraceContext from this ID on startup. This is simpler and more reliable than trying to use OTel's propagate.inject()/extract() with TaskIQ's serialization layer.

---

6. Span Data Storage — Same DB vs OTel Collector

The spec proposes a PostgreSQL agent_spans table via Alembic migration. This is the right call for v1 because:

1. Thread data is already stored in LangGraph's AsyncPostgresStore (via store.aput() in ThreadRepo) — the spans table lives alongside it in the same Postgres instance.
2. The frontend needs low-latency access to span data per-thread for the timeline component. Querying an external OTel collector (Jaeger/Tempo) would add an extra service dependency and API surface.
3. The OTLP exporter (Phase 5) is correctly positioned as optional, additive — it dual-writes spans to both Postgres (for frontend) and the external collector (for platform observability).

One concern: The agent_spans table will grow quickly for active deployments. Consider adding a TTL/cleanup strategy (e.g., created_at index + periodic deletion older than 30 days), which the spec does not mention.

---

7. Relationship with Cost Tracking (#905)

Issue #905 proposes:

• TurnMetrics schema with model, input_tokens, output_tokens, cost_usd, duration_ms, phase
• A turn_metrics table via Alembic migration
• A metrics capture middleware in backend/src/utils/metrics.py

Overlap with #904 is significant. The AgentSpan schema (from the observability spec) contains model, input_tokens, output_tokens, cost_usd, duration_ms, phase — identical fields.

Recommendation: Merge the schemas. AgentSpan should be the single source of truth, with cost tracking (#905) being a view or aggregation over span data rather than a separate table. Specifically:

• Phase 1 of feat: Agent phase observability — structured traces with frontend timeline #904 creates the agent_spans table with all needed columns (including cost_usd, token counts).
• feat: Per-run cost tracking, phase-level metering, and budget caps #905's ThreadCostSummary becomes a SQL aggregate query over agent_spans WHERE thread_id = ?.
• feat: Per-run cost tracking, phase-level metering, and budget caps #905's budget enforcement reads from the same span data.
• This avoids dual-write, dual-migration, and schema drift between two tables capturing the same events.

Both issues target the v0.9/v0.10 milestone, so coordinating them now prevents rework.

---

8. Simpler v1 Approach — Structured Logging Before Full OTel

Yes, a phased approach is strongly recommended. The current codebase uses loguru with no structured span concept. Going from zero to full OTel + Postgres spans + frontend timeline is a big jump.

Proposed v0 (structured logging, 1-2 days):

1. Add a @trace_phase context manager that logs span-like JSON via loguru (start/end/duration/tokens/error), keyed by trace_id + span_id.
2. Use contextvars for trace_id propagation (no external deps needed).
3. Attach trace_id to the existing HTTP middleware log line in main.py.
4. This gives immediate debuggability with zero new dependencies and validates the instrumentation points before committing to a Postgres table.

Then v1 (the spec's Phase 1-3) adds persistence and API endpoints on top of the proven instrumentation points.

Key file paths referenced:

• /home/ryaneggz/ruska-ai/orchestra/backend/src/utils/middleware.py — agent middleware stack
• /home/ryaneggz/ruska-ai/orchestra/backend/src/utils/stream.py — streaming pipeline
• /home/ryaneggz/ruska-ai/orchestra/backend/src/workers/tasks.py — distributed task execution
• /home/ryaneggz/ruska-ai/orchestra/backend/src/agents/__init__.py — agent construction and Orchestra class
• /home/ryaneggz/ruska-ai/orchestra/backend/src/controllers/llm.py — LLM controller
• /home/ryaneggz/ruska-ai/orchestra/backend/src/routes/v0/llm.py — HTTP route entry points
• /home/ryaneggz/ruska-ai/orchestra/backend/main.py — FastAPI app with HTTP logging middleware
• /home/ryaneggz/ruska-ai/orchestra/backend/pyproject.toml — dependencies

[ryaneggz]

Frontend Timeline Visualization Analysis

1. Existing ToolTimeline Architecture

The current timeline system lives in two files:

• frontend/src/components/timeline/ToolTimeline.tsx — A simple vertical dot-timeline that renders a list of ToolMessage objects. It draws a vertical line (w-0.5) on the left with color-coded dots (green for success, red for error) and delegates content rendering to ToolTimelineItem.

• frontend/src/components/timeline/ToolTimelineItem.tsx — Renders each tool call as a collapsible card with a wrench icon, tool name badge, status indicator, and expandable content area. Handles special cases for search results, Plotly charts, JSON, and Markdown.

Key observation: The existing ToolTimeline is a vertical sequential list — it has no concept of time, duration, concurrency, or phases. It cannot be extended into a swimlane/Gantt-style visualization. The phase timeline must be a new component.

2. How Tool Calls Currently Display in Chat

ChatMessages.tsx uses @tanstack/react-virtual for virtualized scrolling. Message rendering logic (in the Message component):

• type: "tool" messages render inline via <ToolTimeline messages={[message]} /> (one tool per timeline instance)
• type: "tool_input" messages render via <DefaultTool> showing the input args
• AI messages with agent_name get a <SubagentBadge> left-border treatment

Messages are flat — there is no grouping by "turn", "trace", or "phase". Each tool result is its own message in the virtualized list.

3. Data Currently Available to the Frontend

The SSE stream delivers these event types (defined in frontend/src/lib/entities/stream.ts):

• metadata — thread_id, run_id, assistant_id, project_id
• messages — individual AIMessageChunk with tool_call_chunks, agent_name, content
• values — full state snapshot with messages, files, todos
• error, aborted, done

Critical gap: There is no timing data on any stream event. Messages have no started_at, ended_at, duration_ms, input_tokens, output_tokens, or cost_usd. The frontend has zero phase/span data today. The backend has no agent_spans table, no tracing utilities, and no trace API endpoints — these must all be built as specified.

The frontend does track TTFT (time-to-first-token) and streamingRate (tokens/sec) locally, but these are per-submission, not per-phase.

4. Visualization Libraries Available

Already in package.json:

• plotly.js-dist-min + react-plotly.js — Used for chart artifacts. Could render a Gantt chart but Plotly's Gantt support is limited to horizontal bar charts via px.timeline (not available in plotly.js). Would require manual bar trace construction.
• react-chrono — Listed as a dependency but not used anywhere in the codebase (zero imports found). This is a vertical timeline library, not suitable for horizontal swimlane visualization.
• @xyflow/react (React Flow) — Listed but not used anywhere. Designed for node-based graphs, not timeline visualization.
• react-big-calendar — Used for schedule management. Calendar-oriented, not suitable.

Recommendation: Build the phase timeline as a custom Tailwind/CSS component using div elements with percentage-based widths. This approach:

• Keeps bundle size minimal (no new dependency)
• Matches the existing design system (Tailwind + shadcn)
• Allows full control over the swimlane layout, hover tooltips, and click-to-drill-down
• Works naturally with the existing dark-mode theming (useTheme hook)

If a charting library is desired later, recharts would be a lightweight addition — but it is not needed for horizontal colored bars.

5. Proposed Component Architecture

frontend/src/components/timeline/
├── ToolTimeline.tsx          (existing - vertical tool result list)
├── ToolTimelineItem.tsx      (existing - individual tool result card)
├── PhaseTimeline.tsx         (NEW - swimlane container)
├── PhaseTimelineBar.tsx      (NEW - individual phase bar)
└── PhaseTimelineDetail.tsx   (NEW - drill-down panel)

PhaseTimeline receives an array of AgentSpan[] for a given trace and renders:

• A horizontal time axis (relative, starting at 0ms)
• Colored bars per phase type:
  • planner = blue (bg-blue-500)
  • generator = green (bg-green-500)
  • evaluator = amber (bg-amber-500)
  • handoff = purple (bg-purple-500)
  • tool_call = slate (bg-slate-500)
  • retrieval = cyan (bg-cyan-500)
• Bars positioned using percentage widths: left = (span.started_at - trace_start) / trace_duration * 100%
• <Tooltip> on hover (using existing @radix-ui/react-tooltip) showing duration, tokens, cost

Where it mounts: Two options:

1. Thread detail panel — A new "Traces" tab alongside the chat, accessible from the ChatNav or a sidebar toggle. Fetches spans via GET /threads/{id}/traces on-demand.
2. Inline per-turn — Collapsed above each AI response in the message list, expandable. This requires span data to be associated with messages.

Option 1 (dedicated panel) is simpler and avoids cluttering the chat flow. It can use the existing ResizablePanelGroup pattern from ChatPanel.tsx.

6. Real-Time Span Updates During Streaming

The spec does not explicitly address real-time span streaming. Two approaches:

A. Post-hoc only (simpler, recommended for v1): Spans are only fetched after a stream completes. The trace timeline is a retrospective view. This avoids complexity with partially-complete spans and WebSocket subscriptions.

B. Live spans via SSE: Add a new "spans" event type to the existing SSE stream. The convertEventToLegacy function in useChat.ts would need a new case. Partially-complete spans (status: "running") would render as pulsing/animated bars. This is significantly more complex and could be a v2 enhancement.

7. Click-to-Drill-Down Implementation

When a phase bar is clicked:

• Open a <Sheet> (existing shadcn component) or expand an inline <Collapsible> (existing component)
• Show full span metadata: model, token counts, cost, error info, nested child spans
• For phases like generator, show the associated messages (filter messages array by agent_name or by time window matching the span)
• The existing ToolTimelineItem expand/collapse pattern can be reused for nested spans

8. New Files/Services Needed

• frontend/src/lib/entities/trace.ts — TypeScript interfaces for AgentSpan, Trace, phase enum
• frontend/src/lib/services/traceService.ts — API client functions: getThreadTraces(threadId), getTraceSpans(traceId), getSpan(traceId, spanId)
• frontend/src/hooks/useTraces.ts — Data fetching hook with loading/error states
• frontend/src/components/timeline/PhaseTimeline.tsx — Main swimlane component
• frontend/src/components/timeline/PhaseTimelineBar.tsx — Individual phase bar
• frontend/src/components/timeline/PhaseTimelineDetail.tsx — Drill-down detail view

9. Key Risks & Open Questions

1. Backend must land first. The frontend has no trace data to consume. The agent_spans table, span lifecycle utilities, and REST API endpoints are prerequisites.
2. Span-to-message association. How are spans correlated to specific messages? By trace_id on the message? By time window? The spec defines thread_id on spans but not a direct message_id foreign key. This mapping needs to be defined.
3. Cost/token data availability. The cost_usd and token fields are None-able in the spec. The timeline should degrade gracefully when these are absent.
4. Performance. A thread with many turns could have hundreds of spans. The traces list endpoint should support pagination, and the frontend should lazy-load span details.

[ryaneggz]

Migration Strategy Note

Schema Changes Required

• New agent_spans SQL table -- this is the only spec in the batch that truly requires a new Alembic migration creating a dedicated SQL table. The spec calls for a PostgreSQL table with columns: id (UUID PK), trace_id (UUID), parent_span_id (UUID nullable), thread_id (text), phase (text/enum), model (text nullable), started_at (timestamptz), ended_at (timestamptz nullable), duration_ms (int nullable), input_tokens (int nullable), output_tokens (int nullable), cost_usd (numeric nullable), status (text), error (text nullable), metadata (jsonb nullable).
• Indexes needed on: trace_id, thread_id, started_at (for efficient timeline queries).

Migration File

• Create backend/migrations/versions/0002_add_agent_spans.py with down_revision = "0001".
• This is a pure additive migration (new table, new indexes) with no dependency on existing tables other than the uuid-ossp extension already enabled in 0000_init.

Parallel Development Safety

• Low conflict risk. This migration creates a brand new table with no foreign keys to existing user-managed tables. As long as the revision number doesn't collide with Cost Tracking (feat: Per-run cost tracking, phase-level metering, and budget caps #905), both can be developed in parallel.
• Revision numbering strategy: Use sequential numbering (0002, 0003, etc.). If feat: Per-run cost tracking, phase-level metering, and budget caps #905 is also in flight, coordinate: one takes 0002, the other takes 0003. Alternatively, use descriptive revision IDs (e.g., 0002_add_agent_spans and 0002_add_turn_metrics) and resolve the down_revision chain at merge time.

Shared Fields with Other Specs

• The input_tokens, output_tokens, cost_usd fields on agent_spans overlap significantly with Cost Tracking (feat: Per-run cost tracking, phase-level metering, and budget caps #905) turn_metrics. Consider whether these should be the same table or separate:
  • Option A (recommended): agent_spans stores per-phase trace data including tokens/cost. Cost Tracking (feat: Per-run cost tracking, phase-level metering, and budget caps #905) aggregates from agent_spans rather than maintaining a separate turn_metrics table. This avoids double-writing and data inconsistency.
  • Option B: Separate tables with turn_metrics being a denormalized summary. More storage, but simpler queries for cost-only dashboards.
• If Option A is chosen, this migration should land before feat: Per-run cost tracking, phase-level metering, and budget caps #905, and feat: Per-run cost tracking, phase-level metering, and budget caps #905 should add run_budgets and pricing_overrides tables only (not turn_metrics).

Recommendation

• This is the foundational data table for the entire harness observability story. Consider landing this migration first, then having feat: Per-run cost tracking, phase-level metering, and budget caps #905 build budget/pricing tables on top.
• Migration revision: 0002_add_agent_spans, down_revision = "0001".

[ryaneggz]

Dependency Graph Validation

Position in Dependency Graph

Agent Observability (#904) — SEMI-INDEPENDENT (Layer 1)
  └─ soft dep on: Cost Tracking (#905) for cost_usd field
  └─ soft dep on: Planner (#898), Evaluator (#899) for phase enum values
  └─ depended upon by: (nothing directly, but enriches all other specs)

Declared Dependencies

• "Works standalone; most valuable when planner + evaluator phases are active"
• "Integrates with existing cost-tracking infrastructure for cost_usd field"
• "No external service dependencies required"

Validated Dependencies

Mostly correct, but the cost tracking dependency is understated.

The AgentSpan schema includes a cost_usd: float | None field. The spec says this "integrates with existing cost-tracking infrastructure" — but there IS no existing cost-tracking infrastructure. Cost Tracking (#905) is a sibling spec that hasn't been built. This means:

1. If Observability ships before Cost Tracking: cost_usd will always be None. The field exists but is never populated.
2. If Cost Tracking ships first: Observability can read from TurnMetrics to populate cost_usd.

This is a soft dependency, not a hard one — the feature works without cost data, but the spec's language ("integrates with existing") is misleading.

Implicit Dependencies Found

1. Phase enum coupling — The AgentSpan.phase enum includes values planner, generator, evaluator, handoff, tool_call, retrieval. The planner and evaluator phases only exist if Planner (feat: Planner Agent — auto-expand short prompts into full product specs #898) and Evaluator (feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899) are built. The handoff phase only exists if Context Reset (feat: Agent phase observability — structured traces with frontend timeline #904 — this issue references feat: Agent phase observability — structured traces with frontend timeline #904 but the spec is about observability). Without those specs, only generator, tool_call, and retrieval will produce spans. This isn't a bug but means the feature is less useful in isolation.
2. @trace_phase decorator — This decorator wraps LLMController phases. It needs to be inserted at the same code locations that Planner (feat: Planner Agent — auto-expand short prompts into full product specs #898) and Evaluator (feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899) modify. If built in parallel, there will be merge conflicts in controllers/llm.py.
3. Database table — New agent_spans table with Alembic migration. Must coordinate migration ordering with Planner, Evaluator, and Cost Tracking migrations.

Interface Contracts Needed

1. Phase enum — Should be defined as a shared constant NOW, since Cost Tracking (feat: Per-run cost tracking, phase-level metering, and budget caps #905) also has a phase enum (planning, execution, reflection, tool_call) that uses DIFFERENT names for the same concepts:

   • Observability: planner vs Cost Tracking: planning
   • Observability: generator vs Cost Tracking: execution
   • Observability: evaluator vs Cost Tracking: reflection (?)

   This is a naming conflict that will cause confusion. Both specs need to agree on a single phase enum.

2. Trace ID propagation — The spec uses contextvars for trace context. Cost Tracking (feat: Per-run cost tracking, phase-level metering, and budget caps #905) captures per-turn metrics. These two systems need a shared trace_id or turn_id to correlate spans with cost data. Neither spec defines this join key.

Schema Conflict Risk

• Medium. Phase enum naming conflict with Cost Tracking. Migration ordering with 3+ other specs.

Critical Path Assessment

Not on critical path. Can be built after Planner + Evaluator + Cost Tracking to maximize value (more phases to observe). However, it CAN be built in parallel if the phase enum is agreed upon first. Recommended: Wave 2.

Milestone Note

This spec targets v0.10.0 while Planner, Evaluator, Cost Tracking target v0.9.0. This is consistent — observability ships after the features it observes.

[ryaneggz]

Testing Strategy Recommendations

Infrastructure Context

This feature introduces new database tables (agent_spans) and REST endpoints, plus a frontend timeline component. Testing patterns from tests/integration/test_distributed_stream.py (SSE streaming), tests/unit/utils/test_middleware_integration.py (middleware stack), and tests/benchmarks/test_route_benchmarks.py (endpoint performance) are directly applicable.

Recommended Tests

Unit Tests (tests/unit/schemas/test_agent_span.py)

• AgentSpan Pydantic validation: all fields, phase enum values, status enum transitions
• duration_ms computed correctly from started_at and ended_at
• parent_span_id nullable for root spans, valid UUID for nested spans
• Metadata dict accepts arbitrary key-value pairs

Unit Tests (tests/unit/utils/test_tracing.py) — CRITICAL

• start_span(): creates span with status="running", persists to store, returns span ID
• end_span(): updates ended_at, computes duration_ms, sets status="completed"
• end_span() with error: sets status="errored", includes error message
• @trace_phase context manager: auto-creates span on enter, auto-ends on exit, handles exceptions (sets errored status)
• Trace context propagation: verify trace_id propagates via contextvars across async calls. Test with nested spans (tool_call inside generator)
• @trace_phase is no-op when tracing is disabled (environment flag)

Unit Tests (tests/unit/routes/test_trace_routes.py)

• GET /threads/{id}/traces: returns grouped trace data, empty list for thread with no traces
• GET /threads/{id}/traces/{trace_id}: returns all spans for a trace, 404 for non-existent trace
• GET /traces/{trace_id}/spans/{span_id}: returns single span with full metadata

Integration Test (tests/integration/test_observability.py)

• Full flow: trigger an agent run → verify spans are created for each phase → query API returns correct span data
• Verify span ordering matches execution order (planner before generator before evaluator)

Frontend Tests

• Timeline component: renders colored phase blocks, hover shows tooltip with duration/tokens/cost
• Empty state: renders placeholder when no traces available
• Click drill-down: expanding a span block shows full metadata

Database Migration Testing

• Test agent_spans table creation via test_db fixture
• Verify indexes exist on trace_id, thread_id, started_at

OpenTelemetry Testing

• Unit test: OTLP exporter maps AgentSpan fields to OTel span attributes correctly
• Unit test: exporter is no-op when OTEL_EXPORTER_OTLP_ENDPOINT not set (zero overhead)
• Mock the OTLP endpoint with respx to verify export payload format

Gaps in Spec's Test Section

• Missing: Performance test — span creation overhead should be <1ms per span (use pytest-benchmark)
• Missing: Test for concurrent spans (parallel tool calls) — verify parent_span_id correctly nests
• Missing: Test for very long traces (100+ spans) — pagination on API endpoints
• Missing: Test for span data in SSE stream events (real-time trace updates to frontend)
• Missing: Cleanup/retention policy for old spans — test TTL or archival behavior

Shared Fixtures Needed

• mock_trace_data: a realistic trace with 4-5 spans covering planner→generator→tool_call→evaluator
• sample_agent_span: factory function creating AgentSpan with configurable phase/status

[ryaneggz]

Middleware Stack Ordering Analysis for Agent Observability (Tracing)

Current Stack Context

Orchestra's middleware is appended after deepagents' internal stack via create_deep_agent. The framework composes middleware with first in list = outermost layer. The internal deepagents stack includes: TodoList, Memory, Filesystem, SubAgent, Summarization, PromptCaching, PatchToolCalls. Orchestra's stack then adds: compaction, metadata, cache metrics, retry, PII, and AutoEvict.

Where Tracing Middleware Should Go

The observability spec calls for structured spans per agent phase with timing, token counts, model, cost, and error info. The tracing middleware has two distinct concerns:

1. Outer span: Wraps the entire model call to capture wall-clock duration, errors, and overall timing
2. Inner data capture: Reads token counts and cost from the response (similar to cache_metrics_middleware)

Recommended position: outermost in the entire Orchestra stack (position 1, before context reset).

stack = [
    tracing_middleware,          # 1. NEW: Start/end trace spans, capture timing + errors
    context_reset_middleware,    # 2. Check context utilization, trigger reset if needed (#906)
    compaction_middleware,       # 3. Summarize old messages
    add_ai_message_metadata,    # 4. Tag AI messages
    cache_metrics_middleware,    # 5. Log cache hit ratios
    retry_model,                # 6. Retry on failure
    metrics_capture_middleware,  # 7. Record tokens, cost, duration (#905)
    *pii_middleware(),           # 8-9. PII filtering
    AutoEvictMiddleware(...),    # 10. Evict large tool results
]

Rationale:

1. Must be outermost to capture everything: A tracing span should measure the full wall-clock time of a model call, including time spent in compaction (which makes its own LLM call), context reset handoff generation, retry delays, and the actual model call. If tracing were placed after retry, it would miss retry overhead. If placed after compaction, it would miss compaction time. As outermost, it wraps all other middleware and captures the true end-to-end latency.

2. Error capture: If any inner middleware throws (compaction LLM fails, budget enforcement rejects, PII blocks), the outermost tracing span catches the error and records it with status: errored. Placing tracing deeper would miss errors from outer middleware.

3. Context reset visibility: When context reset triggers (feat: Context reset with structured handoff artifacts for long-running sessions #906), the tracing span should capture the entire reset flow (handoff generation + new session bootstrap) as a single top-level span, with nested child spans for individual LLM calls within the reset. Being outermost ensures this.

Implementation Approach: Single Middleware vs. Per-Phase Decorators

The spec describes a @trace_phase decorator for per-phase tracing. Two approaches:

Option A: Single wrap_model_call middleware (recommended for the Orchestra stack)

• A single middleware at position 1 creates a span around each model call
• Uses contextvars to propagate trace_id so nested calls (compaction LLM, handoff LLM) automatically become child spans
• The span captures: start time, end time, model, token counts (from response), cost (from response usage metadata), error (if any)

Option B: Multiple decorators on each phase

• @trace_phase("compaction") on compaction middleware
• @trace_phase("model_call") on the actual LLM call
• More granular but harder to maintain and requires modifying every middleware

Recommendation: Use Option A for the Orchestra middleware stack, and use @trace_phase decorators inside deepagents' internal code if/when that upstream project adds native tracing. The single outermost middleware plus contextvars propagation gives you both the top-level span and nested child spans without touching existing middleware code.

Interaction with Cost Tracking (#905)

The tracing span schema includes cost_usd, input_tokens, and output_tokens fields. These overlap with the cost tracking middleware's per-turn metrics. Two strategies:

1. Independent capture: Both middlewares read usage_metadata independently. Tracing stores spans in agent_spans table; cost tracking stores records in turn_metrics table. Simple but duplicated.

2. Shared capture (preferred): Cost tracking middleware writes to turn_metrics, and the tracing middleware reads the cost from a shared contextvars value set by the cost tracking middleware. Since cost tracking runs inner to tracing, it completes first, and the tracing middleware can read the computed cost when it finalizes the span.

Note: If cost tracking is at position 7 (inner) and tracing is at position 1 (outer), the execution flow is:

tracing.start_span()
  -> context_reset (maybe)
    -> compaction (maybe)
      -> retry
        -> cost_tracking.record()  # sets contextvars cost
          -> LLM call
        -> cost_tracking.persist()
      -> retry done
    -> compaction done
  -> context_reset done
tracing.end_span(cost=contextvars.cost)  # reads cost set by inner middleware

This composition is clean and avoids duplicate cost calculation.

Interaction with Context Reset (#906)

When context reset produces a handoff, that handoff LLM call should appear as a child span under the parent trace. Since tracing is outermost and uses contextvars for trace propagation, the handoff's inner LLM call will automatically inherit the trace_id and create a child span. The parent span's phase can be labeled "handoff" and the child span "generator".

Tool Call Tracing

The spec mentions tool_call as a phase. The AutoEvictMiddleware already uses awrap_tool_call. Tracing should also implement awrap_tool_call to create spans around tool executions. Since middleware composition for tool calls also follows "first = outermost", tracing should be listed before AutoEvictMiddleware in the stack (which it already is at position 1 vs. AutoEvict at position 10). This means tool call spans will include eviction time, which is correct.

OpenTelemetry Export

The OTLP exporter should be a separate, non-middleware concern. It subscribes to span events (or reads from the agent_spans table) and exports asynchronously. It should NOT be in the middleware stack -- keep it as a background service configured via environment variables. This prevents OTLP network latency from affecting agent response times.

[ryaneggz]

SSE Streaming Design — Observability Spans (Real-Time Trace Updates)

Context

Agent phase observability produces structured AgentSpan records with timing, token counts, cost, and error info. These spans need to update in the frontend timeline component in real-time as phases start and complete — not just after the entire run finishes.

Recommended Approach

Use "custom" stream mode with "span" sub-type. Same mechanism as cost (#905), evaluation (#899), planning (#898), and context resets (#906).

Backend — emit span lifecycle events from @trace_phase decorator / start_span / end_span utilities:

# When a phase starts (e.g., planner, generator, evaluator, tool_call):
stream_writer(("custom", {
    "type": "span",
    "data": {
        "event": "span_start",
        "trace_id": "trace-uuid",
        "span_id": "span-uuid",
        "parent_span_id": "parent-uuid-or-null",
        "phase": "evaluator",          # "planner" | "generator" | "evaluator" | "tool_call" | "handoff"
        "name": "Evaluate output quality",
        "started_at": "2026-03-24T15:30:00Z",
        "model": "anthropic:claude-sonnet-4-20250514",
    }
}))

# When a phase completes:
stream_writer(("custom", {
    "type": "span",
    "data": {
        "event": "span_end",
        "trace_id": "trace-uuid",
        "span_id": "span-uuid",
        "phase": "evaluator",
        "ended_at": "2026-03-24T15:30:12Z",
        "duration_ms": 12340,
        "input_tokens": 2400,
        "output_tokens": 580,
        "cost_usd": 0.0038,
        "error": null,
        "metadata": {"iteration": 2, "score": 0.92},
    }
}))

Frontend consumption:

1. Handle custom events where payload.type === "span" in useChat.ts
2. Accumulate spans in a traceState ref: Map<spanId, SpanData>
3. On span_start: add a new entry with status: "running", start rendering a growing timeline block
4. On span_end: update the entry with duration, tokens, cost; mark status: "complete"
5. The swimlane timeline component reads from traceState and renders:
   • Color-coded horizontal blocks per phase (planner=blue, generator=green, evaluator=orange, tool_call=purple)
   • Hover tooltips: duration, token counts, cost
   • Click drill-down: full span details including error info
6. Parent-child relationships (via parent_span_id) enable nested visualization

Relationship to cost streaming (#905):

Span events include per-phase cost_usd and token counts. The cost tracking feature (#905) provides the aggregate running total. They are complementary:

• "cost_update" events give the user a simple running total badge
• "span" events give the detailed per-phase breakdown in the timeline

The cost middleware can emit its "cost_update" event by summing completed span costs, avoiding redundant LLM call tracking.

Performance consideration: Span events are low-frequency (one start + one end per phase, typically 2-6 phases per turn). This adds negligible overhead to the SSE stream. For OpenTelemetry export, use the REST API (GET /threads/{id}/traces) rather than SSE — external tools poll or receive push via OTLP.

Distributed mode: Works automatically via existing Redis stream passthrough.

Unified Custom Event Architecture

Across all five features, the pattern is consistent:

Feature	Custom sub-type	Typical events/turn
Cost tracking (#905)	cost_update	1-3
Evaluation (#899)	eval_progress	2-8
Planning (#898)	plan_status	2-4
Context resets (#906)	context_reset	0-2
Observability (#904)	span	4-12

All share the same wire format: ["custom", {"type": "<sub_type>", "data": {...}}]. The frontend needs a single custom event dispatcher in useChat.ts that routes on payload.type to the appropriate state handler. This is far more maintainable than adding five new top-level stream modes.

One-time frontend plumbing needed:

1. Add "custom" case to parseEvent() in fetchStreamReader.ts (and ResponseBodyReader)
2. Add CustomEvent to SSEEventType union in frontend/src/lib/entities/stream.ts
3. Add case "custom" in convertEventToLegacy() in useChat.ts
4. Add a custom event dispatcher in handleMessages() that switches on payload[1].type

After this one-time setup, each new feature only needs to: (a) emit ("custom", {...}) on the backend, and (b) add a handler case in the frontend dispatcher.