Skip to content

production.md

Latest commit

 

History

History
136 lines (98 loc) · 3.72 KB

production.md

File metadata and controls

136 lines (98 loc) · 3.72 KB

Production tracing (OTLP export)

Export sessions as OpenTelemetry spans to your existing observability stack. Sessions become traces. Tool calls become spans with duration and inputs. Errors get exception events. No new dependencies.

OTel GenAI semantic conventions

Use --format otlp-genai to export with OTel GenAI semantic conventions. This produces AI-native spans that populate token usage charts, cost views, and LLM dashboards automatically.

agent-strace export <session-id> --format otlp-genai \
  --endpoint http://localhost:4318
Aspect --format otlp --format otlp-genai
LLM calls Events on root span gen_ai.client.operation child spans
Tool calls tool/<name> spans gen_ai.tool.call/<name> spans
Root span agent.name attribute gen_ai.agent.id + gen_ai.agent.name
Errors Custom error span OTel exception event format

--format otlp is unchanged for backwards compatibility. See ADR-0011 for design rationale.

Per-backend setup

Datadog

# Via the Datadog Agent's OTLP receiver (port 4318)
agent-strace export <session-id> --format otlp \
  --endpoint http://localhost:4318

# Via Datadog's OTLP intake directly
agent-strace export <session-id> --format otlp \
  --endpoint https://http-intake.logs.datadoghq.com:443 \
  --header "DD-API-KEY: $DD_API_KEY"

Honeycomb

agent-strace export <session-id> --format otlp \
  --endpoint https://api.honeycomb.io \
  --header "x-honeycomb-team: $HONEYCOMB_API_KEY" \
  --service-name my-agent

Grafana Tempo / Jaeger

# Local collector
agent-strace export <session-id> --format otlp \
  --endpoint http://localhost:4318

New Relic

agent-strace export <session-id> --format otlp \
  --endpoint https://otlp.nr-data.net \
  --header "api-key: $NEW_RELIC_LICENSE_KEY"

Splunk

agent-strace export <session-id> --format otlp \
  --endpoint https://ingest.<realm>.signalfx.com \
  --header "X-SF-Token: $SPLUNK_ACCESS_TOKEN"

Langfuse

Sessions appear as Traces, tool calls as Spans, LLM calls as Generations, and eval scores as Langfuse Scores.

export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...

# Export latest session with eval scores
agent-strace export --scores --backend langfuse

# Export last 7 days
agent-strace export --since 7d --scores --backend langfuse

# Self-hosted Langfuse
agent-strace export --scores --backend langfuse \
  --langfuse-host https://langfuse.your-domain.com

Behavioral metrics

Export per-session behavioral metrics as OTLP gauge metrics:

export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io

agent-strace export --metrics --backend otlp --since 30d
Metric Description
agent_strace.session.cost_usd Estimated cost per session
agent_strace.session.error_rate Errors / tool calls
agent_strace.session.retry_rate Consecutive same-tool retries / tool calls
agent_strace.session.blast_radius Distinct files written
agent_strace.session.duration_s Wall-clock session duration
agent_strace.eval.score Judge score per session

How sessions map to OpenTelemetry

agent-trace OpenTelemetry
session trace
tool_call + tool_result span (with duration)
error span with error status + exception event
user_prompt event on root span
assistant_response event on root span
session_id trace ID
event_id span ID
parent_id parent span ID

Inspect the OTLP payload

agent-strace export <session-id> --format otlp > trace.json