Epic: add full OpenTelemetry observability across st2 and distributed agent workflows

[schickling]

Goal

Make OpenTelemetry the coherent observability contract for st2: correlated traces, metrics, and structured logs for every st2-owned operation, followed by explicit context propagation across process and durable-message boundaries.

The end state is distributed, but delivery is phased. Milestone 1 instruments the st2 control plane without claiming visibility into work performed inside an arbitrary harness. Later milestones add provider-neutral Agent Run/Session correlation and typed propagation only where the receiving process can establish a truthful bounded execution scope.

Telemetry is descriptive evidence. It never grants authority, drives reconciliation, replaces durable receipts/inventory, or changes lifecycle outcomes.

Completed slice: reconciliation signals (2026-08-26)

Merged atomically in #347, #348, and #350 (stack #349; main 19485cf):

• endpoint-driven, synchronous OTLP/HTTP JSON traces, metrics, and correlated structured logs;
• process-oriented service.name values and machine-version service.version;
• semantic reconciliation hierarchy (st2.reconcile_pass with bounded lock, discovery, hook, materialization, runtime-observation, and execution children), each with span.label;
• RED-minimal bounded-cardinality metrics with seconds-scale histogram buckets;
• systemd OTEL_* propagation and allocation-free exporter-disabled paths;
• otelite/Nix gates proving exactly one trace tree, metric point set, and correlated log, plus no-export and clean-shutdown controls;
• live dev3 Alloy → Tempo/Mimir/Loki → Grafana readback from real st2 traffic.

This completes the reconciliation slice only. The unchecked workstreams below remain authoritative for the broader epic: other control-plane surfaces, standard exporter-variable completeness, adverse controls/overhead budgets, and distributed context propagation.

Current gap and feasibility evidence

At main commit 6376c410, st2 has no structured logging, metrics, tracing, or OpenTelemetry dependency and is intentionally synchronous/no-Tokio (Cargo.toml). The architecture already exposes good instrumentation seams:

• one coherent reconciliation pass and typed UpReport outcomes (run.rs, run.rs);
• a narrow side-effecting Runner boundary over PTY and exec operations (run.rs);
• explicit DING delivery states and FIFO transitions (ding/mod.rs);
• digest-bearing catalog transaction receipts (agent_publish.rs);
• stable runtime-generation evidence and a build identity already intended for telemetry (task_inventory.rs, version.rs).

Research against the current OpenTelemetry Rust 0.32 family established:

• OTLP/HTTP traces, metrics, and logs can be exported from a plain synchronous fn main() using SDK-owned worker threads and a blocking HTTP client; Tokio is not required;
• a disposable positive-control process emitted all three OTLP paths (/v1/traces, /v1/metrics, /v1/logs) with correlated trace/log IDs and clean provider shutdown;
• an unreachable-endpoint failure control showed that explicit force_flush() reports failures that trace/log shutdown alone may not surface, so shutdown success cannot be treated as delivery proof;
• traces remain Beta; metrics API/SDK are Stable and OTLP is RC; logs API/SDK and the tracing appender are Stable while logs OTLP is RC (upstream status).

The initial implementation should therefore prefer tracing as st2's application-facing span/event API, direct OTel instruments for domain metrics, OTLP/HTTP protobuf to an external Collector, and the supported non-Tokio batch/periodic processors. Adding Tokio solely for telemetry or enabling experimental exporter retries needs separate measured justification.

Accepted architecture

st2 CLI / st2 up / st2 ding / st2 eval
        |
        +-- structured tracing spans and events
        +-- explicit OTel metric instruments
        `-- bounded in-process OTel SDK
                         |
                         `-- OTLP --> external Collector/backend

later distributed milestones:

bounded synchronous call  : parent -> child span
durable message/adoption  : prior context ~link~> new bounded operation
long-lived agent process  : independent service, correlated by domain IDs

Each independently executing process is an OTel service instance. Use:

• service.name=st2;
• service.version=st2::version::machine_version();
• a unique service.instance.id for the emitting process;
• instrumentation scope compoundingtech.st2, versioned with st2;
• a bounded st2.component.name=cli|supervisor|ding|eval attribute (final placement—resource or signal attribute—is an open question).

Per-agent, per-task, per-message, Session, Run, runtime-generation, and launch-generation identities are not process resources on the multi-agent supervisor. They are operation correlation attributes/links and must remain pairwise distinguishable. Trace IDs are never substituted for domain IDs.

Signal contract

Traces

Add bounded root spans for CLI commands and independent long-running operations rather than one supervisor- or session-long span. Initial coverage should include:

• st2.command and st2.reconcile;
• reconciliation stages: lock, discover, hooks, materialize, observe, plan, execute;
• runtime operations: list/observe, spawn, reap, kill, remove, presentation patch;
• bounded external pty, systemctl, and related subprocess calls;
• DING backlog adoption, delivery attempt, receipt classification, deferral, retry, and archive settlement;
• message send/list/read/archive/reply operations;
• agent publish and catalog diff/snapshot/bootstrap/apply transactions;
• eval setup, run steps, judge collection, and teardown.

Every span needs a concise span.label, typed outcome/error status, duration, and only the attributes necessary to identify the bounded operation. A successful terminal poke is not proof that the message was consumed; span names/statuses must preserve that distinction.

Metrics

Define and version a project-owned st2.* metric contract using OTel names and UCUM units. Prometheus translation adds underscores/unit/counter suffixes; do not bake Prometheus _total names into OTel instruments.

Initial instruments:

Instrument	Type/unit	Bounded dimensions	Operator question
st2.reconcile.duration	histogram, s	trigger, outcome	Is reconciliation slow or unexpectedly frequent?
st2.reconcile.stage.duration	histogram, s	stage, outcome	Which stage is slow/failing?
st2.reconcile.action.count	counter, {action}	action, outcome	Is the fleet converging or churning?
st2.runtime.operation.duration	histogram, s	operation, backend, outcome, bounded error.type	Is PTY/exec wedged or unreliable?
st2.task.count	observable up/down counter, {task}	kind, lifecycle, desired state, runtime state	Where does desired state disagree with observation?
st2.watch.event.count	counter, {event}	component, event kind, outcome	Are watchers noisy or self-waking?
st2.inbox.scan.duration	histogram, s	trigger, outcome	Is idle DING repeatedly/slowly scanning?
st2.ding.notice.count	observable up/down counter, {notice}	pending/staged, notice kind	Is FIFO work accumulating?
st2.ding.notice.oldest.age	observable gauge, s	pending/staged	How long has owned work been blocked?
st2.ding.delivery.duration	histogram, s	operation, guarded outcome	Which delivery state is slow/failing?
st2.message.operation.duration	histogram, s	operation, outcome, bounded error.type	Is the filesystem bus slow/failing?
st2.message.archive.delay	histogram, s	none	How long from send time to durable archive receipt?

Use standard process instruments where their semantic-convention stability and platform support are acceptable rather than inventing st2-prefixed CPU/RSS/thread/fd copies. #22 and #23 should become explicit regression scenarios.

Metric dimensions use a reviewed allowlist. Agent/task/session/run/generation/message/trace IDs, PIDs, paths, filenames, sender/recipient, tags, free-form error strings, and provider identities are forbidden metric attributes. Sampled trace exemplars provide drill-down without creating an unbounded series per operation.

Observable task metrics must reuse the coherent runtime observation from reconciliation/inventory; they must not trigger an independent backend scan that can disagree with the pass or add load.

Structured logs

Introduce structured tracing events and bridge them to OTel logs with TraceId/SpanId correlation. Preserve existing human stdout/stderr, machine JSON, exit codes, and bounded task/crash logs byte-for-byte; OTel logs are a parallel signal, not a reinterpretation or replacement.

OTLP exporter/client diagnostics must be filtered out of the OTel log bridge to prevent telemetry-induced telemetry loops. Exporter failures use a separate, rate-limited local diagnostic path.

Activation and export

Activation is endpoint/exporter driven:

• with no generic or signal-specific standard OTel exporter configuration, install no providers, start no telemetry workers, and perform no DNS/network access;
• OTEL_EXPORTER_OTLP_ENDPOINT activates all supported OTLP signals unless a signal-specific exporter disables/overrides one;
• signal-specific exporter/endpoint settings activate only that signal;
• OTEL_SDK_DISABLED=true always forces the no-op path;
• once active, honor standard signal-specific-over-generic endpoint, protocol, TLS, certificate, header, compression, timeout, sampling, limit, batch, and metric interval settings where the Rust SDK supports them;
• never log OTLP header/certificate values;
• no embedded backend and no st2-owned durable telemetry spool.

The Rust SDK does not auto-install providers from environment configuration, so this activation/precedence contract belongs to st2 and needs direct tests. Unsupported standard values must produce one actionable, rate-limited local diagnostic and remain fail-open; they must not silently select a different protocol.

Telemetry initialization, export, force-flush, and shutdown never replace the primary command result. One-shot commands use one shared bounded final-flush deadline across all active signals; provider shutdown runs exactly once. SIGKILL/abort remains explicitly best-effort with no flush guarantee.

Distributed context and identity

Durable messages

Add optional typed traceparent and tracestate fields through the raw message frontmatter, Message, JSON surfaces, and st2-wire. Older readers already tolerate additive fields and archive settlement preserves message bytes, but typed surfaces currently discard unknown fields; mixed-version behavior must be explicit.

• send creates a bounded producer span and injects its context;
• DING delivery attempts, message reads, archive/settlement, replies, and eventual agent work create their own bounded spans linked to message creation context;
• durable asynchronous work and supervisor adoption use span links by default, not an unbounded parent-child trace;
• replies/conversations are domain relationships, not automatic trace parentage;
• invalid, oversized, or unsupported context never makes a message unreadable and never crashes delivery;
• baggage is disabled by default. Any future baggage support requires a small allowlist, size limits, trust-boundary stripping, and independent review.

Managed processes

Do not persist per-launch traceparent in a task's ordinary PTY environment: that environment is fingerprinted/replayed on manual restart and would resurrect stale trace context. Add a distinct ephemeral launch-context seam if process-start propagation is implemented.

The launch span ends after bounded readiness. A long-lived agent is a separate service/run, not a child span kept open for the session. st2 cannot claim provider work is traced until the relevant harness driver can establish and clear a per-operation execution scope; provider-specific and GenAI conventions remain outside generic st2 core.

Agent Session and Agent Run IDs from #124/#145 remain the universal correlation baseline when no sampled trace is present. They are attributes/links, never trace IDs, baggage-based authority, or lifecycle capabilities.

Privacy, security, and cardinality

Default-deny telemetry capture of:

• message subject/body/tags and context/resource/rendered-file contents;
• terminal/pane content, prompts, tool input/output, transcripts, or provider payloads;
• command lines/argv, environment keys or values, OTLP headers, credentials, tokens, or certificates;
• absolute catalog/workspace/state paths and raw free-form errors.

Use stable typed outcome/error codes and bounded enums. Exact agent/task/Run/Session/message/generation IDs may appear on sampled spans/logs only after an explicit privacy review; they never appear as ordinary metric dimensions. Incoming W3C context is untrusted correlation data: validate and bound it, apply local sampling/cost policy, and never treat it as authorization.

Failure-capable acceptance

Semantic oracle

Use in-memory span/log exporters and a manual metric reader with explicit force-flush barriers—no sleeps—to assert one known operation produces the exact span tree/status/attributes, correlated structured logs, and metric points. Run each signal independently disabled to prove there is no hidden cross-signal dependency.

Add an in-process OTLP/HTTP receiver that decodes protobuf for /v1/traces, /v1/logs, and /v1/metrics. A pinned Collector smoke test may supplement it, but is not the semantic oracle.

Required adverse controls

• unconfigured and OTEL_SDK_DISABLED=true: no provider/worker/network, and unchanged stdout, stderr, JSON bytes, exit status, reconcile result, process/thread lifecycle, and managed-task survival;
• invalid config, closed port, accept-and-stall, malformed response, 429, and 503: bounded time/queues/RSS; no mutation of lifecycle reports or ownership state;
• queue overflow: producers do not block; drops are visible through a non-recursive diagnostic surface;
• exporter internal logs: no recursive export, deadlock, or unbounded log growth;
• final flush failure: visible locally but never overwrites the command's primary result; success is proven by explicit flush receipt rather than shutdown return alone;
• SIGINT/SIGTERM: providers shut down once, supervisor locks release, and detached PTY/exec tasks remain alive;
• pre_exec/setsid: no telemetry work runs after fork in the child; process group, PID/start-token, adoption, and Nomad-style survival invariants remain unchanged;
• supervisor restart/adoption: a new supervisor operation links to existing generation evidence rather than inventing continuous parentage;
• initial PTY launch/manual restart: no stale launch context replay and identical intended propagation semantics;
• privacy fixture: decoded OTLP contains none of the prohibited data;
• cardinality fixture: every metric attribute/value comes from the reviewed bounded schema;
• two messages handled by one long-lived agent: no trace-context bleed;
• old -> new, new -> old, and new -> new message versions: safe degradation to domain-ID correlation;
• invalid/oversized traceparent/tracestate: message remains usable and no invalid context is inherited.

Overhead evidence

Before merge, record telemetry-free versus instrumented release binary/closure size, cold-start time, p50/p95 command/reconcile CPU and wall time, RSS, allocation/thread count, and enabled-unreachable-collector behavior. Establish explicit budgets from repeated real-process measurements. A standalone three-signal probe is not a valid estimate of the final st2 delta because dependency overlap and optimization change the result.

Workstreams

• M0: accept and version the resource, span, log, metric, privacy, cardinality, activation, and failure contracts.
• M1: add the internal instrumentation facade and endpoint-driven no-op/OTLP initialization.
• M1: instrument control-plane commands, reconciliation, runtime operations, DING, messages, catalog transactions, and evals.
• M1: add deterministic three-signal semantic, protocol, failure, and overhead evidence.
• M2: consume provider-neutral Agent Session/Run identity after Define renderer-owned harness sessions and the generic st2 lifecycle boundary #124/Model Agent Run identity in task inventory #145 and expose pairwise-distinct correlation attributes.
• M3: add the optional versioned message carrier and link-based send/read/archive/DING semantics with mixed-version tests.
• M3: add an ephemeral process-launch carrier that cannot enter the persisted launch fingerprint/environment.
• M4: add provider/harness execution-scope adapters and cross-repository E2E traces before claiming agent work is covered.
• M4: publish operator queries/dashboards and alert examples only after the emitted metric contract is stable.

Dependencies and related work

• Model Agent Run identity in task inventory #145 — provider-neutral Agent Run identity; primary distributed-correlation dependency.
• Define renderer-owned harness sessions and the generic st2 lifecycle boundary #124 — logical Agent Session and provider-native identity ownership.
• Define Axe harness drivers and extract all harness-specific behavior from st2 core #162 — provider-specific behavior belongs in harness drivers, not st2 core.
• Expose desired versus observed launch generation for adopted tasks #40 / Add an exact generation-fenced recreate operation for one live task #110 — launch generation and generation-fenced recreation remain distinct lifecycle evidence.
• Epic: stable agent identity and mutable presentation metadata #128 — stable Agent Spec identity owns automation/telemetry identity; presentation metadata does not.
• Expose host-qualified, non-authoritative reconciler observations per agent #79 — durable host-qualified observations remain available when telemetry is absent.
• st2 ding consumes ~60% CPU while waiting on an idle inbox #22 / st2 up consumes ~40% CPU while supervising an idle stable catalog #23 — watcher/supervisor CPU regressions become observability and overhead controls.
• ExecBackend::remove deletes the exec log as part of GC, so a crash-looping sidecar erases its own diagnostic on every relaunch #15 / Activation: deploy managed PTY exit-diagnostic retention #34 — local bounded crash/exit diagnostics remain independent of OTLP availability.
• Define Agent Spec schema versioning and compatibility semantics #127 — Agent Spec schema versioning remains separate from telemetry schema/convention versioning.

Non-goals

• embedding a Collector, backend, query engine, dashboard store, or durable telemetry spool;
• replacing task inventory, catalog receipts, local crash logs, exit codes, or CLI JSON with telemetry;
• capturing message/prompt/transcript/terminal content;
• treating trace/baggage context as identity, authorization, delivery proof, or lifecycle truth;
• making experimental GenAI semantic conventions part of generic st2 core;
• introducing Tokio solely for telemetry without a measured requirement.

Remaining design questions

1. Which exact standard exporter variables activate each signal, and how do explicit none values interact with a generic endpoint?
2. Is initial support intentionally OTLP/HTTP protobuf only, or must standard grpc configuration work in M1 despite its runtime/dependency cost?
3. What is the one shared final-flush deadline, and should a strict test-only mode make flush failure fatal?
4. What local non-recursive self-diagnostic contract reports queue drops/export failures while the Rust SDK's self-observability matures?
5. Which exact IDs are allowed on sampled spans/logs by default, and which require an operator opt-in?
6. Should st2.component.name be a process resource attribute or a common signal attribute, especially for one-shot CLI invocations?
7. Which process metrics are stable/portable enough for M1, and what platform-specific gaps are acceptable?
8. What sampling policy applies at untrusted message boundaries so a remote sampled flag cannot dictate local cost?
9. Should one-shot CLI coverage land with the daemons or after bounded force-flush semantics have independent proof?

Primary standards: OTel error handling, OTLP exporter configuration, Rust component status, messaging spans, context propagators, W3C Trace Context, metric naming, and handling sensitive data.

Posted on behalf of @schickling

field	value
agent_identity	unknown
agent_persona	generalist
agent_supervisor	unavailable
agent_tool	OMP
agent_tool_version	18.0.3
agent_runtime	OMP 18.0.3
tooling_profile	dotfiles@e4789b0