feat(otel): tracing facade — unified span+log subscriber, diagnostics migration, log CI assertions (#350)

* feat(otel): tracing facade — unified span+log subscriber, diagnostics migration, log CI assertions

PR3 per interview decision Q6 (docs/vrs/06-observability/open-questions.md):

- Telemetry::init now installs one tracing subscriber: a stderr fmt layer
  runs unconditionally so migrated diagnostics stay visible with or without
  an endpoint (documented deviation from PR1's zero-output unset case;
  RUST_LOG overrides the INFO default), plus — behind the endpoint guard —
  a tracing-opentelemetry span layer and an opentelemetry-appender-tracing
  bridge over a new SDK logger provider sharing endpoint/protocol/resource.
  experimental_use_tracing_span_context stamps active span ids onto records.
  shutdown flushes logger+meter+tracer together.
- st2.reconcile_pass spans move to tracing::info_span! at all three sites
  (up_once, up_loop_until pass, reconcile_pass_specs_with_sessions); names
  and st2.host/st2.crash_loops/st2.unparked attributes unchanged; metric
  recording calls untouched. Each pass emits a deterministic INFO record
  (target st2, "reconcile pass complete", result=pass|fail).
- Non-user-facing eprintln! diagnostics migrate to tracing::warn!/error!
  with unchanged text: crash-loop give-ups and supervisor notification,
  park-channel setup, catalog watching, ding transport ambiguity, driver
  session degradation, harness-state floor writes. CLI command output,
  boot banners, and validation reports stay println!/eprintln!.
- tests/otel_export.rs additionally asserts logs.ndjson carries the
  completion record (body/severity) correlated to the captured
  st2.reconcile_pass span via trace/span ids; existing span+metric
  assertions unchanged.

agent-identity: unknown
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: OMP
agent-tool-version: 18.0.3
agent-runtime: OMP 18.0.3
tooling-profile: dotfiles@929dc21

* fix(otel): scope RUST_LOG to stderr layer; make metric setup best-effort

Two review findings on the log-bridge PR:

1. The EnvFilter was attached to the base subscriber, so RUST_LOG=warn
   silently disabled INFO span/log export even with an endpoint set.
   The stderr fmt layer now carries the filter alone. The OpenTelemetry
   layers carry their own static filter, independent of RUST_LOG, that
   silences only exporter-internal targets (opentelemetry*): the SDK's
   BatchLogProcessor emits tracing events from inside its own emit(),
   and feeding those back through the unfiltered log bridge recursed
   until the export thread overflowed its stack (gdb-proven). App
   signals export unfiltered.

2. A metric exporter build failure early-returned before logger setup,
   killing logs along with metrics. Metric setup is now best-effort:
   warn and continue to the logger provider.

agent-identity: unknown
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: OMP
agent-tool-version: 18.0.3
agent-runtime: OMP 18.0.3
tooling-profile: dotfiles@929dc21

* fix(otel): keep piped stderr diagnostics plain — fmt layer ansi follows tty

tracing_subscriber's fmt layer enables ANSI whenever the 'ansi' feature is on, regardless of destination, so redirected stderr carried SGR escapes where the pre-migration eprintln! path emitted plain bytes. Gate the layer on std::io::stderr().is_terminal().

agent-identity: unknown
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: OMP
agent-tool-version: 18.0.3
agent-runtime: OMP 18.0.3
tooling-profile: dotfiles@929dc21

* fix(otel): emit one correlated signal set and shut providers down cleanly

agent-identity: unknown
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: OMP
agent-tool-version: 18.0.3
agent-runtime: OMP 18.0.3
tooling-profile: dotfiles@e4789b0

* feat(otel): add semantic reconciliation trace hierarchy

agent-identity: unknown
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: OMP
agent-tool-version: 18.0.3
agent-runtime: OMP 18.0.3
tooling-profile: dotfiles@e4789b0

Author: schickling-assistant

Cargo.lock

@@ -37,8 +37,14 @@ st2-wire = { path = "crates/st2-wire" }
 tempfile = "3"
 toml = "0.9"
 opentelemetry = "0.30"
-opentelemetry-otlp = { version = "0.30", default-features = false, features = ["http-json", "reqwest-blocking-client", "internal-logs"] }
-opentelemetry_sdk = { version = "0.30", features = ["spec_unstable_metrics_views"] }
+opentelemetry-otlp = { version = "0.30", default-features = false, features = ["http-json", "reqwest-blocking-client", "internal-logs", "logs"] }
+opentelemetry_sdk = { version = "0.30", features = ["spec_unstable_metrics_views", "logs"] }
+# `experimental_use_tracing_span_context` stamps the active tracing span's trace/span ids onto
+# emitted log records — required so logs correlate with spans without manual context plumbing.
+opentelemetry-appender-tracing = { version = "0.30", features = ["experimental_use_tracing_span_context"] }
+tracing = "0.1"
+tracing-opentelemetry = "0.31"
+tracing-subscriber = { version = "0.3", default-features = false, features = ["ansi", "env-filter", "fmt", "registry", "std"] }
 tungstenite = "0.30"
 
 [dev-dependencies]

Cargo.toml

@@ -1,6 +1,6 @@
 # All three signals via a three-PR gh stack
 
-Status: draft
+Status: accepted
 
 Recorded 2026-08-25 from the aligned observability interview (axe decision catalog Q1 + Q3).
 
@@ -17,6 +17,12 @@ is shared by every signal; once it lands, metrics and logs are incremental. Defe
 invites "traces shipped, rest never happens" — and the CI proof obligation (otelite capture +
 assertions) is signal-generic anyway.
 
+## Evidence and Argument
+
+The exporter/provider plumbing and otelite capture harness are shared across all three signals,
+while rates/durations require metrics and correlated diagnostics require logs. Splitting delivery
+at signal boundaries isolates review and CI failures without treating traces as the finished scope.
+
 ## Decision
 
 **Q1 — Scope**: all three signals (traces, metrics, logs) are the target. Not traces-only; not
@@ -31,18 +37,14 @@ traces-first-with-maybe-later.
 Each PR lands green independently; 2 and 3 build on 1's shared plumbing
 (provider/resource/exporter) only.
 
-## Alternatives considered
-
-- **Traces-first, defer the rest** — rejected: defers most of the value (rates/durations live in
-  metrics; correlated diagnostics in logs) for no real risk reduction, since the risky part
-  (feature set, sync export, flush-at-exit) is identical across signals.
-- **Single PR** — rejected: couples an unreviewable diff (instrumentation across `run.rs`,
-  `exec_backend.rs`, `hooks.rs`, plus unit changes and test harness) to the plumbing; a regression
-  in any slice blocks all of it.
-- **Spool-files instead of direct OTLP export** — writing telemetry records to local spool files
-  for a separate shipper to forward — rejected: adds a moving part st2 must own (rotation,
-  retention, crash-safety) to solve a problem the fleet pipeline already solves at
-  `127.0.0.1:4318`; the ambient-endpoint no-op contract would need re-inventing.
+## Options
+
+| Option | Result | Reason |
+| --- | --- | --- |
+| All three signals in a three-PR stack | Selected | Shares the risky plumbing while isolating signal-specific review and CI failures. |
+| Traces first, defer the rest | Rejected | Defers rates, durations, and correlated diagnostics without reducing exporter risk. |
+| Single PR | Rejected | Couples all instrumentation sites and the capture harness into one blocking review surface. |
+| Spool files instead of direct OTLP | Rejected | Adds rotation, retention, and crash-safety machinery for a pipeline the fleet already provides. |
 
 ## Consequences
 

docs/vrs/06-observability/.decisions/0001-all-signals-via-pr-stack.md

@@ -1,6 +1,6 @@
 # Numbered subsystem under st2's VRS root, referencing dotfiles' central tree
 
-Status: draft
+Status: accepted
 
 Recorded 2026-08-25 from the aligned observability interview (axe decision catalog Q2).
 
@@ -19,6 +19,20 @@ central tree (`01-conventions`, `09-integration/spec.md`). But st2-specific deci
 feature set, trace roots in `src/run.rs`, unit env propagation — have no home there and would be
 invisible to anyone working in this repo.
 
+## Evidence and Argument
+
+The repository already uses numbered VRS subsystems for product-specific lifecycle contracts,
+while the dotfiles observability tree owns fleet-wide semantics. A local subsystem with explicit
+central references preserves both contributor locality and one authority per shared rule.
+
+## Options
+
+| Option | Result | Reason |
+| --- | --- | --- |
+| Numbered st2 subsystem | Selected | Keeps local mechanisms and evidence beside their code while referencing central rules. |
+| Standalone st2 documentation root | Rejected | Duplicates the established VRS hierarchy. |
+| Central dotfiles tree only | Rejected | Leaves st2-specific crate, process, and CI decisions without repository-local authority. |
+
 ## Decision
 
 This tree is a **numbered subsystem under st2's existing `docs/vrs` root**: `06-observability`

docs/vrs/06-observability/.decisions/0002-subsystem-placement-and-central-reference.md

@@ -1,6 +1,6 @@
 # Blocking HTTP-JSON exporter, single-client feature set, explicit flush at exit
 
-Status: draft
+Status: accepted
 
 Recorded 2026-08-25 from the aligned observability interview, backed by prototype evidence
 ([../.experiments/2026-08-25-rust-to-otelite-capture.md](../.experiments/2026-08-25-rust-to-otelite-capture.md)).
@@ -24,6 +24,22 @@ Two traps surfaced during prototyping:
 2. **The async-batch trap.** The default async batch exporter requires a tokio runtime; under
    st2's sync process model it panicked at export time.
 
+## Evidence and Argument
+
+The linked otelite prototype exported real OTLP/HTTP JSON from a synchronous Rust process only
+with the blocking client and an exclusive reqwest-client feature selection. It reproduced both
+the dual-client `NoHttpClient` failure and the async exporter's missing-reactor panic, and proved
+that explicit shutdown delivers short-lived spans.
+
+## Options
+
+| Option | Result | Reason |
+| --- | --- | --- |
+| Blocking HTTP-JSON with one reqwest client feature | Selected | Proven end to end in the synchronous process model. |
+| Async HTTP client/batch runtime | Rejected | st2 has no tokio reactor and the prototype panicked at export time. |
+| Default or dual reqwest client features | Rejected | Compilation succeeds but exporter construction fails with `NoHttpClient`. |
+| gRPC | Rejected | The fleet ingestion contract is OTLP/HTTP JSON only. |
+
 ## Decision
 
 st2 uses `opentelemetry-otlp` 0.30 with `default-features = false` and exactly