feat(otel): add semantic reconciliation trace hierarchy

agent-identity: unknown
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: OMP
agent-tool-version: 18.0.3
agent-runtime: OMP 18.0.3
tooling-profile: dotfiles@e4789b0

Author: schickling-assistant

docs/vrs/06-observability/.decisions/0001-all-signals-via-pr-stack.md

@@ -1,6 +1,6 @@
 # All three signals via a three-PR gh stack
 
-Status: draft
+Status: accepted
 
 Recorded 2026-08-25 from the aligned observability interview (axe decision catalog Q1 + Q3).
 
@@ -17,6 +17,12 @@ is shared by every signal; once it lands, metrics and logs are incremental. Defe
 invites "traces shipped, rest never happens" — and the CI proof obligation (otelite capture +
 assertions) is signal-generic anyway.
 
+## Evidence and Argument
+
+The exporter/provider plumbing and otelite capture harness are shared across all three signals,
+while rates/durations require metrics and correlated diagnostics require logs. Splitting delivery
+at signal boundaries isolates review and CI failures without treating traces as the finished scope.
+
 ## Decision
 
 **Q1 — Scope**: all three signals (traces, metrics, logs) are the target. Not traces-only; not
@@ -31,18 +37,14 @@ traces-first-with-maybe-later.
 Each PR lands green independently; 2 and 3 build on 1's shared plumbing
 (provider/resource/exporter) only.
 
-## Alternatives considered
-
-- **Traces-first, defer the rest** — rejected: defers most of the value (rates/durations live in
-  metrics; correlated diagnostics in logs) for no real risk reduction, since the risky part
-  (feature set, sync export, flush-at-exit) is identical across signals.
-- **Single PR** — rejected: couples an unreviewable diff (instrumentation across `run.rs`,
-  `exec_backend.rs`, `hooks.rs`, plus unit changes and test harness) to the plumbing; a regression
-  in any slice blocks all of it.
-- **Spool-files instead of direct OTLP export** — writing telemetry records to local spool files
-  for a separate shipper to forward — rejected: adds a moving part st2 must own (rotation,
-  retention, crash-safety) to solve a problem the fleet pipeline already solves at
-  `127.0.0.1:4318`; the ambient-endpoint no-op contract would need re-inventing.
+## Options
+
+| Option | Result | Reason |
+| --- | --- | --- |
+| All three signals in a three-PR stack | Selected | Shares the risky plumbing while isolating signal-specific review and CI failures. |
+| Traces first, defer the rest | Rejected | Defers rates, durations, and correlated diagnostics without reducing exporter risk. |
+| Single PR | Rejected | Couples all instrumentation sites and the capture harness into one blocking review surface. |
+| Spool files instead of direct OTLP | Rejected | Adds rotation, retention, and crash-safety machinery for a pipeline the fleet already provides. |
 
 ## Consequences
 

docs/vrs/06-observability/.decisions/0002-subsystem-placement-and-central-reference.md

@@ -1,6 +1,6 @@
 # Numbered subsystem under st2's VRS root, referencing dotfiles' central tree
 
-Status: draft
+Status: accepted
 
 Recorded 2026-08-25 from the aligned observability interview (axe decision catalog Q2).
 
@@ -19,6 +19,20 @@ central tree (`01-conventions`, `09-integration/spec.md`). But st2-specific deci
 feature set, trace roots in `src/run.rs`, unit env propagation — have no home there and would be
 invisible to anyone working in this repo.
 
+## Evidence and Argument
+
+The repository already uses numbered VRS subsystems for product-specific lifecycle contracts,
+while the dotfiles observability tree owns fleet-wide semantics. A local subsystem with explicit
+central references preserves both contributor locality and one authority per shared rule.
+
+## Options
+
+| Option | Result | Reason |
+| --- | --- | --- |
+| Numbered st2 subsystem | Selected | Keeps local mechanisms and evidence beside their code while referencing central rules. |
+| Standalone st2 documentation root | Rejected | Duplicates the established VRS hierarchy. |
+| Central dotfiles tree only | Rejected | Leaves st2-specific crate, process, and CI decisions without repository-local authority. |
+
 ## Decision
 
 This tree is a **numbered subsystem under st2's existing `docs/vrs` root**: `06-observability`

docs/vrs/06-observability/.decisions/0003-blocking-http-json-exporter.md

@@ -1,6 +1,6 @@
 # Blocking HTTP-JSON exporter, single-client feature set, explicit flush at exit
 
-Status: draft
+Status: accepted
 
 Recorded 2026-08-25 from the aligned observability interview, backed by prototype evidence
 ([../.experiments/2026-08-25-rust-to-otelite-capture.md](../.experiments/2026-08-25-rust-to-otelite-capture.md)).
@@ -24,6 +24,22 @@ Two traps surfaced during prototyping:
 2. **The async-batch trap.** The default async batch exporter requires a tokio runtime; under
    st2's sync process model it panicked at export time.
 
+## Evidence and Argument
+
+The linked otelite prototype exported real OTLP/HTTP JSON from a synchronous Rust process only
+with the blocking client and an exclusive reqwest-client feature selection. It reproduced both
+the dual-client `NoHttpClient` failure and the async exporter's missing-reactor panic, and proved
+that explicit shutdown delivers short-lived spans.
+
+## Options
+
+| Option | Result | Reason |
+| --- | --- | --- |
+| Blocking HTTP-JSON with one reqwest client feature | Selected | Proven end to end in the synchronous process model. |
+| Async HTTP client/batch runtime | Rejected | st2 has no tokio reactor and the prototype panicked at export time. |
+| Default or dual reqwest client features | Rejected | Compilation succeeds but exporter construction fails with `NoHttpClient`. |
+| gRPC | Rejected | The fleet ingestion contract is OTLP/HTTP JSON only. |
+
 ## Decision
 
 st2 uses `opentelemetry-otlp` 0.30 with `default-features = false` and exactly

docs/vrs/06-observability/.decisions/0004-semantic-reconcile-trace-hierarchy.md

@@ -0,0 +1,82 @@
+# Semantic reconcile trace hierarchy
+
+Status: accepted
+
+Recorded 2026-08-26 from the PR3 otelite hierarchy proof
+([../.experiments/2026-08-26-otelite-reconcile-hierarchy.md](../.experiments/2026-08-26-otelite-reconcile-hierarchy.md)).
+
+## Context
+
+PR3's tracing-facade migration exported exactly one `st2.reconcile_pass` span. The span correlated
+logs and carried final pass counts, but the trace remained flat: a slow or failing pass could not
+show whether catalog locking, desired-state discovery, hook verification, materialization, runtime
+observation, or runner mutation owned the time or failure. Function-level instrumentation would
+make that question worse by exposing planning and wrapper structure rather than reconciliation
+semantics.
+
+The hierarchy must preserve the existing root and control flow, remain bounded per pass, obey the
+central non-empty `span.label` and cardinality contract, and construct no child telemetry when the
+tracer exporter is absent even though the stderr tracing subscriber is always installed.
+
+## Evidence and Argument
+
+The linked otelite capture proves the flat baseline can become one root plus five direct,
+same-trace children on an empty catalog without changing the existing metric/log assertions.
+Reading the reconciliation lifecycle identifies exactly six external operation boundaries; all
+other candidate seams are pure planning, bookkeeping, wrappers, or unbounded per-task detail.
+
+## Decision
+
+Keep `st2.reconcile_pass` as the compatibility root and add only aggregate direct children at real
+I/O or mutation boundaries:
+
+- `st2.catalog.lock`
+- `st2.catalog.discover`
+- `st2.hooks.verify`, only when verification is required
+- `st2.catalog.materialize`
+- `st2.runtime.observe`, only for an authoritative in-pass `Runner::list_sessions`
+- `st2.reconcile.execute`
+
+Every span has a bounded, public `span.label`. Attributes contain bounded enums and counts only;
+ids, paths, selectors, and error prose are excluded. A child gets OTel `ERROR` status when its
+boundary fails or reports errors; the root gets `ERROR` when the pass fails. All child constructors
+early-return behind the tracer-export-enabled `AtomicBool`, not `tracing::enabled!`. Children are
+always direct root children, never nested by incidental call structure.
+
+## Options
+
+| Option | Result | Reason |
+| --- | --- | --- |
+| Bounded semantic aggregate children | Selected | Locates lifecycle latency/failure while keeping one fixed detail budget. |
+| Flat root only | Rejected | Cannot locate latency or failure within a reconcile pass. |
+| Function-mirroring spans | Rejected | Exposes implementation structure rather than external work. |
+| Unbounded per-task or per-owner spans | Rejected | Volume and cardinality scale with catalog and mutation size. |
+
+
+### Flat root only
+
+Rejected. It preserves minimum volume but cannot locate latency or failure within a reconcile
+pass; the trace adds little beyond the correlated completion log.
+
+### Function-mirroring spans
+
+Rejected. Spans for validation, planning, compilation, debounce, wrappers, watcher callbacks,
+and report absorption would expose implementation structure, inflate volume, and make refactors
+look like telemetry contract changes without representing additional external work.
+
+### Unbounded per-task or per-owner spans
+
+Rejected for this hierarchy. They may diagnose individual runner operations but cardinality and
+volume scale with catalog size and mutation count. Such detail requires a separately specified hard
+budget and privacy/cardinality review before adoption.
+
+## Consequences
+
+- One pass exports at most the fixed aggregate set; hook verification and external-snapshot
+  observation are truthfully omitted when not performed.
+- Grafana can attribute pass duration and error status to lifecycle boundaries without querying
+  error strings.
+- The empty-catalog otelite test proves one root, five direct children, shared trace identity,
+  exact parent ids, and non-empty labels.
+- Detail below aggregate execution remains unavailable by design; a future per-task proposal must
+  define and prove its own bound rather than extending this decision implicitly.

docs/vrs/06-observability/.experiments/2026-08-26-otelite-reconcile-hierarchy.md

@@ -0,0 +1,63 @@
+# Otelite reconciliation hierarchy proof
+
+2026-08-26. End-to-end evidence for the bounded semantic hierarchy in
+[decision 0004](../.decisions/0004-semantic-reconcile-trace-hierarchy.md).
+
+## Question
+
+One real `st2 up --catalog <empty> --once` invocation can export a non-flat reconcile trace over
+OTLP/HTTP JSON while retaining exactly one compatibility root, one metric set, and one correlated
+completion log. Every semantic child should be a direct root child with the same trace id and a
+non-empty bounded `span.label`.
+
+## Method
+
+Ran the repository integration test against the flake-provided real `otelite` receiver:
+
+```console
+nix develop -c cargo test --test otel_export -- --nocapture
+```
+
+The test launches the real `st2` binary through `otelite run --protocol http/json`, supplies an
+empty catalog, decodes `traces.ndjson`, `metrics.ndjson`, and `logs.ndjson`, and asserts ids and
+attributes from the raw OTLP export request rather than an in-memory tracing subscriber.
+
+## Result
+
+The command passed both integration tests (`2 passed; 0 failed; 0 ignored`). The exported trace
+for the endpoint-enabled invocation contained exactly six spans:
+
+```text
+st2.reconcile_pass             span.label=catalog
+├── st2.catalog.lock           span.label=shared
+├── st2.catalog.discover       span.label=catalog
+├── st2.catalog.materialize    span.label=catalog
+├── st2.runtime.observe        span.label=all sessions
+└── st2.reconcile.execute      span.label=apply plan
+```
+
+The decoded/raw assertions proved:
+
+- exactly one `st2.reconcile_pass` root;
+- every listed child occurred exactly once;
+- every child's `parentSpanId` equaled the root `spanId`;
+- every child's `traceId` equaled the root `traceId`;
+- every root and child had a non-empty `span.label`;
+- `st2.hooks.verify` was absent because an empty catalog has no hook consumer;
+- the pre-existing one reconcile counter point, one reconcile duration sample, one completion log
+  correlated to the root trace/span ids, and no `BatchLogProcessor.Emit.AfterShutdown` assertions
+  remained green;
+- the endpoint-unset invocation completed and left traces, metrics, and logs absent or empty.
+
+## Conclusion
+
+The aggregate semantic hierarchy is visible in real OTLP output, direct parentage is stable, and
+conditional omissions represent operations that did not occur. The same test is the durable CI
+reproduction required by O11Y-R03.
+
+## VRS Impact
+
+The hierarchy, labels, attributes/status rules, path applicability, rejected spans, and disabled
+construction guard are normative in [the specification](../spec.md#reconciliation-trace-hierarchy).
+No requirement text changes: the evidence strengthens O11Y-R01, O11Y-R02, and O11Y-R03 without
+changing their scope.

docs/vrs/06-observability/spec.md

@@ -48,7 +48,7 @@ One module, `src/telemetry.rs`, owns init and teardown via `Telemetry::init(unit
 - **Endpoint**: none configured in code. The exporter resolves `OTEL_EXPORTER_OTLP_ENDPOINT` and
   related `OTEL_*` variables from the environment automatically. Unset → no SDK provider or
   exporter is built (R02); since PR3 the human-readable stderr `tracing` layer still runs so
-  diagnostics stay visible — see [Log bridge](#log-bridge-pr3).
+  diagnostics stay visible — see [Log bridge](#log-bridge-pr3-landed).
 - **Protocol**: HTTP JSON (`http-json` + protobuf-free wire), batch exporter, targeting the local
   Alloy forwarder at `127.0.0.1:4318` by convention.
 - **Resource**: `service.name` = `st2-<unit>` selected per entrypoint (below; `src/main.rs`
@@ -69,28 +69,62 @@ One module, `src/telemetry.rs`, owns init and teardown via `Telemetry::init(unit
 | One-shot CLI invocations | `st2-cli` |
 | Hook executions (`st2 driver claude-observe`; other hook surfaces not instrumented yet) | `st2-hook` |
 
-## Trace roots
+## Reconciliation trace hierarchy
 
-Instrumented via the `tracing` facade since PR3 (`tracing::info_span!`, not manual SDK tracer
-calls), one root span per unit of work:
+Instrumented exclusively through the `tracing` facade (`tracing::info_span!` plus
+`tracing-opentelemetry` status extension), one bounded trace represents one reconcile pass:
 
-- **Supervisor loop pass** — each iteration of the `up_loop_until` loop (`src/run.rs`,
-  `up_loop_until`) wraps one reconcile pass in a span named `st2.reconcile_pass` with field
-  `st2.host`; after the pass it records `st2.crash_loops` and `st2.unparked` counts.
-- **One-shot up** — `up_once` (`src/run.rs`, `up_once`) wraps its single pass in the same
-  `st2.reconcile_pass` shape.
-- **Single-file spec path** — `reconcile_pass_specs_with_sessions` emits the same root span.
-
-A deterministic INFO event (target `st2`, message `reconcile pass complete`, `result` =
-`pass`\|`fail`) closes every pass so log-based assertions need no fault injection.
-
-Not yet instrumented (follow-ups, not PR1 scope):
-
-- Provider session lifecycles (claude / codex / opencode spawn, attach, teardown) beyond the
-  PR2 launch/reap counters, exec sidecars (`src/exec_backend.rs`).
+```
+st2.reconcile_pass
+├── st2.catalog.lock
+├── st2.catalog.discover
+├── st2.hooks.verify                 # only when a consumer requires hooks
+├── st2.catalog.materialize
+├── st2.runtime.observe              # omitted for an externally supplied snapshot
+└── st2.reconcile.execute
+```
 
-Span names follow the central `01-conventions` rules (`span.label` discipline included). Names are
-registered st2-side; this list plus PR2's metric set is that registry's seed.
+`st2.reconcile_pass` remains the compatibility root at the supervisor-loop, one-shot catalog,
+selected-task, and single-file-spec sites. Its `span.label` and `st2.reconcile.path` are the enum
+`catalog | selected | spec`. Root outcome attributes are `st2.host`, `st2.crash_loops`,
+`st2.unparked`, `st2.report.errors`, `st2.report.warnings`, `st2.reconcile.skipped`, and
+`st2.result = pass | fail`; non-empty errors set OTel status `ERROR`. A deterministic INFO event
+(target `st2`, message `reconcile pass complete`, `result = pass | fail`) closes every root so
+log-based assertions need no fault injection.
+
+| Span name | Parent | `span.label` | Operation boundary | Attributes and status | Path applicability |
+| --- | --- | --- | --- | --- | --- |
+| `st2.reconcile_pass` | none | `catalog` \| `selected` \| `spec` | One complete reconcile pass | Root attributes above; `ERROR` when the pass returns/collects an error | Catalog loop/once, selected task, spec loop/once |
+| `st2.catalog.lock` | `st2.reconcile_pass` | `shared` | Shared catalog-authoring lock acquisition | `st2.result`; `ERROR` on acquisition failure | Catalog, selected |
+| `st2.catalog.discover` | `st2.reconcile_pass` | `catalog` | Recursive desired-state snapshot | `st2.catalog.spec_count`, report warning/error counts, `st2.result`; `ERROR` when discovery reports errors even though the pass may continue | Catalog, selected |
+| `st2.hooks.verify` | `st2.reconcile_pass` | `lifecycle hooks` | Required lifecycle-hook receipt/set verification | `st2.hooks.consumer = codex \| pi \| codex+pi`, `st2.result`; `ERROR` on verification failure | Catalog or selected, only when required |
+| `st2.catalog.materialize` | `st2.reconcile_pass` | `catalog` \| `selected owner` | Aggregate catalog/selected-owner materialization call | Materialization failure and report warning/error counts, `st2.result`; `ERROR` when materialization reports errors | Catalog, selected |
+| `st2.runtime.observe` | `st2.reconcile_pass` | `all sessions` | Authoritative `Runner::list_sessions` call | `st2.runtime.session_count`, `st2.result`; `ERROR` on list failure | Catalog, selected, spec; omitted by `_with_sessions` because that snapshot is external |
+| `st2.reconcile.execute` | `st2.reconcile_pass` | `apply plan` | Aggregate mutation call around `execute_with_presentation_cursor` | Plan launch/GC/teardown counts, newly added report warning/error counts, `st2.result`; `ERROR` only when execution adds errors | Catalog, selected, spec |
+
+Every first-party root and child has a non-empty `span.label`. The exporter-enabled
+`AtomicBool` in `src/telemetry.rs` is the hierarchy gate; `tracing::enabled!` is insufficient
+because the stderr formatter remains installed without an endpoint. When the tracer exporter is
+unset, child constructors return before span construction, label handling, collection allocation,
+or count inspection. All children are aggregates and trace volume is bounded by the table.
+
+Attribute policy follows the central `01-conventions` contract:
+
+| Attribute family | Value type | Cardinality | Privacy | Metric-label policy |
+| --- | --- | --- | --- | --- |
+| `span.label` | enum string | bounded | public | forbidden |
+| `st2.reconcile.path`, `st2.result`, `st2.hooks.consumer` | enum string | tiny/bounded | public | spanmetrics-only |
+| All `*_count`, `st2.crash_loops`, `st2.unparked`, `st2.report.errors`, `st2.report.warnings` | integer | bounded numeric | public | forbidden |
+| `st2.reconcile.skipped` | boolean | tiny | public | spanmetrics-only |
+| `st2.host` | string | bounded fleet identity | internal | forbidden |
+
+No span or status description carries an id, filesystem path, selector, or error prose.
+
+Explicitly rejected spans: pure reconcile planning, identity validation,
+`compile_generated_tasks`, debounce, report absorption, wait/sleep, watcher callbacks, and
+wrapper functions. Per-task and per-owner spans are also rejected from this hierarchy: they need
+a separately specified hard detail budget. Provider-session lifecycles and exec sidecars remain
+follow-up surfaces beyond the PR2 launch/reap counters.
 
 ## Metrics (PR2)