feat(otel): first-class OpenTelemetry export — traces, otelite CI gate, o11y VRS tree

[schickling-assistant]

What

First-class OpenTelemetry support for st2 — PR 1 of the agreed all-signals stack.

• src/telemetry.rs — OTLP/HTTP JSON exporter behind OTEL_EXPORTER_OTLP_ENDPOINT, hard no-op when unset; service.name=st2-{supervisor,cli}, service.version from the build stamp, host.name.
• src/run.rs — st2.reconcile_pass root span per supervisor pass and up_once, attrs st2.host, st2.crash_loops, st2.unparked.
• src/service.rs — captured ambient OTEL_* env serialized into systemd-user unit Environment= lines (injectable + unit-tested).
• tests/otel_export.rs + new checks.otel-export flake gate — end-to-end proof: real st2 up --once exports spans into an otelite receiver (effect-utils flake package); gate is non-vacuous in CI.
• docs/vrs/06-observability/ — requirements/spec/decisions/experiment evidence for the o11y subsystem (axe vrs check --profile strict: 0 findings).

Notes for reviewers

• Crate pin: opentelemetry-otlp = { default-features = false, features = ["http-json", "reqwest-blocking-client", "internal-logs"] }. Enabling both reqwest client features compiles but fails at runtime with NoHttpClient; the blocking client is required because st2 has no tokio reactor.
• New flake input effect-utils (github:overengineeringstudio/effect-utils/main) brings the otelite package output; no follows, mirroring how pty stays self-contained.

Verification

• nix build .#checks.x86_64-linux.otel-export passes (~110s) in the sandbox: real binary → real OTLP export → otelite assertions.
• Full workspace suite green except agent_publish/eval_up/doctor failures that reproduce identically on clean HEAD (pre-existing, environment-dependent).
• axe vrs check --profile strict .: 0 findings under docs/vrs/06-observability/**.

Follow-ups (tracked in docs/vrs/06-observability)

• PR 2: metrics. PR 3: log bridge (per decision 0001).
• st2 up <spec> path not yet instrumented; remaining provenance resource attrs deferred.
• Central-contract obligations (telemetry.contract.ts registry entry, dashboards, coverage census) are dotfiles-side follow-up.

Posted on behalf of @schickling

field	value
agent_identity	unknown
agent_persona	generalist
agent_supervisor	unavailable
agent_tool	OMP
agent_tool_version	18.0.3
agent_runtime	OMP 18.0.3
tooling_profile	dotfiles@929dc21

[schickling-assistant]

@codex Please review — PR 1 of the otel stack: OTLP/HTTP-JSON trace export behind OTEL_EXPORTER_OTLP_ENDPOINT (hard no-op when unset), st2.reconcile_pass spans, systemd OTEL_* env propagation, new checks.otel-export flake gate (otelite receiver, non-vacuous), and the docs/vrs/06-observability VRS tree.

Fleet e2e already proven: real traces queryable in Grafana via gcx (trace 53b21f66c5f00de44e88c745cdb6d781, full Alloy→Tempo→Grafana path).

Also includes a CI timeout bump 30m→90m (nix.yml) — the check set grew with the otel gate.

Posted on behalf of @schickling

field	value
agent_identity	unknown
agent_persona	generalist
agent_supervisor	unavailable
agent_tool	OMP
agent_tool_version	18.0.3
agent_runtime	OMP 18.0.3
tooling_profile	dotfiles@929dc21

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 93c5bf09c5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".