feat(otel): tracing facade — unified span+log subscriber, diagnostics migration, log CI assertions - #350
Merged
schickling-assistant merged 5 commits intoAug 26, 2026
Conversation
schickling-assistant
marked this pull request as ready for review
August 26, 2026 09:19
Contributor
Author
|
@codex Please review — PR 3 of the otel stack (base: #348): tracing facade per decision Q6. Unified subscriber: stderr fmt layer (always on for visibility), Note for reviewers: Posted on behalf of @schickling
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 757a8291c9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
schickling-assistant
force-pushed
the
schickling/2026-08-26-otel-logs
branch
2 times, most recently
from
August 26, 2026 11:20
308cf0c to
506171d
Compare
schickling-assistant
force-pushed
the
schickling/2026-08-26-otel-logs
branch
from
August 26, 2026 12:01
506171d to
6568b7b
Compare
schickling-assistant
force-pushed
the
schickling/2026-08-26-otel-logs
branch
2 times, most recently
from
August 26, 2026 16:22
5f6363b to
b34aaef
Compare
schickling-assistant
force-pushed
the
schickling/2026-08-26-otel-logs
branch
from
August 26, 2026 16:54
b34aaef to
48cd14b
Compare
… migration, log CI assertions PR3 per interview decision Q6 (docs/vrs/06-observability/open-questions.md): - Telemetry::init now installs one tracing subscriber: a stderr fmt layer runs unconditionally so migrated diagnostics stay visible with or without an endpoint (documented deviation from PR1's zero-output unset case; RUST_LOG overrides the INFO default), plus — behind the endpoint guard — a tracing-opentelemetry span layer and an opentelemetry-appender-tracing bridge over a new SDK logger provider sharing endpoint/protocol/resource. experimental_use_tracing_span_context stamps active span ids onto records. shutdown flushes logger+meter+tracer together. - st2.reconcile_pass spans move to tracing::info_span! at all three sites (up_once, up_loop_until pass, reconcile_pass_specs_with_sessions); names and st2.host/st2.crash_loops/st2.unparked attributes unchanged; metric recording calls untouched. Each pass emits a deterministic INFO record (target st2, "reconcile pass complete", result=pass|fail). - Non-user-facing eprintln! diagnostics migrate to tracing::warn!/error! with unchanged text: crash-loop give-ups and supervisor notification, park-channel setup, catalog watching, ding transport ambiguity, driver session degradation, harness-state floor writes. CLI command output, boot banners, and validation reports stay println!/eprintln!. - tests/otel_export.rs additionally asserts logs.ndjson carries the completion record (body/severity) correlated to the captured st2.reconcile_pass span via trace/span ids; existing span+metric assertions unchanged. agent-identity: unknown agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.3 agent-runtime: OMP 18.0.3 tooling-profile: dotfiles@929dc21
Two review findings on the log-bridge PR: 1. The EnvFilter was attached to the base subscriber, so RUST_LOG=warn silently disabled INFO span/log export even with an endpoint set. The stderr fmt layer now carries the filter alone. The OpenTelemetry layers carry their own static filter, independent of RUST_LOG, that silences only exporter-internal targets (opentelemetry*): the SDK's BatchLogProcessor emits tracing events from inside its own emit(), and feeding those back through the unfiltered log bridge recursed until the export thread overflowed its stack (gdb-proven). App signals export unfiltered. 2. A metric exporter build failure early-returned before logger setup, killing logs along with metrics. Metric setup is now best-effort: warn and continue to the logger provider. agent-identity: unknown agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.3 agent-runtime: OMP 18.0.3 tooling-profile: dotfiles@929dc21
…ws tty tracing_subscriber's fmt layer enables ANSI whenever the 'ansi' feature is on, regardless of destination, so redirected stderr carried SGR escapes where the pre-migration eprintln! path emitted plain bytes. Gate the layer on std::io::stderr().is_terminal(). agent-identity: unknown agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.3 agent-runtime: OMP 18.0.3 tooling-profile: dotfiles@929dc21
…anly agent-identity: unknown agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.3 agent-runtime: OMP 18.0.3 tooling-profile: dotfiles@e4789b0
agent-identity: unknown agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.3 agent-runtime: OMP 18.0.3 tooling-profile: dotfiles@e4789b0
schickling-assistant
force-pushed
the
schickling/2026-08-26-otel-logs
branch
from
August 26, 2026 19:17
5ed906f to
80c7a28
Compare
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
PR3 of the otel stack (#347 → #348 → #350): adopts the
tracingfacade so spans and logs share one subscriber, migrates supervisor-relevant diagnostics onto it, and extends the otelite CI gate to assert log delivery.src/telemetry.rs): a human-readable stderr fmt layer runs unconditionally, and behind theOTEL_EXPORTER_OTLP_ENDPOINTguard atracing-opentelemetrylayer exports spans whileopentelemetry-appender-tracingexports logs via a new SDK logger provider sharing endpoint/protocol/resource with tracer+meter.experimental_use_tracing_span_contextstamps active span ids onto records.Telemetry::shutdownflushes logger+meter+tracer together.st2.reconcile_passsites (up_once,up_loop_untilpass,reconcile_pass_specs_with_sessions) now usetracing::info_span!; name andst2.host/st2.crash_loops/st2.unparkedattributes unchanged; metric calls untouched.st2, messagereconcile pass complete, fieldresult=pass|fail.println!/eprintln!.Documented deviation from PR1's literal zero-output unset-endpoint behavior: the stderr fmt layer stays installed without an endpoint so migrated diagnostics remain visible (level default INFO;
RUST_LOGoverrides).Design
Per interview decision Q6 recorded in
docs/vrs/06-observability/open-questions.md: tracing facade adopted (tracing-opentelemetry+opentelemetry-appender-tracing), journald/dual-emit alternatives declined. Spec updated to landed reality in this PR.Verification
cargo buildgreencargo test --lib: 471 passedcargo test --test otel_export(withST2_OTELITE_BIN): asserts spans (st2.reconcile_pass,st2-cliresource) + metrics (reconcile_passes_total, duration histogram,result=pass) + NEW:logs.ndjsoncarries thereconcile pass completeINFO record with severity number 9 correlated to the captured span via trace/span ids (32/16-hex)Posted on behalf of @schickling
agent_identityagent_personaagent_supervisoragent_toolagent_tool_versionagent_runtimetooling_profile