feat(otel): RED-minimal metrics + spec-path spans - #348
Merged
schickling-assistant merged 6 commits intoAug 26, 2026
Merged
Conversation
schickling-assistant
marked this pull request as ready for review
August 26, 2026 09:19
Contributor
Author
|
@codex Please review — PR 2 of the otel stack (base: #347): RED-minimal metrics per design decision Q5. Counters: Labels bounded to enums only; ids stay in span attributes per the central cardinality policy. Also instruments the spec-up path with Posted on behalf of @schickling
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 766137accc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
schickling-assistant
force-pushed
the
schickling/2026-08-26-otel-metrics
branch
5 times, most recently
from
August 26, 2026 16:53
fa2b896 to
d784fba
Compare
Meter provider alongside the tracer in Telemetry::init (same endpoint
guard, resource, and OTLP/HTTP-JSON wire); shutdown flushes both.
Instruments per interview decision Q5, labels only from bounded enums:
reconcile_passes_total{result}, task_launches_total{driver},
task_reaps_total{driver}, hook_invocations_total{hook,event},
message_deliveries_total{result}, crash_loops_total,
reconcile_pass_duration_seconds, session_start_duration_seconds.
All record calls early-out unless a provider is installed.
The single-file spec path (reconcile_pass_specs_with_sessions) now emits
the same st2.reconcile_pass root span as the catalog paths. otelite e2e
asserts metric rows land in metrics.ndjson.
agent-identity: unknown
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: OMP
agent-tool-version: 18.0.3
agent-runtime: OMP 18.0.3
tooling-profile: dotfiles@929dc21
… metrics - crash_loops_total: record once per park (inside the reported_flapping first-transition blocks) instead of every reconcile pass a task stays parked; execute no longer records per-pass. - hook metrics: classify `st2 driver claude-observe` as telemetry unit "hook" in main.rs so hook_invocations_total exports under st2-hook per the documented process-unit contract (only claude observe records hook metrics today). Spec doc updated to match. - message_deliveries_total: record result="fail" when the archived-path read_to_string fails instead of returning before any recording. agent-identity: unknown agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.3 agent-runtime: OMP 18.0.3 tooling-profile: dotfiles@929dc21
…vocations Duration histograms recorded as_secs_f64() against the SDK's millisecond-tuned default boundaries, collapsing sub-second reconcile passes and session spawns into the lowest buckets. Register an explicit view mapping both duration instruments onto 1ms..10s explicit buckets. hook_invocations_total fired before agent-dir resolution, counting invocations that never applied (undeclared agent). Record it after the application target resolves. agent-identity: unknown agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.3 agent-runtime: OMP 18.0.3 tooling-profile: dotfiles@929dc21
…uckets agent-identity: unknown agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.3 agent-runtime: OMP 18.0.3 tooling-profile: dotfiles@929dc21
Replace the OTLP JSON substring match with parsed assertions on the reconcile passes counter attribute and the duration histogram's explicitBounds (12 seconds-scale boundaries, lowest 1ms). agent-identity: unknown agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.3 agent-runtime: OMP 18.0.3 tooling-profile: dotfiles@929dc21
agent-identity: unknown agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.3 agent-runtime: OMP 18.0.3 tooling-profile: dotfiles@e4789b0
schickling-assistant
force-pushed
the
schickling/2026-08-26-otel-metrics
branch
from
August 26, 2026 19:17
d784fba to
d314338
Compare
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #347 (PR1 — traces). Part 2 of the observability stack (VRS 06-observability).
What
RED-minimal metric set per interview decision Q5, plus the spec-up span coverage folded into this slice:
Telemetry::initalongside the tracer: sameOTEL_EXPORTER_OTLP_ENDPOINTguard, same resource (st2-<unit>, version, host), OTLP/HTTP-JSONMetricExporter+PeriodicReader. Unset endpoint → no provider; the global meter is a silent no-op and every record call early-outs before touching an instrument (zero-overhead guarantee, R02).Telemetry::shutdownforce-flushes points with spans so short-lived CLI runs deliver them.src/metrics.rs), labels only from bounded enums — ids never become labels:reconcile_passes_totalresult=pass|failtask_launches_totaldriver=codex|claude|opencode|pi|exec|othertask_reaps_totaldriver(same enum)hook_invocations_totalhook(registry name),event(bounded Claude set →other)message_deliveries_totalresult=pass|failcrash_loops_totalreconcile_pass_duration_secondssession_start_duration_secondsreconcile_pass_specs_with_sessionsnow emits the samest2.reconcile_passroot-span shape as the catalog loop pass andup_once.tests/otel_export.rsassertsreconcile_passes_total/reconcile_pass_duration_secondsrows land in otelite'smetrics.ndjson, including theresult=passattribute.Instrumentation stays strictly observational: counters/histograms only, no behavior change to reconcile/launch/reap logic (INVARIANTS).
Verification
nix develop -c cargo buildgreennix develop -c cargo test --lib: 486 passed (incl. new metrics unit tests)ST2_OTELITE_BIN=/tmp/st2-otel-interview/otelite/bin/otelite nix develop -c cargo test --test otel_export: 2 passed (spans + metrics captured end-to-end)Posted on behalf of @schickling
agent_identityagent_personaagent_supervisoragent_toolagent_tool_versionagent_runtimetooling_profile