Skip to content

feat(dc_measurements): add the fault Measurement, raise/change/clear from diagnostics - #374

Merged
Minipada merged 3 commits into
jazzyfrom
feature/365-add-measurement-fault-raise-and-clear-ev
Aug 18, 2026
Merged

feat(dc_measurements): add the fault Measurement, raise/change/clear from diagnostics#374
Minipada merged 3 commits into
jazzyfrom
feature/365-add-measurement-fault-raise-and-clear-ev

Conversation

@Minipada

@Minipada Minipada commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds a fault Measurement subscribing to /diagnostics that emits a Record only when a component's DiagnosticStatus level changes -- diagnostics records periodic snapshots, so recovering "when did this break, when was it fixed" today means scanning for where a value changed in SQL, silently producing a wrong duration the moment one Record is dropped.
  • One dc_common::StateTransitionDetector<uint8_t> per watched component (the same building block as intervention, Add Measurement: intervention — human takeovers derived from driving-mode transitions #362), classifying each transition as raise (left OK), change (between two non-OK levels, e.g. WARNERROR), or clear (back to OK, carries duration_s). Both the raise and the clear share fault_started_at, so the pair joins downstream for MTTR; MTBF is the gap between successive raises.
  • Levels are reported by name (OK/WARN/ERROR/STALE) so STALE (silent) is never mistaken for ERROR (broken). A fault still open at shutdown keeps its last Record's state: "open" and no duration_s -- no closing Record is invented, so it can't be read as a zero-length outage.
  • seq increments across every watched component (one global stream), so a dropped Record is a detectable gap. names (default: watch everything seen) limits which components are tracked; a pending-Records queue caps at 64, mirroring battery.cpp's charging-session flood guard.
  • JSON schema, gtest coverage (a raise, a clear, a change that is not a clear, a flapping component, a fault open at shutdown), and docs (doc/src/dc/measurements/fault.md, SUMMARY.md, plugin table).

Test plan

  • colcon build + colcon test for dc_measurements (running locally via tools/e2e/scripts/build.sh in parallel with this PR's CI build; will report back if either turns up something the other didn't)
  • prek run --all-files --skip build-doc: all 24 hooks pass

Closes #365

🤖 Generated with Claude Code

https://claude.ai/code/session_0144yNQyXN9xNdGTuj3K8m23

@codecov

codecov Bot commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.23529% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.09%. Comparing base (8447a0c) to head (38b697e).

Files with missing lines Patch % Lines
dc_measurements/plugins/measurements/fault.cpp 88.24% 10 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##            jazzy     #374      +/-   ##
==========================================
+ Coverage   67.82%   68.09%   +0.28%     
==========================================
  Files         101      102       +1     
  Lines        6235     6320      +85     
==========================================
+ Hits         4228     4303      +75     
- Misses       2007     2017      +10     
Flag Coverage Δ
cpp-jazzy 68.09% <88.24%> (+0.28%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Minipada
Minipada force-pushed the feature/365-add-measurement-fault-raise-and-clear-ev branch from 595569d to 04b3a97 Compare August 18, 2026 14:13
Minipada and others added 3 commits August 18, 2026 16:47
…from diagnostics

MTBF and MTTR need Records that say when something broke and when it was
fixed. diagnostics records snapshots -- every poll writes the current
DiagnosticStatus level -- so recovering a transition means scanning for
where a value changed in SQL, silently producing a wrong duration the
moment one Record is dropped. fault subscribes to the same /diagnostics
topic and emits a Record only when a component's level changes, following
intervention's precedent (#362): a separate Measurement projecting the
same signal rather than a mode on it.

Built on #360's StateTransitionDetector, one instance per watched
component, so components fault and recover independently. Each transition
is classified as raise (left OK, opens a fault), change (between two
non-OK levels, e.g. WARN to ERROR, stays open) or clear (back to OK,
closes the fault and reports duration_s). Both the raise and the clear
carry the same fault_started_at so the pair joins downstream for MTTR;
MTBF is the gap between successive raises. Levels are reported by name
so STALE is never mistaken for ERROR. A fault still open at shutdown
keeps its last Record's state "open" and no duration_s -- no closing
Record is invented, so it cannot be read as a zero-length outage. seq
increments across every watched component, so a dropped Record is a
detectable gap. names (default: everything seen) limits which components
are tracked; the pending-Records queue caps at 64 like battery's charging
events, since the DiagnosticArray callback and the polling timer run on
different callback groups.

Verified with 5 gtest cases: a raise, a clear, a change that is not a
clear, a flapping component, and a fault open at shutdown. Documented in
doc/src/dc/measurements/fault.md, listed in SUMMARY.md and the plugin
table.

Closes #365

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0144yNQyXN9xNdGTuj3K8m23
Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0144yNQyXN9xNdGTuj3K8m23
Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
… transition in tests

CI failed 4 of the 5 test_measurement_fault gtest cases: StateTransitionDetector
treats a component's very first observed sample as a baseline, not a transition
-- no Record comes out of it, matching every other consumer of the detector
(battery, and the not-yet-merged intervention). Four of the five fault tests
published their "raising" status as the very first sample ever seen for that
component, so the plugin correctly produced no Record and every publishUntilRecord
call past that point timed out waiting for one. Only the first test happened to
publish OK first (to also assert "staying put produces nothing"), which incidentally
established the baseline its own raise then relied on.

Adds establishOkBaseline() and calls it before the first real transition in the
other four tests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0144yNQyXN9xNdGTuj3K8m23
Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
@Minipada
Minipada force-pushed the feature/365-add-measurement-fault-raise-and-clear-ev branch from 04b3a97 to 38b697e Compare August 18, 2026 14:47
@Minipada
Minipada merged commit c1391db into jazzy Aug 18, 2026
10 checks passed
@Minipada
Minipada deleted the feature/365-add-measurement-fault-raise-and-clear-ev branch September 2, 2026 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant