feat(health): add skip_initial_history flag to periodic log collector. False by default#3694
Merged
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (4)
Summary by CodeRabbit
WalkthroughAdds ChangesPeriodic log history handling
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant PeriodicLogConfig
participant LogsCollector
participant LogService
participant PersistentState
PeriodicLogConfig->>LogsCollector: provide skip_initial_history
LogsCollector->>LogService: fetch entries
LogsCollector->>PersistentState: persist highest parseable entry ID
LogsCollector-->>LogService: emit no historical records on first poll
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/health/src/collectors/logs/periodic.rs`:
- Around line 331-364: Update the skip_initial_history initialization flow
around last_seen_ids so services are durably marked initialized even when
all_entries is empty or contains no parseable numeric IDs; distinguish this
marker from a missing last_seen_id so subsequent entries are collected rather
than discarded as initial history. Add table-driven coverage for empty,
non-parseable, and numeric initial histories, preserving the existing numeric
anchor behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: bcb88826-4d76-49fc-8281-f0300655f4db
📒 Files selected for processing (4)
crates/health/example/config.example.tomlcrates/health/src/collectors/logs/periodic.rscrates/health/src/config.rscrates/health/src/discovery/spawn.rs
|
🌿 Preview your docs: https://nvidia-preview-pull-request-3694.docs.buildwithfern.com/infra-controller |
When `forge-hw-health` runs in periodic log collection mode, the first poll against a LogService with no saved cursor drains the full entry history. For clusters that have accumulated thousands of historical entries — including past XID faults or CPER records — this creates noise on startup and makes it hard to distinguish pre-existing events from new real-time faults. Add `skip_initial_history` to `collectors.logs.periodic` (and the auto-mode fallback config). When `true`, the collector fetches all current entries on the first encounter of an unseen LogService, records the highest entry ID as the cursor, and discards the entries without emitting them. Subsequent polls collect only new entries from that point forward, matching the real-time-only behaviour of SSE mode. For services that are empty or have no parseable numeric entry IDs on first encounter, the collector inserts a sentinel value of -1. This ensures the service is marked as seen so subsequent polls correctly collect new entries rather than re-entering the initial-history path. Real Redfish entry IDs are non-negative, so -1 is safe and passes all new entries through. Closes NVIDIA#3636 ## Related issues Closes NVIDIA#3636 ## Type of Change - [x] **Add** - New feature or capability - [ ] **Change** - Changes in existing functionality - [ ] **Fix** - Bug fixes - [ ] **Remove** - Removed features or deprecated functionality - [ ] **Internal** - Internal changes (refactoring, tests, docs, etc.) ## Breaking Changes - [ ] **This PR contains breaking changes** ## Testing - [x] Unit tests added/updated - [ ] Integration tests added/updated - [x] Manual testing performed - [ ] No testing required (docs, internal refactor, etc.) ## Additional Notes Validated on `vr-nvl72-ts2-l11-032-c01` (10.71.23.88). Journal services suppressed via `exclude_services = ["Journal"]` (19 active services, 2 excluded). - **skip_initial_history=false (baseline)**: collector drained full log history on first poll, emitting historical XID events and producing PHRs. - **skip_initial_history=true**: collector logged `skip_initial_history: anchored at current log position` for all 19 services, emitting zero historical log events. Empty services (e.g. EventLog, FDR) received `anchor_id=-1` sentinel and were correctly handled on the next poll. Signed-off-by: Dasa Chandramouli <dchandramoul@nvidia.com>
yoks
approved these changes
Jul 20, 2026
mkoci
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When
forge-hw-healthruns in periodic log collection mode, the first poll against a LogService with no saved cursor drains the full entry history. For clusters that have accumulated thousands of historical entries including past XID faults or CPER records, this creates noise on startup and makes it hard to distinguish pre-existing events from new real-time faults.Add
skip_initial_historytocollectors.logs.periodic(and the auto-mode fallback config). Whentrue, the collector fetches all current entries on the first encounter of an unseen LogService, records the highest entry ID as the cursor, and discards the entries without emitting them. Subsequent polls collect only new entries from that point forward, matching the real-time-only behavior of SSE mode.skip_initial_historyis set to false by default and hence there is no change in behaviorRelated issues
Closes #3636
Type of Change
Breaking Changes
Testing
Additional Notes
Validated against a BMC with 4 GPU XID log services containing historical XID 149 entries:
skip_initial_history: anchored at current log positionfor all 4 services, emitting zero historical log events.