[Bug]: syslog-health-monitor can miss XIDs when journald rotates before scan

[XRFXLP]

Prerequisites

• I searched existing issues
• I can reproduce this issue

Code of Conduct

• I agree to follow NVSentinel's Code of Conduct

Bug Description

syslog-health-monitor can miss XID events when journald rotates or vacuums entries before the monitor reaches them.
In an observed incident, XIDs were still visible through dmesg and detected by DCGM, but the retained journald range started after the XID timestamps. Since syslog-health-monitor reads /var/log/journal via journald rather than dmesg or DCGM, it had no retained journal entries to parse.

Expected behavior: XID-related syslog checks should process only kernel-origin journal entries so the monitor can catch up quickly and avoid falling behind due to unrelated audit/container log volume.

Proposed fix: add kernel-only journald filtering for syslog checks, for example SYSLOG_FACILITY=0, by default for XID/GPU-fallen/SXID checks.

Component

Health Monitor

Steps to Reproduce

1. Run syslog-health-monitor on a GPU node where it reads from /var/log/journal.
2. Generate enough non-kernel journald traffic to put journald under retention pressure.
3. Trigger or inject an NVIDIA XID kernel log entry.
4. Continue generating journald traffic until the journal segment containing the XID is rotated or vacuumed before syslog-health-monitor processes it.
5. Confirm the XID was produced by checking dmesg -T | grep -i "nvrm: xid".
6. Confirm the XID is no longer available through journald using journalctl | grep -i "nvrm: xid" or by checking that journalctl --list-boots starts after the XID timestamp.
7. Observe that syslog-health-monitor does not emit a corresponding health event.

Idea here is to delay syslog HM log processing completion so logs get rotated before HM could process it.

Environment

• NVSentinel version: v1.9.0
• Kubernetes version:
• Deployment method:

Logs/Output

No response

[jackyliusohu]

I'd like to take this. I verified the gap in code and on a live deployment.

The SYSLOG_FACILITY=0 mechanism already exists (gated on a -k/--dmesg tag in syslogmonitor.go), but it's never wired up for the default checks:
buildChecksFromFlag() builds SysLogsXIDError / SysLogsSXIDError / SysLogsGPUFallenOff with empty Tags. Confirmed on a prod deployment — neither the regular nor
the kata DaemonSet args carry -k/--dmesg.

One refinement on the proposed fix: the kernel filter should default for the non-Kata path only. On Kata nodes XIDs surface via the guest kernel relayed through

[XRFXLP]

hey @jackyliusohu ,

Thanks for taking this up. I've assigned this issue to you.

[jackyliusohu]

Thanks @XRFXLP Plan: default -k (SYSLOG_FACILITY=0) for the three kernel-origin checks in the non-Kata path, hardcoded in buildChecksFromFlag, leaving Kata's
-u containerd.service untouched. I'll add unit tests for both paths. Let me know if you'd rather have it as a per-check Helm option — otherwise I'll go with the
hardcoded default to keep the change focused.