Release v1.15.0

Release v1.15.0

This release completes the operator-triggered cold-start/change-stream reset begun in v1.14.0, adds kubectl get extrr printer columns, and delivers a set of high-impact reliability fixes — most importantly detecting frozen health-monitor loops via the liveness probe, hardening GPU reset job creation and partial drains, and correcting NIC monitor false positives and dropped failure events. It also expands operator documentation across the PostgreSQL/MongoDB stores, fault-quarantine rule sets, and the Kubernetes Object Monitor.

Major New Features

Cold-Start Reset via resume-control ConfigMap (#1476)

Completes the operator-triggered change-stream reset introduced in v1.14.0 (#1472). Previously, setting a module's key to CREATE in the resume-control ConfigMap only reset the change-stream resume token — a subsequent restart could still replay old persisted records because cold-start recovery queries the datastore directly. Modules that support cold-start recovery now record a component-managed <module>.coldStartAfter timestamp when they consume CREATE, and future cold-start queries only consider events created after that timestamp, making the reset durable. Operators still only set the module key to CREATE; the timestamp is internal status managed by the module.

ExternalRemediationRequest Printer Columns (#1495)

kubectl get extrr now shows NODE, CHECKNAME, and COMPLETIONTIME columns (with completion time rendered in a readable date format), so operators can see the state of ExternalRemediationRequest objects at a glance without inspecting each one.

Bug Fixes & Reliability

• Detect frozen health-monitor loops via /healthz (#1477): The gpu-, syslog-, and nic-health-monitor liveness probes targeted /metrics, served by an independent HTTP goroutine, so when a monitor's main polling loop froze the probe kept returning 200 and Kubernetes never restarted the pod. On one 370-GPU-node cluster, 13 gpu-health-monitor pods showed Running 1/1 with zero restarts while their reconciliation loops had been frozen for 7–24 days — those nodes had zero GPU health monitoring despite appearing healthy. A new reusable PollingHealthChecker tracks the last successful poll timestamp and makes /healthz return unhealthy once it exceeds a staleness threshold (3× the poll interval), so a stalled loop now trips the liveness probe and restarts the pod.
• GPUReset job creation failures and invalid partial drains (#1500): Two fixes for GPU reset. (1) Depending on the node name, the reset-job name truncation could produce an invalid RFC 1123 subdomain (e.g. a trailing .- sequence), so the controller failed to create the job indefinitely — and because this phase has no timeout, the CRD never reached a terminal state. Job names are now always valid RFC 1123 subdomains. (2) Full and partial drains skipped pods that were stuck terminating or NotReady, which may still hold running processes with GPU contexts and can cause a GPU reset to fail; partial drains now ensure targeted pods are removed or in a terminal state before the reset proceeds.
• NIC monitor no longer misses or drops failure events (#1492, #1488): Fixes six findings where the NIC health monitor could silently lose or never emit failure events or fabricate fatal disappearance events from transient sysfs read errors. Notably, fatal counter events were relabeled under the state check's name, so an ordinary ACTIVE/LinkUp recovery cleared a counter-originated condition while the breach latch kept suppressing further breaches — the link_downed safety net worked only once per boot; fatal counter events now keep their *DegradationCheck identity so a state recovery can no longer clear them. FATAL/recovery events were also persisted before publish (violating the healthpub caller contract) so a platform-connector outage permanently consumed the health boundary; checks are now transactional (Prepare/Commit).
• NIC monitor requires positive evidence for FATALs; adds inclusion override (#1462, #1361, #1379): A port the monitor had never seen healthy was reported FATAL unless proven expected-down, producing false alarms on ports that are down by design (e.g. unprovisioned Ethernet/RoCE aux ports on some cloud shapes). The rule is inverted: a first-sight DOWN port is now failed only when there is positive peer evidence it should be up (grouping cards by role and flagging those below the untied modal active-port count), otherwise it is suppressed; runtime ACTIVE → DOWN remains always fatal. Adds a nicInclusionRegexOverride to pin monitoring to an explicit NIC list, and fixes restart/config-change state bugs (fabricated "device disappeared" FATALs, lost counter latches, and card-level FATALs that could hold a node quarantined forever with no recovery event).
• Circuit breaker counts only GPU nodes (#1482, #1228): The fault-quarantine circuit breaker computed its thresholds, trip, and progress against all cluster nodes rather than GPU nodes, skewing the denominator on mixed clusters. It now counts only nodes labeled nvidia.com/gpu.present=true, with the label configurable via fault-quarantine.gpuNodeLabel (Helm) / --gpu-node-label (CLI).

Documentation

• PostgreSQL Store configuration guide (#1490): Adds a configuration guide for the PostgreSQL datastore.
• Percona Operator for MongoDB store (#1489): Adds Percona Operator documentation to the MongoDB store guide.
• Managing fault-quarantine rule sets (#1486): Documents how to add, modify, disable, apply, and verify fault-quarantine rule sets.
• Using the Kubernetes Object Monitor (#1483): Adds a guide for configuring and using the Kubernetes Object Monitor.
• Custom remediation plugin tutorial (#1493): Adds a tutorial for writing a custom remediation plugin.

Acknowledgments

This release includes contributions from:

• @jtschelling
• @KaivalyaMDabhadkar
• @natherz97
• @nitz2407
• @rupalis-nv
• @shaq918
• @XRFXLP
• @lalitadithya

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback!

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.15.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.14.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.15.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.14.0

Release v1.14.0

This release adds TTL-based health-event deduplication in the platform connector, re-lands GPU-reset-safe kernel filtering for the syslog health monitor, gives operators a one-shot change-stream resume reset, and garbage-collects stale preflight coordination ConfigMaps. It also lands a cluster of fault-management correctness fixes covering concurrent-event handling, entity matching, cancellation scoping, stuck state labels, and a PostgreSQL change-stream race.

Major New Features

Health Event Deduplication in Platform Connector (#1283)

Implements ADR-039: a TTL-based deduplication filter that runs once inside the platform-connector pipeline, after transformers and before fan-out to the enabled connectors, so duplicate health events are suppressed before they reach either the Kubernetes connector or the store connector. Events are deduplicated by a canonical key within a configurable suppression window; healthy events clear the matching cached unhealthy key so recovery is never dropped, per-check opt-out is supported, and cached keys expire on a cleanup ticker and on restart. This reduces redundant node-condition writes and datastore churn from health monitors that re-emit the same condition on every poll.

Kernel Filtering and GPU-Reset Acknowledgement for Syslog Health Monitor (#1475)

The XID, SXID, and GPU-fallen-off-bus checks now filter journald to kernel-origin entries (SYSLOG_FACILITY=0) so that, under journald retention pressure, unrelated high-volume userspace logs no longer push the read cursor past the retained window and cause XIDs to be missed (originally #1417). The earlier attempt at this in v1.13.0 (#1426) was reverted (#1473) because the kernel-only filter also dropped the GPU reset job's own logger messages, breaking GPU reset acknowledgement. This re-lands the filter with an added journald disjunction that matches the nvsentinel-gpu-reset message tag, so reset acknowledgements are captured alongside kernel XIDs. The Kata path continues filtering by -u containerd.service.

Operator-Triggered Change-Stream Resume Reset (#1472)

Adds a runtime-owned resume-control ConfigMap that lets operators perform a one-shot resume-token reset for watcher-based components. Each component reads its own key on startup: RESUME (the default) preserves normal resume behavior, while CREATE makes the component delete only its own stored resume token and start from the current stream head, then rewrite its key back to RESUME. The ConfigMap is not Helm/GitOps-managed — components self-populate their key as RESUME when it is missing — providing a recovery lever for a component wedged on a bad resume token without disturbing the others.

Garbage Collection of Preflight Coordination ConfigMaps (#1459)

Preflight gang-coordination ConfigMaps now carry an OwnerReference back to their scheduler gang owner object, so Kubernetes garbage-collects them automatically when the owning PodGroup/workload is deleted rather than leaving stale ConfigMaps behind. Owner-reference resolution covers native workloadRef (Kubernetes 1.35), native schedulingGroup/PodGroup (Kubernetes 1.36), and generic-scheduler PodGroup, and the reference is backfilled onto existing skeleton and provisional ConfigMaps without disturbing peer data.

Bug Fixes & Reliability

• Stuck remediation-failed label on partial recovery (#1421, #1416): When a node had multiple active failures and one with an unsupported action (e.g. CONTACT_SUPPORT) set the terminal dgxc.nvidia.com/nvsentinel-state=remediation-failed label, that label could remain stuck after the unsupported check recovered while other failures kept the node quarantined — cleanup only happened on full UnQuarantined/Cancelled. Fault-quarantine now propagates a partial recovery (a healthy event clearing a tracked failure while the node stays quarantined) via the existing AlreadyQuarantined status instead of dropping it, and node-drainer marks it AlreadyDrained so it reaches fault-remediation for label cleanup. Fault-quarantine stays domain-agnostic and does not interpret remediation-action support.
• Platform connector no longer clears unrelated conditions on partial entity match (#1468, #1360): A healthy event could clear an existing unhealthy node condition if the condition message matched any single impacted entity from the healthy event. For composite identities such as NIC + NICPort, mlx5_0/port 1 and mlx5_1/port 1 are different entities that share a port number, so OR-based matching silently cleared unrelated NIC failures. Matching now requires the full composite entity rather than any single component.
• Fault-quarantine preserves concurrent health events (#1464, #1463): When two unhealthy events for the same node arrived within ~90 ms, the informer cache had not yet observed the first annotation update, so both took the "fresh quarantine" path and the second was treated as a no-op — dropping it from quarantineHealthEvent. The live node-update path now merges the incoming event with the node's current annotation using the Kubernetes state fetched for the update rather than trusting informer state, so near-concurrent events are both preserved (the second correctly classified AlreadyQuarantined).
• Node-drainer cancellation scoped by event cutoff (#1461, #1451): cancelledNodes[node] was a temporary presence flag, so while it existed node-drainer cancelled non-UnQuarantined events for that node — including a fresh quarantine that arrived before cleanup of the previous session finished. It is now a cutoff timestamp set to UnQuarantined.CreatedAt, so only events created at or before the cutoff are cancelled and a new quarantine after recovery survives.
• PostgreSQL change-stream race skipping UnQuarantined events (#1469, #1466): In the PostgreSQL changelog-based change-stream emulation, the watcher could apply a partially converted server-side SQL filter, fetch a later changelog row, advance its lastEventID bookmark, and then skip an earlier relevant row when its NOTIFY arrived later — causing node-drainer to miss an UnQuarantined event and leave a node stuck in draining. Server-side filtering now fails closed: the generated SQL filter is used only when the entire pipeline converts successfully, otherwise the watcher fetches changelog rows in order and relies on the application-side filter.

Documentation

• Writing your own preflight check tutorial (#1470): Adds a tutorial walking through building a custom NVSentinel preflight check.
• Preflight guide: OSMO+KAI, Grove, and debugging (#1467): Adds OSMO + KAI Scheduler setup, Grove limitations, and preflight-failure debugging sections to the preflight configuration guide.
• Drain plugin tutorial (#1435): Adds an end-to-end tutorial for building a new NVSentinel drain plugin, from an empty directory to a deployed controller that drains a node on NVSentinel's behalf.

Acknowledgments

This release includes contributions from:

• @lalitadithya
• @nitz2407
• @rupalis-nv
• @XRFXLP

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback!

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.14.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.13.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.14.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.13.1

Release v1.13.1

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel --version v1.13.1

Release v1.13.0

Release v1.13.0

This release broadens DCGM deployment flexibility with three explicit source modes and per-code event suppression in the GPU health monitor, teaches fault-quarantine to coexist with externally-applied cordons and taints, adds runtime per-namespace preflight gang-discovery configuration, and implements the ExternalRemediationRequest reconciler. It also lands several impactful reliability fixes — most notably stopping node-drainer cold-start from replaying stale quarantine events and stranding remediation-failed labels.

Major New Features

DCGM Source Modes for GPU Health Monitor (#1429)

The GPU health monitor now supports three explicit DCGM source modes (see ADR-044): operator-service (the current default — the GPU Operator's nvidia-dcgm service), external-hostengine (a node-local host-installed nv-hostengine, with the DCGM-major image selected from the externally-managed nvsentinel.dgxc.nvidia.com/dcgm.version node label), and embedded (an in-process DCGM hostengine started by the monitor itself with NVIDIA runtime GPU visibility). Helm values and templates were updated for mode-specific endpoint, image, host-networking, and runtime-class selection, and the labeler preserves a valid dcgm.version label in external-hostengine mode when no DCGM pod is present.

Suppressible DCGM Error Codes (#1450)

The GPU health monitor can now be configured to suppress specific DCGM error codes via a SuppressedErrorCodes setting in the [dcgmhealthcheck] config. Non-fatal, non-actionable events (for example DCGM_FR_CLOCK_THROTTLE_POWER) can be dropped before emission rather than persisted, reducing datastore noise and avoiding unintended downstream side effects from high-frequency events.

Preserve Pre-Existing Cordons and Taints in Fault-Quarantine (#1445)

Fault-quarantine now records whether a node's cordon or taints existed before NVSentinel quarantined it, so that unquarantining no longer removes a cordon or taint that some other operator or tool applied. A new quarantineHealthEventCordonPreExisting annotation tracks pre-existing cordons, and tracked taints gain a backwards-compatible PreExisting field; cordon-by / uncordon-by labels are only added when the cordon was not pre-existing. Also fixes a bug where re-quarantining a node removed only the manual-uncordon annotation and not the manual-untaint annotation.

Namespace-Scoped Preflight Gang Discovery (#1436)

Preflight gang discovery can now be configured per namespace via a new namespaced PreflightConfig CRD (preflight.nvsentinel.nvidia.com/v1alpha1). Creating one with a spec.gangDiscovery block makes pods in that namespace use that discoverer, while all other namespaces fall back to the cluster-wide gangDiscovery Helm value. It is reconciled at runtime — no Helm upgrade or controller restart — so a single preflight deployment can serve namespaces running different gang schedulers (e.g. Volcano in one, native Kubernetes or Run:ai/OSMO in another).

ExternalRemediationRequest Reconciler (#1392)

Builds on the ExternalRemediationRequest (ExtRR) CRD foundation from v1.10.0 (#1376) by adding the janitor controller that drives the node-coordination state machine: it applies a release taint (keyed to the ExtRR's own name) plus the managed=false label and reports NVSentinelOwnershipReleased=True, then removes them and drops its finalizer once an external system reports ExternalRemediationComplete=True. Per the ADR-040 contract, ExternalRemediationComplete=False is intentionally asymmetric and does not close the request, and kubectl delete triggers the same node cleanup so operators can reclaim stalled nodes. This remains preview: nothing creates ExtRR objects automatically yet (the fault-remediation wiring is a follow-up), so today the controller only acts on hand-applied objects.

Bug Fixes & Reliability

• Node-drainer cold-start no longer replays stale quarantine events (#1443, #1347): On restart, node-drainer's cold-start logic re-queried the datastore for quarantine events still needing a drain but never checked whether the quarantine session had already ended, so it replayed stale Quarantined/AlreadyQuarantined records from sessions fault-quarantine had already resolved. It marked them drain-succeeded, and fault-remediation then stamped dgxc.nvidia.com/nvsentinel-state=remediation-failed onto already-healthy, uncordoned nodes for unsupported actions like CONTACT_SUPPORT — with no cleanup path, the label persisted forever, and a single restart could do this to many nodes at once. Cold start now verifies the quarantine session is still active before re-queuing a candidate.
• Fixed node conditions wedged by non-canonical recovery messages (#1438): platform-connectors recognized the No Health Failures recovery sentinel only via an exact, case-sensitive comparison, so any stored message that wasn't byte-identical (different casing, a trailing ;, or a value written by an external tool) was parsed as a real fault line and re-asserted Status=True. Because such a phantom line carries no entity token, the recovery path could never remove it, wedging the condition as unhealthy while LastHeartbeatTime kept advancing. The sentinel is now matched case-insensitively so recovery is recognized correctly.
• Kernel-origin syslog checks default to SYSLOG_FACILITY=0 (#1426, #1417): The XID, SXID, and GPU-fallen-off-bus checks were built with empty Tags, so they scanned every journald facility. Under journald retention pressure, unrelated high-volume userspace logs could push the read cursor behind the retained window and the journal segment holding an XID could be vacuumed before the monitor processed it — silently dropping the event. The three kernel-origin checks now default to the -k (SYSLOG_FACILITY=0) filter so they consume only kernel entries and keep up; the Kata path continues filtering by -u containerd.service.
• Skip thermal-margin monitoring on DCGM 3.x (#1448, #1449): The GpuThermalMarginWatch check introduced in v1.10.0 depends on DCGM field 153, which is unavailable on DCGM 3.x and caused the GPU health monitor to fail initialization. The check is now capability-gated on field 153 availability — DCGM 3.x logs a warning and skips the unsupported monitor while continuing standard health monitoring, and DCGM 4.x behavior is unchanged.
• Handle malformed XID 154 lines gracefully (#1440): A crafted XID 154 log line where ) precedes ( could crash the syslog health monitor's CSV parser. Such lines are now logged as a warning and skipped instead of crashing the monitor.
• Fixed TestNICCounterIBDegradation flake (#1453, #1446): The test could fail with the shared worker node "still cordoned" because the preceding TestDCGMBootstrapCompletedAnnotation (added in #1425) deletes the nvidia-dcgm pod, correctly triggering a fatal GpuDcgmConnectivityFailure that cordoned the node; the test ended before the monitor reconnected. It now waits for the gpu-health-monitor to reconnect to DCGM before finishing. Test/CI reliability only; no runtime behavior change.

Documentation

• Per-namespace Node Drainer drain modes tutorial (#1442): Adds a tutorial documenting how to configure Immediate, AllowCompletion, and DeleteAfterTimeout eviction policies per namespace via Helm, with a multi-tier application example and validation steps.
• Writing a new health monitor tutorial (#1428): Adds a developer-facing, end-to-end guide for building, testing, containerizing, and deploying a new NVSentinel health monitor.

Acknowledgments

This release includes contributions from:

• @deesharma24
• @jackyliusohu
• @jtschelling
• @KaivalyaMDabhadkar
• @lalitadithya
• @natherz97
• @nitz2407
• @rupalis-nv
• @XRFXLP

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributor @jackyliusohu.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.13.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.12.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.13.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.12.0

Release v1.12.0

This release prevents the DCGM connectivity error that fired on every new node during GPU Operator bootstrapping, adds device-count labels from the labeler so downstream consumers can detect nodes reporting fewer GPUs or NICs than expected, adds an opt-in Magic SysRq reboot path for the generic bare-metal provider, and includes reliability fixes for the labeler, node-drainer, and the PostgreSQL store.

Major New Features

Prevent DCGM Connectivity Errors on Node Bootstrapping (#1425, #1423)

On a freshly launched node, the gpu-health-monitor pod could become ready before the GPU Operator's nvidia-dcgm pod finished its init-container startup sequence, producing a GpuDcgmConnectivityFailure unhealthy condition (DCGM_CONNECTIVITY_ERROR, CONTACT_SUPPORT) that only cleared minutes later once DCGM came up. The gpu-health-monitor is no longer scheduled until the nvidia-dcgm pod on the node is ready, so a normal node bootstrap no longer emits a false connectivity error. Node-deletion teardown behavior is unchanged.

Expected Device-Count Labels from Labeler (#1395)

The labeler can now write normalized current and expected device-count labels (e.g. nvsentinel.dgxc.nvidia.com/gpu.count.current / .expected) onto nodes, giving downstream modules a signal for detecting nodes that advertise fewer devices than their peers. Current count is derived from a configurable CEL expression supporting both device-plugin/GFD-style node labels and DRA ResourceSlice advertisements; expected count is either learned from peers in the same grouping-label partition or pinned via per-class overrides. Configured per device class (GPU, NIC) in a TOML ConfigMap and disabled by default. See ADR-043 for the design.

Opt-In SysRq Reboot for the Generic Bare-Metal Provider (#1418)

The generic bare-metal janitor provider now supports an opt-in Linux Magic SysRq reboot mode (janitor-provider.csp.generic.useSysrqReboot=true), which reboots a node by writing b to /proc/sysrq-trigger from a privileged Job rather than using the default chroot-based reboot. This is useful on hosts where the chroot path is unreliable. The existing chroot-based reboot remains the default, so existing deployments are unaffected.

Bug Fixes & Reliability

• Lazily initialize ResourceSlice informers in the labeler (#1422): The labeler eagerly started a DRA ResourceSlice informer even when device-count detection used only the device-plugin method, spamming failed to list *v1.ResourceSlice: the server could not find the requested resource errors on clusters without the resource.k8s.io API. The informer is now initialized lazily only when a class actually requires ResourceSlice data, and string digits are normalized to numbers during count evaluation. Follow-up to the device-count feature (#1395).
• Node-drainer ignores stale AlreadyQuarantined events (#1419, #1415): A stale AlreadyQuarantined event re-enqueued via a later change-stream update — after the node had already been unquarantined and its quarantineHealthEvent annotation removed — was treated as "not already drained" and fell through to normal drain evaluation. That marked the stale event Succeeded and mutated the node-state label (triggering an invalid none -> draining transition) despite there being no active quarantine context. The already-drained check now handles a missing annotation on a stale AlreadyQuarantined event correctly instead of proceeding to drain.
• Fixed PostgreSQL UpdateDocument placeholder collision (#1391): In the direct PostgreSQL store, UpdateDocument did not bind SET parameters before WHERE parameters, so combined update+filter statements could apply parameters in the wrong order. WHERE placeholders are now shifted after the update args (regex-based, so multi-digit placeholders such as $10 are not rewritten incorrectly) and executed with update args followed by filter args.

Acknowledgments

This release includes contributions from:

• @fallintoplace
• @natherz97
• @XRFXLP
• @lalitadithya

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback!

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.12.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.11.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.12.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.11.0

Release v1.11.0

This release improves the accuracy of Mean Time To Repair (MTTR) reporting by letting dashboards distinguish automated remediations from events that require manual intervention, and fixes a checkpoint-advancement bug in the fault-remediation reconciler that could silently drop live health events on cold start.

Major New Features

Recommended-action label on MTTR metrics (#1406)

The fault_quarantine_node_remediation_duration_excluding_drain_seconds MTTR histogram now carries a recommended_action label. Previously, nodes that required manual handling (e.g. a CONTACT_SUPPORT recommended action) could sit cordoned for hours before an operator acted, and that long idle time was bucketed alongside genuine automated remediations, inflating MTTR on Grafana dashboards. With the new label, dashboards can filter out CONTACT_SUPPORT and other manual events so MTTR reflects only automated remediations.

Bug Fixes & Reliability

• Fixed cold-start checkpoint advancement on document ID errors (#1411): Cold-start events are enqueued without resume tokens, but the document-ID error path in the fault-remediation reconciler called the watcher directly, where an empty token could resolve to the current MongoDB or PostgreSQL stream position and advance the checkpoint past events that had not yet been handled. Document-ID extraction failures are now routed through safeMarkProcessed, so cold-start events are no longer incorrectly marked processed and remediation events are no longer silently lost.

Acknowledgments

This release includes contributions from:

• @fallintoplace
• @nitz2407
• @XRFXLP
• @lalitadithya

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributor @fallintoplace.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.11.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.10.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.11.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.10.0

Release v1.10.0

NVSentinel v1.10.0 expands GPU health coverage with a new GPU thermal-margin watch, reduces memory footprint through optional per-policy namespace scoping in the Kubernetes Object Monitor, and lays the API foundation for external breakfix coordination via the ExternalRemediationRequest CRD. This release also adds finer-grained scheduling control for platform connectors, fixes fault-remediation handling of deleted nodes, and corrects Helm rendering for proxy-terminated PostgreSQL TLS configurations.

Major New Features

GPU Thermal Margin Watch (#1371, #1388)

Adds a new GpuThermalMarginWatch health check to the GPU health monitor that detects when a GPU crosses its hardware thermal-slowdown boundary. The monitor samples DCGM field 153 (DCGM_FI_DEV_GPU_TEMP_LIMIT) as a signed thermal-margin signal and compares it against a per-GPU hardware slowdown threshold; because that offset varies by SKU and is not exposed by DCGM, the metadata-collector reads it once per GPU via NVML field 194 (NVML_FI_DEV_TEMPERATURE_SLOWDOWN_TLIMIT) and publishes it in gpu_metadata.json. When a GPU's live margin falls below its slowdown threshold, NVSentinel raises a fatal GpuThermalMarginWatch event (error code GPU_TEMP_HW_SLOWDOWN_VIOLATION, recommended action CONTACT_SUPPORT) and clears it automatically once the margin recovers. Unlike the existing GpuThermalWatch, which only signals that throttling increased, this gives operators a quantifiable measure of how far past the hardware slowdown line a GPU has gone. The feature is opt-in via enable/store-only toggles. A companion operator runbook walks responders through confirming the alert with live telemetry and nvidia-smi, checking per-GPU threshold metadata, applying remediation, and reproducing the condition under load.

Optional Namespace Scoping in KOM Policies (#1394)

The Kubernetes Object Monitor (KOM) now supports optional per-policy namespace scoping. Setting resource.namespace on a namespaced resource in a KOM policy instructs controller-runtime to build an informer cache scoped to a single namespace rather than watching every object of that GVK cluster-wide, dramatically reducing memory usage for high-cardinality resources such as Pods. In testing, a Pod-watching policy scoped to one namespace held steady at ~18Mi even with 2000 pods scheduled in an unmonitored namespace, versus ~153Mi (roughly 9x) when watching cluster-wide. The field is rejected for cluster-scoped resources; leave it unset when cluster-wide monitoring is genuinely required.

ExternalRemediationRequest CRD Foundation (#1376)

Introduces the foundation for the ExternalRemediationRequest (ERR) CRD, a new coordination surface in the nvsentinel.dgxc.nvidia.com API group that lets NVSentinel hand off node ownership to an external breakfix system. This first PR ships the API shape only: the apiserver now accepts ERR objects via a new proto-generated CRD packaged in the janitor Helm chart, with scheme registration and RBAC granting janitor access to externalremediationrequests plus its status and finalizers subresources. It also adds custom protojson marshaling so proto well-known types (such as the Timestamp on Condition.lastTransitionTime) serialize as RFC3339, and centralizes the nvsentinel.dgxc.nvidia.com/managed node label and ERR identity constants in a new commons/pkg/managed package. This is foundational/preview only: no component observes ERR objects yet, and the reconciler, fault-remediation producer, and node-labeler gating land in follow-up PRs.

Affinity Support for Platform Connector (#1375)

The NVSentinel Helm chart now supports a platformConnector.affinity value, letting operators control how the platform connector DaemonSet pods are scheduled onto nodes. When set, the affinity block (for example, nodeAffinity rules matching custom node labels) is rendered into the DaemonSet's pod spec; when left empty (the default is {}) it renders nothing, so existing deployments are unaffected. This is useful for pinning connectors to specific node pools or hardware. The change includes a new scheduling configuration reference in the platform-connectors docs.

Bug Fixes & Reliability

• Ignore deleted nodes in fault remediation (#1396, #1387): Fixed fault-remediation retrying health events forever when the target node had been deleted from the cluster. Previously, GetRemediationState/checkExistingCRStatus failed with a Kubernetes "Node not found" error before the event could be marked terminal, so controller-runtime kept retrying and cold-start re-enqueued the stale event on every restart. The reconciler now detects the not-found error via apierrors.IsNotFound, marks remediation events for deleted nodes terminal with faultRemediated=false (cancellation events terminal with faultRemediated=true), and advances the change-stream resume token so the event is recorded as processed and never retried again.
• Render valid platform-connectors DaemonSet without a client cert (#1397, #1241): Fixed the platform-connectors DaemonSet in the umbrella Helm chart so it renders a valid manifest when PostgreSQL is the datastore but no client certificate is mounted (platformConnector.postgresqlStore.clientCertMountPath set to ""), a common configuration when TLS is terminated by a cloud-sql-proxy sidecar. Previously the template emitted a volumeMount referencing a non-existent volume, causing ArgoCD and Kubernetes to reject the DaemonSet as invalid. The cert volume, volumeMount, and fix-cert-permissions init container are now only rendered when a mount path is actually configured, bringing the DaemonSet in line with the subchart Deployments that already handled this case.
• Fixed preflight E2E test flakiness (#1374): Hardened the preflight E2E test helper to comprehensively wait for and validate all preflight-related init containers before assertions run, eliminating race conditions where assertions could execute against pods whose init containers had not yet finished. Test/CI reliability only; no runtime behavior change.
• Fixed preflight test flakiness on webhook restart (#1372): Fixed intermittent preflight test failures that occurred when the admission webhook restarted mid-test by wrapping test GPU pod creation in an automatic retry and correcting pod object construction in the test utilities. Test/CI reliability only; no runtime behavior change.

Documentation

• Supported GPU architectures (#1389): Adds an official "GPU Support" section to the README and OVERVIEW docs listing the validated NVIDIA GPU architectures: Volta (V100), Ampere (A100), Hopper (H100), Ada Lovelace (L4/L40/L40S), and Blackwell (B200, GB200, GB300, RTX Pro 6000). It clarifies that NVSentinel works with any GPU supported by the NVIDIA GPU Operator, and documents that while most components run across all architectures, the optional nccl-loopback preflight check compiles GPU kernels targeting only Ampere, Ada Lovelace, Hopper, and Blackwell, so it must be disabled or skipped on Volta/V100 nodes.

Acknowledgments

This release includes contributions from:

• @deesharma24
• @jtschelling
• @lalitadithya
• @natherz97
• @XRFXLP

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback!

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.10.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.9.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.10.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.9.0

Release v1.9.0

This release tightens preflight semantics so DCGM execution errors and non-actionable diagnostic failures no longer block workloads, adds per-init-container controls for inheriting workload env/volume mounts (so NCCL loopback can run in a clean environment while allreduce can pick up workload fabric config), introduces an optional image-cache DaemonSet for preflight images, adds an out-of-cluster deployment mode for platform-connector, and fixes a startup-race bug that left syslog-health-monitor using an empty GPU driver version for the lifetime of the pod.

Major New Features

Per-Init-Container Env & Volume Inheritance Flags (#1370)

Preflight init containers previously inherited workload environment variables matching ncclEnvPatterns and volume mounts matching volumeMountPatterns uniformly across every check. That was too broad — workload-specific NCCL/fabric configuration could poison checks meant to run with a curated environment (e.g. NCCL loopback inheriting workload settings that alter local GPU P2P/NVLink/NVSwitch behavior). Each preflight init container can now opt in or out of inheritance independently:

- name: preflight-nccl-loopback
  inheritUserEnv: false
  inheritUserVolumeMounts: false
- name: preflight-nccl-allreduce
  inheritUserEnv: true            # workload fabric config still flows through
  inheritUserVolumeMounts: true

Built-in checks default to curated environments; deployments can opt in selectively where inheritance is actually required.

Image-Cache DaemonSet for Preflight Images (#1365)

New optional DaemonSet preflight-image-cache pre-pulls all preflight check images on every node, eliminating cold-start image-pull latency from the critical path of a workload's first preflight run. Each container in the DaemonSet idles after pulling its image. Gated behind imageCache.enabled (default false), with configurable resources, pod annotations, and scheduling overrides. A pod-template config checksum annotation forces a rollout when the config content changes.

Out-of-Cluster Platform-Connector Deployment (#1359)

platform-connectors accepts an optional --kubeconfig flag for explicit out-of-cluster Kubernetes authentication. The kubeconfig path is threaded through startup, connector initialization, and pipeline transformer creation so both the Kubernetes connector and MetadataAugmentor use the same client config when platform-connectors runs outside the cluster (e.g., under systemd). When --kubeconfig is unset, existing in-cluster auth behavior is unchanged.

Synced DCGM Error Mappings (#1369)

Updated dcgmerrorsmapping.csv to match the latest upstream DCGM dcgm_errors.h enum. New mappings:

• DCGM_FR_SRAM_THRESHOLD, DCGM_FR_NVLINK_EFFECTIVE_BER_THRESHOLD, DCGM_FR_NVLINK_SYMBOL_BER_THRESHOLD, DCGM_FR_IMEX_UNHEALTHY, DCGM_FR_FABRIC_PROBE_STATE, DCGM_FR_BINARY_PERMISSIONS, DCGM_FR_GPU_RECOVERY_DRAIN_P2P → CONTACT_SUPPORT
• DCGM_FR_FALLEN_OFF_BUS, DCGM_FR_GPU_RECOVERY_REBOOT → RESTART_BM
• DCGM_FR_GPU_RECOVERY_RESET, DCGM_FR_GPU_RECOVERY_DRAIN_RESET, DCGM_FR_NCCL_ERROR → COMPONENT_RESET

Bug Fixes & Reliability

• DCGM_ST_* Should Not Fail Preflight (#1364, #1363): DCGM_ST_* codes (e.g. DCGM_ST_IN_USE, DCGM_ST_DIAG_ALREADY_RUNNING) are diagnostic execution failures — the framework could not complete the run — not confirmed hardware faults. Previously these surfaced as fatal health events that cordoned the node. preflight-dcgm-diag now retries on DCGM_ST_* for a configurable number of attempts (DCGM_DIAG_STATUS_RETRY_MAX_ATTEMPTS, DCGM_DIAG_STATUS_RETRY_INTERVAL_SECONDS); if the status persists it emits a non-fatal unhealthy HealthEvent with RecommendedAction=NONE (carrying the DCGM_ST_* status name in the errorCode) and exits successfully so the workload is not blocked. Also adds clean shutdown — dcgmStopDiagnostic is called on termination signals.
• Preflight-DCGM-Diag Non-Actionable Failures Are Non-Fatal (#1358): DCGM diag failures whose recommended action resolves to NONE (e.g., XID detected during the run with no actionable remediation) are now emitted as non-fatal — the init container exits 0 and the workload's next preflight container runs. Previously these triggered Init:Error and blocked the workload. Bumped DCGM to 4.5.2 to match gpu-health-monitor.
• Syslog-HM Driver Version Startup Race (#1362): Fixed a long-standing startup race where syslog-health-monitor cached DriverVersion = "" if the monitor started before metadata-collector populated /var/lib/nvsentinel/gpu_metadata.json. The stale empty value was then used for the lifetime of the pod, breaking driver-version-dependent XID 144–150 decoding (the analyzer fell back to WORKFLOW_NVLINK5_ERR → CONTACT_SUPPORT instead of returning RESET_GPU → COMPONENT_RESET). Subtle because #1302 had already fixed metadata recovery for PCI → GPU UUID lookups, masking this code path. GetDriverVersion() now reloads metadata at request time when the cached value is empty, so the monitor recovers once metadata-collector writes the file. A new Prometheus metric tracks XID decode requests that ran without a driver version.

Acknowledgments

This release includes contributions from:

• @XRFXLP
• @sulixu
• @lalitadithya

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributor @sulixu.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.9.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.8.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.9.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.8.0

Release v1.8.0

This release replaces node-drainer's FIFO worker queue with a two-lane priority queue so a single noisy node can no longer starve drains on other nodes, adds a drainGPUPods flag to scope eviction to GPU-requesting workloads, makes drain and quarantine overrides configurable from the kubernetes-object-monitor, fills in missing recommended actions for newer XIDs, and remediates several CVEs across container images and the Go toolchain.

Major New Features

Priority Queue for Node-Drainer (#1341)

Replaced node-drainer's ready-FIFO ordering with a two-lane priority queue layered under the existing Kubernetes rate-limiting workqueue. Events for nodes that have not yet reached draining get one high-priority representative; additional queued work for the same node stays low-priority to prevent grouped floods from blocking later nodes. Queue priority state is in-memory and follows successful node label transitions — setting draining marks the node as draining, while unquarantine or terminal drain labels clear it. Retry, drain action evaluation, and health-event lifecycle semantics are unchanged. A new Prometheus counter node_drainer_queue_items_assigned_total{priority, reason} tracks assignment decisions.

drainGPUPods Filter (#1310, #1264)

New Helm flag node-drainer.drainGPUPods (default false) restricts pod eviction during fault remediation to workloads that request GPU resources (nvidia.com/gpu or nvidia.com/pgpu). When enabled, CPU-only pods (logging agents, monitoring sidecars, infrastructure DaemonSets) stay running on the node, while GPU workloads — the ones actually blocked by the GPU fault — are evicted. The filter inspects both regular containers and init containers. Default behavior is unchanged so existing deployments are unaffected.

Drain & Quarantine Overrides from Kubernetes Object Monitor (#1342)

drainOverrides and quarantineOverrides are now configurable on health events emitted by kubernetes-object-monitor policies, matching the support that already existed in other monitors. Cluster operators can declare per-policy overrides directly in the TOML/YAML config:

healthEvent:
  componentClass: Node
  isFatal: true
  message: "Node is not ready"
  recommendedAction: CONTACT_SUPPORT
  errorCode:
    - NODE_NOT_READY
  quarantineOverrides:
    force: true                # or skip: true; do not set both
  drainOverrides:
    skip: true                 # or force: true; do not set both

force and skip are mutually exclusive per override block; the chart validates this at template time. This unlocks scenarios like "cordon the node but do not evict pods" (the example tested in the PR) without requiring a separate health monitor.

Bug Fixes & Reliability

• Missing XID Recommended Actions (#1343): Filled in recommended actions for XIDs that were missing from the gpu-health-monitor mapping but listed in the XID analyzer catalog — adds an additional GPU recovery scenario that now triggers COMPONENT_RESET and fabric-related failures that now trigger RESTART_VM. Bringing the mapping in line with the catalog prevents these XIDs from being silently classified as NONE/CONTACT_SUPPORT.
• Preflight Build Platform Arg + FQ CEL for Preflight (#1352): Fixed a missing --platform argument in the preflight-checks Docker build/publish targets that caused multi-platform image operations to silently produce single-platform artifacts. Also added a new fault-quarantine CEL policy so nodes are cordoned when preflight agents emit fatal health events (respecting existing node-exclusion settings) — preflight failures now flow through the same cordon path as other monitors.

Security & Infrastructure

• Go Toolchain 1.26.3 (#1346): Bumped Go from 1.26.2 → 1.26.3, remediating CVE-2026-39820, CVE-2026-42499, CVE-2026-42501, CVE-2026-33814, CVE-2026-39836, and CVE-2026-33811.
• Image-Level CVE Remediation (#1340):
  • preflight-nccl-loopback and preflight-nccl-allreduce: PyTorch base image nvcr.io/nvidia/pytorch:26.03-py3 → 26.04-py3 (pillow 12.1.1 → 12.2.0 fixing GHSA-whj4-6x5x-4v2j and GHSA-pwv6-vv43-88gr; onnx 1.18.0 → 1.21.0 fixing GHSA-q56x-g2fj-4rj6, GHSA-hqmj-h5c6-369m, GHSA-538c-55jv-c5g9, GHSA-3r9x-f23j-gc73). Unused uv/uvx binaries removed to eliminate the embedded vulnerable rustls-webpki (GHSA-82j2-j2ch-gfr8).
  • log-collector: kubectl v1.34.1 → v1.34.8 (picks up github.com/moby/spdystream v0.5.1, fixing GHSA-pc3f-x583-g7j2).
  • file-server-cleanup: base image python:3.13-alpine → python:3.14-alpine (fixes CVE-2026-7210 in expat, CVE-2026-6100 in Python decompression modules, CVE-2026-4786 in webbrowser).
  • gpu-health-monitor and preflight-dcgm-diag: removed unused gnupg package, eliminating CVE-2025-68973 (gnupg2).

Acknowledgments

This release includes contributions from:

• @XRFXLP
• @coderuhaan2004
• @deesharma24
• @jtschelling
• @lalitadithya

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributor @coderuhaan2004.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.8.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.7.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.8.0 \
  --namespace nvsentinel \
  --reuse-values

Release v1.7.0

Release v1.7.0

This release fixes a fault-remediation bug where every historical cancellation replayed on every restart (eventually causing OOM kills), adds a Helm gate to disable the external-MongoDB setup job for tenants who provision the database themselves, brings the docs site onto NVIDIA's shared Fern global theme, and ships a large set of GitHub repository automation workflows.

Major New Features

External MongoDB Setup Job Gate (#1311)

The post-install/post-upgrade hook job that provisions collections, indexes, and x509 users on external MongoDB can now be disabled independently of the external-MongoDB configuration. Set global.datastore.setupJob.enabled: false to opt out — useful for deployments where the datastore is provisioned out-of-band and the setup job's auth requirements don't match the tenant identity. Defaults to true, so existing deployments are unaffected.

Repository Automation Workflows (#1306)

Adds a suite of GitHub Actions workflows and issue templates for repository hygiene:

• Merge conflict check — runs on PR creation and main push; adds a needs-rebase label when a PR diverges from main.
• Dependabot auto-merge — auto-merges Dependabot PRs that contain only semver-patch updates.
• Issue triage — applies needs-triage and area/* labels to new issues.
• Labeler — applies area/* labels to PRs based on the paths touched.
• Welcome — posts a templated message on first-time contributors' issues and PRs.
• Inactive PR reminder — comments on PRs that have been inactive for 14–30 days.
• Issue SLAs — labels and comments on issues that have breached priority-tiered SLAs.
• Lock threads — locks closed issues and PRs after 90 days.

New issue templates for documentation requests and updates are added; the Question template is removed in favor of Discussions; the Bug/Feature templates now require a contributor agreement checkbox and add a component selector.

Bug Fixes & Reliability

• Fault-Remediation Cancellation Completion Marker (#1335): Fixed a bug where handleCancellationEvent cleared Kubernetes annotations and advanced the change-stream resume token but never wrote faultRemediated back to MongoDB, while the cold-start cancellation query had no faultremediated == nil filter. Together this meant every historical cancellation replayed on every fault-remediation restart, growing monotonically and eventually causing OOM kills. The fix:

  • handleCancellationEvent now calls updateNodeRemediatedStatus(true) after clearing annotations, writing the same completion marker the remediation path already writes.
  • The cold-start cancellation query leg now requires faultremediated == nil, so already-processed cancellations are excluded.
  • The call returns an error (rather than just logging) if the marker write fails, preventing the resume token from advancing without a durable terminal state.

• Slinky Drainer Annotation Prefix (#1318): Corrected the node annotation prefix used by the Slinky Drainer plugin from [J] [NVSentinel] to [T] [NVSentinel] so automated breakfix is detected with the expected T prefix. Demo documentation updated to match.

Docs Site

• NVIDIA Global Theme (#1320, #1321): Migrated the Fern docs site from per-repo theme assets to the shared global-theme: nvidia, deleting ~1,126 lines of custom theme code (footer/badge components, NVIDIA SVGs, main.css, and the footer/layout/colors/theme/logo/favicon/js/css blocks in docs.yml). Added multi-source: true to the Fern instance config so the global theme's JS bundle (OneTrust cookie consent SDK) loads alongside the CSS portion. Fern CLI was bumped to 5.30.2 (required for global-theme support).

• Frozen-Only Versioning (#1319, #1315): All versions in the docs dropdown now serve frozen content from their git tag — the "live docs" entry served from main has been removed. The newest version is stamped "Latest · vX.Y.Z" transiently at publish time. Eliminates duplicate dropdown entries, off-by-one pruning, and the dependency on the GitHub releases API for stamping. Version entries are sorted by semver descending (sort -rV) after insertion, so backport patches like v1.5.1 don't end up above newer releases; registration is now skipped when the publishing tag equals the latest release (the "Latest" stamp already covers it).

• CI Runner Migration (#1324): Standardized CI runners onto a dedicated linux-amd64-cpu4 flavor to unblock Dependabot PR merging.

Acknowledgments

This release includes contributions from:

• @KaivalyaMDabhadkar
• @XRFXLP
• @pdmack
• @zydee3
• @lalitadithya

Thanks also to @rohansav for diagnosing and authoring the cancellation completion marker fix that was cherry-picked into #1335.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.7.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.6.0:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.7.0 \
  --namespace nvsentinel \
  --reuse-values