[Question]: Don’t you think that whole thing is overengineered and too complex?

[antonibertel]

Prerequisites

• I searched existing issues and docs

Question

Hi guys,

It feels like extreme overkill to have a modal architecture and install cert-manager, Prometheus, and MongoDB just to identify incorrectly behaving nodes, don’t you think?

Category

Best Practices

Context

No response

[lalitadithya]

Hey! Totally fair question from the outside the dependency list can look heavy. Here's how we got here:

NVSentinel began as a DaemonSet watching for GPU errors and rebooting nodes. As we moved through GPU generations, failure modes changed. Some errors need a node reboot, others need a full VM replacement from the cloud provider, and some just need the workload moved off a specific GPU while we do a reset. A single "cordon the node" response no longer cuts it we needed a pipeline that can detect → quarantine → drain → remediate, each with genuinely different logic.

Then clusters became multi-workload. When customers started running training and inference on the same cluster, drain behavior had to diverge. A distributed training job needs time to checkpoint, while a stateless inference pod should be evicted immediately. This is configured per-namespace but requires structured event data flowing between independent modules.

Why MongoDB?

We actually started with Kubernetes Custom Resources as our event store. The problem is that GPU XIDs tend to arrive in bursts — a failing GPU can generate several error events in minutes across multiple monitors. At scale, this hammered the Kubernetes API server: etcd compaction pressure, list/watch latency spikes, and API server throttling that affected other cluster operations. Moving to a dedicated datastore solved this and gave us:

1. Change streams for reactive, decoupled module coordination
2. Aggregation queries for correlation. e.g., "5+ XID 13 errors on the same GPU in 24h = possible hardware degradation" vs. "1 XID 13 = application error." Without time-windowed queries over historical events, you're either over-reacting to every blip or missing real failures.

Why cert-manager?

In production, connections to the database need to be secured with TLS — cert-manager handles certificate issuance and rotation automatically.

Why Prometheus?

It's optional.

Happy to dive deeper into any component if you're curious!