ComputeDomain: explore exposing Prometheus metrics

[jgehrcke]

It might make sense to expose system state via canonical Prometheus metrics. Let's not do this only for the sake of "adding metrics", but instead properly think through what is going to be of value for health monitoring, alerting, and debugging.

Some thoughts:

• The controller pod might be the component of choice for exposing metrics about global system state, current ComputeDomain count, and transient error count, state of any individual ComputeDomain, ...
• Maybe each plugin pod should also expose a Prometheus endpoint exposing metrics about itself
• Think through entire pipeline: how to point canonical scrapers to these endpoints? Maybe with the ServiceMonitor primitive from Prometheus Operator?

The real task here is to do quite a bit more thinking and planning before building anything. Because what to build isn't quite obvious at all.

[shengnuo]

Currently only compute-domain-controller exposes Prometheus, but they are not project specific and only platform metrics are exposed. gpu-kubelet-plugin, compute-domain-kubelet-plugin and compute-domain-daemon do not expose any metrics.

Here are some potential metrics category we could explore.

1. DRA request lifecycle

Gives direct SLO visibility for preparing and unpreparing latency

Metrics

a. nvidia_dra_request_total (counter)
b. nvidia_dra_request_duration_seconds (histogram)
c. nvidia_dra_requests_inflight (gauge)

Instrumentation

PrepareResourceClaims and UnprepareResourceClaims in gpu-kubelet-plugin and compute-domain-kubelet-plugin

2. GPU / compute-domain usage

Provides real time capacity and consumption insight

Metrics

a. nvidia_dra_prepared_claims (gauge)
b. nvidia_dra_prepared_devices (gauge)

Instrumentation

Prepare, Unprepare and check point transitions in device_state.go in gpu-kubelet-plugin and compute-domain-kubelet-plugin

3. Device Healthiness

Provides signal into device failure

Metrics

a. nvidia_dra_allocatable_devices_healthiness (gauge)
b. nvidia_dra_device_health_events_total (counter)

Instrumentation

device_health.go and deviceHealthEvents in driver.go for gpu-kubelet-plugin
compute domain TBD

4. IMEX daemon operations

Provide insights into IMEX daemon instability, such as config/update or restart loops

Metrics

a. nvidia_dra_imex_updates_total (counter)
b. nvidia_dra_imex_update_duration_seconds (histogram)
c. nvidia_dra_imex_process_restarts_total (counter)

[guptaNswati]

what will nvidia_dra_requests_inflight capture?

this seems like a good metric on the healthnvidia_dra_device_health_events_total

[shengnuo]

what will nvidia_dra_requests_inflight capture?

Thanks for the feedback @guptaNswati . nvidia_dra_requests_inflight will capture number of active DRA operations being processed right now (prepare and unprepare).

It goes up when a request starts processing and goes down when processing is finished.
This should give visibility into saturation/backpressure in DRA

[shivamerla]

@shengnuo Thanks for taking this up. For the kubelet plugins, we should add metrics to track failures during the Prepare and Unprepare lifecycle operations. Specifically, we can expose counters such as node_prepare_errors_total and node_unprepare_errors_total, labeled by error type and device type. It would also be useful to track the type of devices being allocated so we can understand allocation patterns across the cluster. For example, a labeled counter or gauge for allocations by device type (GPU, MPS, MIG, VFIO) would help with observability and debugging capacity issues.

For the CD controller, we should add cluster-level metrics that capture the overall state of the system. This could include the total number of ComputeDomains in the cluster, as well as counts grouped by status (e.g., Ready, NotReady, Failed). We should also expose metrics for the total number of IMEX daemons and their current status to help quickly identify unhealthy or missing daemons.

For the IMEX daemon controller, we should expose metrics for the total number of CDClique resources and the number of IMEX daemons associated with each clique. This would make it easier to understand the topology and ensure the expected number of daemons are running per clique.

Finally, within the IMEX daemon itself, we can emit runtime metrics that reflect the state of the ensemble. For example, a gauge for the total number of peer nodes currently visible in the ensemble, and a counter for the number of daemon restarts triggered by SIGUSR1. This will help to understand how long it took for the IMEX ensemble to be ready.