Implement Prometheus metrics for gpu-kubelet-plugin, compute-domain-plugin, and dra-controller

[shengnuo]

Fixes #352

Metrics prefix

All DRA Prometheus metrics use prefix nvidia_gpu (exported names are nvidia_gpu_dra_<name>).

---

Kubelet Plugins (GPU & ComputeDomain)

Metric	Type	Labels	Note
requests_total	Counter	driver, operation	Count of prepare/unprepare requests
request_duration_seconds	Histogram	driver, operation	Latency; uses exponential bucket 0.05*2^n, 0<=n<=8
requests_inflight	Gauge	driver, operation	Concurrent prepare/unprepare
prepared_devices	Gauge	node, driver, device_type	Current prepared devices from the checkpoint file
node_prepare_errors_total	Counter	driver, error_type	Node prepare failures (plugin-scoped)
node_unprepare_errors_total	Counter	driver, error_type	Node unprepare failures

---

Compute-domain controller

Metric	Type	Labels	Note
compute_domain_info	GaugeVec	status	Expose ComputeDomain objects by status, from informer cache

Sample

Sample metrics on the GPU kubelet plugin after preparing and unpreparing a GPU and a MIG device

# HELP nvidia_gpu_dra_prepared_devices Current number of prepared devices by device type.
# TYPE nvidia_gpu_dra_prepared_devices gauge
nvidia_gpu_dra_prepared_devices{device_type="gpu",driver="gpu.nvidia.com",node="nim-operator-9vg0z43"} 0
nvidia_gpu_dra_prepared_devices{device_type="mig",driver="gpu.nvidia.com",node="nim-operator-9vg0z43"} 0
nvidia_gpu_dra_prepared_devices{device_type="unknown",driver="gpu.nvidia.com",node="nim-operator-9vg0z43"} 0
nvidia_gpu_dra_prepared_devices{device_type="vfio",driver="gpu.nvidia.com",node="nim-operator-9vg0z43"} 0
# HELP nvidia_gpu_dra_request_duration_seconds Duration of DRA prepare and unprepare requests.
# TYPE nvidia_gpu_dra_request_duration_seconds histogram
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="prepare",le="0.05"} 0
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="prepare",le="0.1"} 1
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="prepare",le="0.2"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="prepare",le="0.4"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="prepare",le="0.8"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="prepare",le="1.6"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="prepare",le="3.2"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="prepare",le="6.4"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="prepare",le="12.8"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="prepare",le="+Inf"} 2
nvidia_gpu_dra_request_duration_seconds_sum{driver="gpu.nvidia.com",operation="prepare"} 0.19524701700000002
nvidia_gpu_dra_request_duration_seconds_count{driver="gpu.nvidia.com",operation="prepare"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="unprepare",le="0.05"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="unprepare",le="0.1"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="unprepare",le="0.2"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="unprepare",le="0.4"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="unprepare",le="0.8"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="unprepare",le="1.6"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="unprepare",le="3.2"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="unprepare",le="6.4"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="unprepare",le="12.8"} 2
nvidia_gpu_dra_request_duration_seconds_bucket{driver="gpu.nvidia.com",operation="unprepare",le="+Inf"} 2
nvidia_gpu_dra_request_duration_seconds_sum{driver="gpu.nvidia.com",operation="unprepare"} 0.004752574
nvidia_gpu_dra_request_duration_seconds_count{driver="gpu.nvidia.com",operation="unprepare"} 2
# HELP nvidia_gpu_dra_requests_inflight Number of in-flight DRA prepare and unprepare requests.
# TYPE nvidia_gpu_dra_requests_inflight gauge
nvidia_gpu_dra_requests_inflight{driver="gpu.nvidia.com",operation="prepare"} 0
nvidia_gpu_dra_requests_inflight{driver="gpu.nvidia.com",operation="unprepare"} 0
# HELP nvidia_gpu_dra_requests_total Total number of DRA prepare and unprepare requests.
# TYPE nvidia_gpu_dra_requests_total counter
nvidia_gpu_dra_requests_total{driver="gpu.nvidia.com",operation="prepare"} 2
nvidia_gpu_dra_requests_total{driver="gpu.nvidia.com",operation="unprepare"} 2

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[shivamerla]

Great work! left some comments. Thanks.

[jgehrcke]

All DRA Prometheus metrics use prefix nvidia_gpu_dra (exported names are nvidia_gpu_dra_).

Iterating on user-facing metric names is really important before a first release. Hence, I'd like to ask a few questions (I am sure there are more):

• Does nvidia need to be part of the prefix (also in view of the upcoming donation -- the answer is probably "yes")?
• What really is the common prefix for all metrics emitted by anything in this DRA driver? Should this maybe just be nvidia_dra?
• If we want "gpu" as part of the prefix: do we want the prefix to be nvidia_dra_gpu (sorting by specificity)?
• If we want "gpu" as part of the prefix -- does this apply to CD-related metrics as well (would we see something like nvidia_dra_gpu_cd_<...>?) My gut feeling is that we should have one metric prefix for each plugin; and that could mean we would end up using nvidia_dra_gpu and nvidia_dra_cd -- which then raises the question of how to prefix controller-emitted metrics.

Just thinking out loud! This needs a bit of brainstorm between a number of people that ask critical questions, and we also need to look at different variants side by side. Bad naming choices will either stick around forever, or create quite a lot of migration pain later on.

[jgehrcke]

compute_domain_daemon_peer_nodes

What does this really measure and why is this useful? Which labels did you consider adding to this metric, and why?

[jgehrcke]

As a general note, let's make sure to read https://prometheus.io/docs/practices/naming :-) (before trying to construct metric names).

compute_domains GaugeVec status Expose ComputeDomain objects by status, from informer cache

Why would this be useful? Let's think about this end to end: let's construct a query-time (dashboarding/alerting) use case that demonstrates value (there may be one, but let's make it obvious).

About "by status": which status do you refer to?

About "Expose ComputeDomain objects": what do you mean with that?

Should this gauge reflect a current count?

compute_domain_cliques_total Gauge (none) Number of ComputeDomainClique CRs in informer cache

Same here: let's discuss the primary use case.

compute_daemons_per_clique

Same here.

When I look at the name compute_daemons I think it's also rather apparent that we need to think about CD-related prefix which should maybe just be cd.

I think we maybe should refrain from adding CD daemon/controller-emitted metrics here in this first patch. The kubelet plugin metrics you have listed all seem to be rather sane, but there are so many questions about the currently proposed metrics emitted by CD daemon/controller. I think this patch may land faster if we keep it focused on the basic plugin metrics.

[shivamerla]

@shengnuo Could you share a few examples of the metrics being collected, along with any sample dashboards (if available)?

[shengnuo]

@shivamerla Updated the PR description with sample Prometheus metrics

[shivamerla]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: shengnuo, shivamerla

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [shivamerla]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment