No GPU metrics are being collected

[przemeklal]

Logs:

2025-08-11T10:20:23Z systemd[1]: Started Service for snap application dcgm.dcgm-exporter.
2025-08-11T10:20:23Z nv-hostengine[1936766]: DCGM initialized
2025-08-11T10:20:23Z dcgm.nv-hostengine[1936766]: Started host engine version 3.3.8 using port number: 5555
2025-08-11T10:20:23Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:23Z" level=info msg="Starting dcgm-exporter"
2025-08-11T10:20:23Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:23Z" level=info msg="Attemping to connect to remote hostengine at localhost:5555"
2025-08-11T10:20:23Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:23Z" level=info msg="DCGM successfully initialized!"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=info msg="Not collecting DCP metrics: This request is serviced by a module of DCGM that is not currently loaded"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=info msg="Falling back to metric file '/var/snap/dcgm/common/dcgm_metrics.csv'"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=warning msg="Skipping line 6 ('DCGM_FI_PROF_PCIE_TX_BYTES'): metric not enabled"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=warning msg="Skipping line 7 ('DCGM_FI_PROF_PCIE_RX_BYTES'): metric not enabled"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=warning msg="Skipping line 48 ('DCGM_FI_PROF_GR_ENGINE_ACTIVE'): metric not enabled"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=warning msg="Skipping line 49 ('DCGM_FI_PROF_SM_ACTIVE'): metric not enabled"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=warning msg="Skipping line 50 ('DCGM_FI_PROF_SM_OCCUPANCY'): metric not enabled"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=warning msg="Skipping line 51 ('DCGM_FI_PROF_PIPE_TENSOR_ACTIVE'): metric not enabled"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=warning msg="Skipping line 52 ('DCGM_FI_PROF_DRAM_ACTIVE'): metric not enabled"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=warning msg="Skipping line 53 ('DCGM_FI_PROF_PIPE_FP64_ACTIVE'): metric not enabled"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=warning msg="Skipping line 54 ('DCGM_FI_PROF_PIPE_FP32_ACTIVE'): metric not enabled"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=warning msg="Skipping line 55 ('DCGM_FI_PROF_PIPE_FP16_ACTIVE'): metric not enabled"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=warning msg="Skipping line 56 ('DCGM_FI_PROF_PCIE_TX_BYTES'): metric not enabled"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=warning msg="Skipping line 57 ('DCGM_FI_PROF_PCIE_RX_BYTES'): metric not enabled"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=info msg="Initializing system entities of type: GPU"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=info msg="Not collecting GPU metrics; Error getting devices count: Cannot perform the requested operation because NVML doesn't exist on this system."
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=info msg="Initializing system entities of type: NvSwitch"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=info msg="Not collecting NvSwitch metrics; no switches to monitor"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=info msg="Initializing system entities of type: NvLink"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=info msg="Not collecting NvLink metrics; no switches to monitor"
2025-08-11T10:20:24Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:24Z" level=info msg="Initializing system entities of type: CPU"
2025-08-11T10:20:25Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:25Z" level=info msg="Not collecting CPU metrics; Error retrieving DCGM MIG hierarchy: This request is serviced by a module of DCGM that is not currently loaded"
2025-08-11T10:20:25Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:25Z" level=info msg="Initializing system entities of type: CPU Core"
2025-08-11T10:20:25Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:25Z" level=info msg="Not collecting CPU Core metrics; Error retrieving DCGM MIG hierarchy: This request is serviced by a module of DCGM that is not currently loaded"
2025-08-11T10:20:25Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:25Z" level=info msg="Starting webserver"
2025-08-11T10:20:25Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:25Z" level=info msg="Pipeline starting"
2025-08-11T10:20:25Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:25Z" level=info msg="Listening on" address="[::]:9400"
2025-08-11T10:20:25Z dcgm.dcgm-exporter[1936877]: time="2025-08-11T10:20:25Z" level=info msg="TLS is disabled." address="[::]:9400" http2=false

Version:

dcgm                 3.3.8+snap-96ac85fd53   56     latest/edge    canonical✓  -

No metrics:

# curl localhost:9400/metrics
<empty output>

GPU:

4e:00.0 3D controller: NVIDIA Corporation Device 26b9 (rev a1)
62:00.0 3D controller: NVIDIA Corporation Device 26b9 (rev a1)
c9:00.0 3D controller: NVIDIA Corporation Device 26b9 (rev a1)
de:00.0 3D controller: NVIDIA Corporation Device 26b9 (rev a1)

Ubuntu:

# cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

NVIDIA packages:

# dpkg -l | grep nvidia | grep ii
ii  libnvidia-compute-570-server:amd64     570.86.15-0ubuntu0.22.04.4              amd64        NVIDIA libcompute package
ii  libnvidia-container-tools              1.17.4-1                                amd64        NVIDIA container runtime library (command-line tools)
ii  libnvidia-container1:amd64             1.17.4-1                                amd64        NVIDIA container runtime library
ii  nvidia-container-toolkit               1.17.4-1                                amd64        NVIDIA Container toolkit
ii  nvidia-container-toolkit-base          1.17.4-1                                amd64        NVIDIA Container Toolkit Base
ii  nvidia-utils-570-server                570.86.15-0ubuntu0.22.04.4              amd64        NVIDIA Server Driver support binaries

nvidia-smi output:

# nvidia-smi
Mon Aug 11 13:15:05 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.86.15              Driver Version: 570.86.15      CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L40S                    Off |   00000000:4E:00.0 Off |                    0 |
| N/A   78C    P0            151W /  350W |   35619MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA L40S                    Off |   00000000:62:00.0 Off |                    0 |
| N/A   66C    P0            124W /  350W |   34375MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA L40S                    Off |   00000000:C9:00.0 Off |                    0 |
| N/A   55C    P0            111W /  350W |   36863MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA L40S                    Off |   00000000:DE:00.0 Off |                    0 |
| N/A   60C    P0            117W /  350W |   34375MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A          779064      C   /usr/local/bin/python3.11              1240MiB |
|    0   N/A  N/A         2792393      C   /usr/src/.venv/bin/python3            34368MiB |
|    1   N/A  N/A         2792403      C   /usr/src/.venv/bin/python3            34368MiB |
|    2   N/A  N/A          807387      C   /usr/local/bin/python3.11              2484MiB |
|    2   N/A  N/A         2792474      C   /usr/src/.venv/bin/python3            34368MiB |
|    3   N/A  N/A         2792544      C   /usr/src/.venv/bin/python3            34368MiB |
+-----------------------------------------------------------------------------------------+

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/SOLENG-1468.

This message was autogenerated

[gabrielcocenza]

After talking with some people in Canonical it seems that the machines swob and plok in testflinger are using NVIDIA L40S.

We need to try to run the snap there to debug and see what is going on.

We might need to use a different CUDA version or set some env vars like @jneo8 pointed in NVIDIA/dcgm-exporter#530

[gabrielcocenza]

After some investigation we discovered that dcgm doesn't allow two or more clients at the same time. See: NVIDIA/dcgm-exporter#537

That means it's not possible to use the dcgm-snap with the dcgm-exporter alongside with a k8s daemonset with the dcgm-exporter.

There are two ways of using the gpu-operator:

1. Manually install the driver in the host and force the gpu-operator use the one already installed
2. Let the gpu-operator install the driver

Option 1 works fine if the dcgm-exporter from the gpu-operator is not installed, but it's not ideal for cloud operations

Option 2 the dcgm snap cannot work without some hacky workaround like manually installing debian packages which I wasn't able to get working, Talking with people from field they don't recommend doing this.

To forward the GPU metrics to COS I found two possible solutions:

• patch the dcgm-exporter daemonset installed from gpu-operator to use dnsPolicy: ClusterFirstWithHostNet. With this HWO could still configure COS to scrape at localhost:9400 and not installing the dcgm snap. We could also check if the enpoint is working as expected and block if metrics are not being generated.

• configure prometheus from COS to use the following config:

job_name: 'custom-dcgm'
scrape_interval: 30s
kubernetes_sd_configs: # tells Prometheus to auto-discover targets in Kubernetes.
- role: endpoints  # discover Endpoints objects (i.e., the pods behind a Service).
  namespaces:  # restricts discovery to only the gpu-operator-resources namespace
    names:
    - gpu-operator-resources
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
  action: keep
  regex: nvidia-dcgm-exporter # Only keep services where app=nvidia-dcgm-exporter
- source_labels: [__meta_kubernetes_endpoint_port_name]
  action: keep
  regex: gpu-metrics # Only keep endpoints whose port is named gpu-metrics

With that all endpoints of the dcgm-exporter pods are configured to be scraped. All of this just using k8s labels and it adjust dinamically if pods change IP.

@Deezzir pointed out that even with those workarounds, our Grafana Dashboard might not work as expected because the snap adds more metrics than what is shipped by default. That would require operators to manually change the k8s ConfigMap

[Pjack]

IMO, it’s not reasonable to combine a non-Juju managed exporter with the hardware-observer operator. The user should choose either Helm (gpu-operator) or Juju (hardware-observer) to manage metrics collection and integration with the monitoring system.

Currently, the setup is still not working even after disabling dcgm-exporter in gpu-operator. The issue is that the driver installation path used by gpu-operator does not align with what dcgm-snap expects. I’m not sure whether installing drivers under /run/nvidia/driver is common practice. It makes dcgm-snap cannot locate the driver correctly.

Therefore, please consider either

1. Pre-install the drivers on the host and configure gpu-operator to use them. Same as Option 1 provided by Gabriel.
   https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html#pre-installed-nvidia-gpu-drivers
   When deploying gpu-operator, make sure to disable dcgm-exporter so that dcgm-snap can function properly on the host.
   dcgmExporter.enabled=false

2. Disable dcgm snap and only install gpu-operator.
   Note: hardware-observer operator. doesn't support to disable dcgm snap right now but it should be easy to implement it.

For long term. We could consider

1. Submitting a request to gpu-operator to allow customizing the driverInstallDir, then dcgm snap can locate the driver correctly. In this commit

We may consider removing this assumption in the future and support driver container implementations
which allow for a custom driverInstallDir to be specified.

2. Extend dcgm-snap to recognize custom driver paths set up by gpu-operator. This need more time to evaluate it.

3. Option 2 provided by @gabrielcocenza

patch the dcgm-exporter daemonset installed from gpu-operator to use dnsPolicy: ClusterFirstWithHostNet. With this HWO could still configure COS to scrape at localhost:9400 and not installing the dcgm snap.

[Pjack]

There is a 4th long-term solution:

• Manage the GPU operator with a charm. The gpu-operator-charm
  is not well maintained currently, but it could serve as a good starting point and integrate it with COS (Canonical Observability Stack).