Skip to content

14-privileged-linux-nvidia.md

Latest commit

 

History

History
115 lines (88 loc) · 5.13 KB

14-privileged-linux-nvidia.md

File metadata and controls

115 lines (88 loc) · 5.13 KB
Raw

Privileged Metrics — Linux + NVIDIA

GPU, power, and thermal on a Linux host with an NVIDIA GPU (workstation, DGX, Grace-Blackwell like a GB10). For the shared helper + systemd deployment, see Privileged Metrics (overview); this guide is the NVIDIA specifics.

What you get, and from where

Metric Source Needs root?
gpu.util, gpu.vram, gpu.temp, power.gpu nvidia-smi no
power.cpu RAPL (/sys/class/powercap/intel-rapl) via the helper yes
temp.pkg hwmon (coretemp / k10temp / zenpower) via the helper yes

GPU metrics are unprivileged — nvidia-smi is readable by any user, so a normal daemon reports the full GPU panel with no helper. The daemon runs:

nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw \
  --format=csv,noheader,nounits

Verify:

nvidia-smi -L                                  # confirm the driver sees the GPU
./bin/heimdall-daemon --once | grep -E "gpu|power"
# e.g. gpu.util=0%  gpu.vram=1%  gpu.temp=33C  power.gpu=4.95W

gpu.vram is reported as a percentage of total, with the absolute used / total GB in the detail view.

Unified memory (GB10 Grace-Blackwell)

On a GB10 the GPU shares the system LPDDR5X pool — there is no discrete VRAM, so nvidia-smi reports memory.used/memory.total as [N/A] and the aggregate counter is empty. Heimdall falls back to per-process GPU memory over system RAM:

nvidia-smi --query-compute-apps=used_memory --format=csv,noheader,nounits
# sum of these ÷ total system RAM → gpu.vram, e.g. "41.6 / 121.6 GB (shared)"

The detail is tagged (shared) to mark it as the unified pool. An idle GPU (no resident contexts) reads a stable 0% rather than dropping the metric. Discrete NVIDIA cards keep using the aggregate counter unchanged.

Troubleshooting — GPU metrics missing

If gpu.* is blank but power.cpu still reads, run the query by hand:

nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits
  • Failed to initialize NVML: Driver/library version mismatch — the driver package was upgraded but the running kernel module is still the old version. Reboot the host (or reload the modules). Confirm with cat /proc/driver/nvidia/version vs nvidia-smi --version; a pending /var/run/reboot-required is the tell. Since v2.2.5 the daemon surfaces this as gpu.util / gpu.vram unavailable with the nvidia-smi reason in the detail, instead of a silent blank.
  • command not found — the NVIDIA driver/utilities are not installed, or not on the daemon's PATH.

CPU power & temperature (helper)

NVIDIA covers the GPU; CPU power and temperature come from the Linux privileged sources served by heimdall-helper. Power standardizes on three metrics — the same CPU / GPU / total split btop and top show:

  • power.cpu — the CPU package (whole socket), from the RAPL package domain, sampled as an energy-counter delta.
  • power.gpu — the GPU, from nvidia-smi (a separate rail).
  • power.total — power.cpu + power.gpu (+ power.npu), the whole-machine figure the top view headlines.
  • temp.pkg from a trusted hwmon chip.

Why power.cpu can read below power.gpu. They are separate rails: the CPU package vs. a discrete NVIDIA card. A workstation happily shows e.g. cpu 33W, gpu 458W, total 491W. power.total is what reflects real draw.

GB10 (Grace/ARM) has no CPU power sensor

A GB10 exposes no RAPL, INA, or SoC power sensor to the OS — only the GPU rail via nvidia-smi. So power.cpu reads unavailable (no RAPL power sensor (SoC/ARM)) and power.total is the GPU power alone. The Grace CPU / module draw is simply not measurable from userspace on this platform. power.npu likewise stays unavailable on Intel/AMD hosts — their NPUs expose no power counter yet, same as Apple's ANE.

Any of these may be absent (no powercap, no recognised sensor); that metric reads unavailable and the daemon keeps running. Set the helper up with the systemd layout in Run both as systemd services.

Multi-GPU

nvidia-smi returns one row per GPU and the daemon reads them all, aggregating into the single gpu.* set: power.gpu sums across cards (so power.total reflects the whole box), gpu.util / gpu.clock / gpu.mem.util average, gpu.temp and gpu.fan report the hottest card, and gpu.vram pools used/total with a (N GPUs) note in the detail. Per-device breakout (separate rows per GPU) is not wired yet — the aggregate is the fleet-level view.

Intel NPU

On hosts with an Intel NPU (the intel_vpu driver — Core Ultra / Arrow Lake and later), npu.util is a real reading, sampled from the driver's cumulative /sys/class/accel/accel*/device/npu_busy_time_us counter (unprivileged; idle reads 0%). AMD XDNA (amdxdna) and Apple's ANE expose no utilisation counter, so npu.util stays unavailable there. No vendor exposes an NPU power counter, so power.npu is unavailable unless the SoC surfaces it (Apple ANE).