What did you do?
On hosts with NVIDIA ConnectX-7 adapters whose ports are firmware-managed (restricted PF — same hardware class as #3434; counter reads on those ports fail per #3265), I used the device filter added in v1.12.0 to stop the collector from touching those devices:
prometheus-node-exporter --collector.infiniband.device-exclude=^mlx5_(12|13|14|15)$ ...
- node_exporter version: 1.12.1 (revision 6044da7)
- OS/kernel: Ubuntu, 6.5.0-45-generic, NVIDIA OFED/DKMS ib_core / mlx5_ib
- Problem devices: mlx5_12..15 (ConnectX-7, MT4129, PCI 0000:a8:00.0-.3)
What did you expect to see?
Excluded devices are skipped before any sysfs access — that is the point of the filter on this hardware class: reading ports/*/counters/* on these ports triggers a firmware command that the device rejects, which floods the kernel log (rate-limited):
mlx5_core 0000:a8:00.0: mlx5_cmd_out_err:839:(pid …): ACCESS_REG(0x805) op_mod(0x1) failed,
status bad operation(0x2), syndrome (0x9a6171), err(-22)
What did you see instead?
The filter works for the emitted metrics (excluded devices disappear from /metrics), but the collector still opens and reads the excluded devices' counter files on every scrape, so the kernel errors continue (~40 lines/min/host after the flag was applied).
Evidence (reproducible):
- Flag present in the running process argv; excluded devices absent from
/metrics.
strace -f --seccomp-bpf -e trace=openat -p <node_exporter pid> for 30 s:
120 opens of /sys/class/infiniband/mlx5_12/ports/1/counters/* (and 120 for mlx5_13) — all excluded devices.- ftrace kprobe on
mlx5_cmd_out_err (fires only on failed firmware commands): 100% of failures come from the node_exporter task, with stack:
mlx5_cmd_out_err ← mlx5_access_reg ← mlx5_core_access_reg ← query_ib_ppcnt
← mlx5_ib_process_mad ← get_perf_mad ← show_pma_counter ← port_attr_show
← sysfs_kf_seq_show ← kernfs_seq_show ← seq_read_iter ← kernfs_fop_read_iter
← vfs_read ← ksys_read
Note this is not the collector-abort problem from #3265 — failed reads are tolerated (EINVAL is skipped) and node_scrape_collector_success{collector="infiniband"} stays 1; the remaining problem is the reads themselves (firmware churn + kernel log spam).
Workaround that fully stops it (for reference): replacing the device's counters/ dir with an empty dir in the node_exporter service's mount namespace:
[Service]
BindReadOnlyPaths=/var/lib/node-exporter-hidden/empty:/sys/class/infiniband/mlx5_12/ports/1/counters
# ... one line per port
After that: zero kernel errors, collector success=1, all non-excluded metrics intact.
Request
Please apply the device include/exclude check before any sysfs access for that device (e.g. at the top of the per-device loop in collector/infiniband_linux.go), or document the current behavior if it is intentional. Happy to test a patch — we have reproducible hardware for this.
What did you do?
On hosts with NVIDIA ConnectX-7 adapters whose ports are firmware-managed (restricted PF — same hardware class as #3434; counter reads on those ports fail per #3265), I used the device filter added in v1.12.0 to stop the collector from touching those devices:
What did you expect to see?
Excluded devices are skipped before any sysfs access — that is the point of the filter on this hardware class: reading
ports/*/counters/*on these ports triggers a firmware command that the device rejects, which floods the kernel log (rate-limited):What did you see instead?
The filter works for the emitted metrics (excluded devices disappear from /metrics), but the collector still opens and reads the excluded devices' counter files on every scrape, so the kernel errors continue (~40 lines/min/host after the flag was applied).
Evidence (reproducible):
/metrics.strace -f --seccomp-bpf -e trace=openat -p <node_exporter pid>for 30 s:120 opens of
/sys/class/infiniband/mlx5_12/ports/1/counters/*(and 120 for mlx5_13) — all excluded devices.mlx5_cmd_out_err(fires only on failed firmware commands): 100% of failures come from the node_exporter task, with stack:Note this is not the collector-abort problem from #3265 — failed reads are tolerated (EINVAL is skipped) and
node_scrape_collector_success{collector="infiniband"}stays 1; the remaining problem is the reads themselves (firmware churn + kernel log spam).Workaround that fully stops it (for reference): replacing the device's
counters/dir with an empty dir in the node_exporter service's mount namespace:After that: zero kernel errors, collector success=1, all non-excluded metrics intact.
Request
Please apply the device include/exclude check before any sysfs access for that device (e.g. at the top of the per-device loop in
collector/infiniband_linux.go), or document the current behavior if it is intentional. Happy to test a patch — we have reproducible hardware for this.