What is the version?
4.4.1-4.6.0
What happened?
DCGM exporter pods stop collecting profiling metrics, e.g. DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE after running for a while (usually around after 5-6 days). Restarting the pods mitigates the problem.
Error logs from nv-hostengine.log:
2026-02-13 19:48:54.762 ERROR [1:38] Got nvmlSt 3 for field 139 from nvmlDeviceGetFieldValues [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13381] [DcgmCacheManager::ReadAndCacheNvLinkBandwidth]
2026-02-13 19:48:54.762 ERROR [1:38] Got nvmlSt 3 for field 138 from nvmlDeviceGetFieldValues [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13381] [DcgmCacheManager::ReadAndCacheNvLinkBandwidth]
2026-02-13 19:48:55.221 ERROR [1:54] GetLatestSample returned Feature not supported for entityId 0 groupId 1 fieldId 449 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4709] [DcgmCacheManager::GetMultipleLatestSamples]
2026-02-13 19:49:02.002 ERROR [1:37] [[NvSwitch]] Not attached to NvSwitches. Aborting [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:680] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches]
2026-02-13 19:49:10.219 ERROR [1:1] GetLatestSample returned Feature not supported for entityId 0 groupId 1 fieldId 449 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4709] [DcgmCacheManager::GetMultipleLatestSamples]
2026-02-13 19:49:24.762 ERROR [1:38] Got nvmlSt 3 for field 139 from nvmlDeviceGetFieldValues [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13381] [DcgmCacheManager::ReadAndCacheNvLinkBandwidth]
2026-02-13 19:49:24.762 ERROR [1:38] Got nvmlSt 3 for field 138 from nvmlDeviceGetFieldValues [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13381] [DcgmCacheManager::ReadAndCacheNvLinkBandwidth]
2026-02-13 19:49:25.220 ERROR [1:53] GetLatestSample returned Feature not supported for entityId 0 groupId 1 fieldId 449 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4709] [DcgmCacheManager::GetMultipleLatestSamples]
2026-02-13 19:49:32.002 ERROR [1:37] [[NvSwitch]] Not attached to NvSwitches. Aborting [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:680] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches]
2026-02-13 19:49:40.221 ERROR [1:61] GetLatestSample returned Feature not supported for entityId 0 groupId 1 fieldId 449 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4709] [DcgmCacheManager::GetMultipleLatestSamples]
What did you expect to happen?
Long running DCGM exporters should keep collecting profiling metrics.
What is the GPU model?
No response
What is the environment?
No response
How did you deploy the dcgm-exporter and what is the configuration?
Deployed via helm-chart
How to reproduce the issue?
No response
Anything else we need to know?
No response
What is the version?
4.4.1-4.6.0
What happened?
DCGM exporter pods stop collecting profiling metrics, e.g. DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE after running for a while (usually around after 5-6 days). Restarting the pods mitigates the problem.
Error logs from nv-hostengine.log:
What did you expect to happen?
Long running DCGM exporters should keep collecting profiling metrics.
What is the GPU model?
No response
What is the environment?
No response
How did you deploy the dcgm-exporter and what is the configuration?
Deployed via helm-chart
How to reproduce the issue?
No response
Anything else we need to know?
No response