Thanks for such awesome software, really appreciate using it.
gpud is collecting all the infiniband counter information:
|
func parseInfiniBandCounters(cd classDirInterface, portDir string, op *Op) (*Counters, error) { |
|
func parseInfiniBandHwCounters(cd classDirInterface, portDir string, op *Op) (*HWCounters, error) { |
It would be really useful to expose these somewhere such that users can better understand errors reported by the drivers.
Ideally, this would be flag that would set a threshold amount as an integer, and detect when certain counters have increased since the last poll by that threshold or more.
Our simplest use case would be that we want to have gpud report unhealthy for this component, for example if we see symbol errors increasing:
{
"component": "accelerator-nvidia-infiniband",
"states": [
{
"time": "2026-03-30T23:21:25Z",
"component": "accelerator-nvidia-infiniband",
"name": "accelerator-nvidia-infiniband",
"health": "Unhealthy",
"reason": "mlx5_11: symbol_error increased more than threshold (0 -> 1, threshold=1)"
}
]
}Would be happy to help code this up and test in our environment, if this sounds of interest to you folks. This would reduce some of our internal NPD custom bash scripts that also collects this information.
Thanks for such awesome software, really appreciate using it.
gpud is collecting all the infiniband counter information:
gpud/components/accelerator/nvidia/infiniband/class/class.go
Line 323 in 50e472c
gpud/components/accelerator/nvidia/infiniband/class/class.go
Line 450 in 50e472c
It would be really useful to expose these somewhere such that users can better understand errors reported by the drivers.
Ideally, this would be flag that would set a threshold amount as an integer, and detect when certain counters have increased since the last poll by that threshold or more.
Our simplest use case would be that we want to have gpud report unhealthy for this component, for example if we see symbol errors increasing:
{ "component": "accelerator-nvidia-infiniband", "states": [ { "time": "2026-03-30T23:21:25Z", "component": "accelerator-nvidia-infiniband", "name": "accelerator-nvidia-infiniband", "health": "Unhealthy", "reason": "mlx5_11: symbol_error increased more than threshold (0 -> 1, threshold=1)" } ] }Would be happy to help code this up and test in our environment, if this sounds of interest to you folks. This would reduce some of our internal NPD custom bash scripts that also collects this information.