[Bug]: metrics-access NetworkPolicy blocks DCGM host engine port 5555 when using GPU Operator

[harinik05]

Prerequisites

• I searched existing issues
• I can reproduce this issue

Bug Description

When NVSentinel is deployed with GPU Operator's dcgm host engine and gpu-health-monitor, the metrics-access NetworkPolicy doesn't allow ingress on DCGM host engine port (by default 5555). As a result, gpu-health-monitor pods cannot reach nvidia-dcgm.gpu-operator.svc and have reported connectivity issues until this network policy was patched.

Component

Health Monitor

Steps to Reproduce

When global.dcgm.enabled and gpu-health-monitor.dcgm.dcgmK8sServiceEnabled are set and DCGM service endpoint is configured, the chart's metrics-access network policy should allow ingress to that DCGM port so that gpu health monitor can talk to DCGM host engine.
Network Policy Link

Environment

• NVSentinel version:
• Kubernetes version:
• Deployment method:

Logs/Output

No response

[deesharma24]

@harinik05 Few things:-

1. Egress is not blocked from the network policy. So, gpu-health-monitor should be able to reach dcgm server without any issue.
2. And as part of the same cycle (Via polling), it gets the health checks info from dcgm. So, ideally -there should not be any issue.

Can you confirm Are you seeing any issue when deployed this ?

[harinik05]

@deesharma24
When I deploy GPU operator's dcgm alongside gpu health monitor, I saw this issue: DCGM connectivity failure detected, sending GpuDcgmConnectivityFailure health event. It was resolved when I added the port patch for ingress mentioned earlier. Is there a different way of deploying this?

[lalitadithya]

@deesharma24 , did you get a chance to triage this issue futher?