Verbose logging when CD pods are not starting due to a resource related issue

[OguzPastirmaci]

Not sure if possible to do it and I guess it's not directly related to DRA, but adding here just in case there's something that can be added.

Would the logs be more verbose about why the CD pods are not being scheduled? Today we had a GB200 rack with a hostPath not mounted properly. Workload pods were scheduled, CD pods were not. We didn't see anything to give us a hint about it in the logs.

[jgehrcke]

@OguzPastirmaci we care about debuggability, and acknowledge that things may still be rough.

We certainly want to make it easier over time to identify common issues.

To make this slightly more actionable:

Which version of the DRA driver are/were you using? Do you have any kubectl output to share?

When you say

Workload pods were scheduled, CD pods were not

Do you know if the corresponding DaemonSet was created? Were the CD daemon pods in any way part of the output of kubectl get pods -n nvidia-dra-driver-gpu?

[shivamerla]

@OguzPastirmaci we have documented steps to increase verbosity of CD controller and kubelet plugins here. Please check if this helps and re-open if you are still seeing the error without any meaningful logs.