KAI scheduler not working with AWS EKS Auto Mode

[msaeedevops]

Hello
I am running eks cluster and I tried deploy KAI Scheduler on it.
Deployed nvidia device plugin and gpu operator.
Spun gpu nodes via karpenter and created queue as well

After spinning nodes up, If I check daemonset and device plugin pods I get this,

kubectl get daemonset -n kube-system | grep nvidia
nvidia-device-plugin                      0         0         0       0            0           accelerator=nvidia-gpu,karpenter.sh/nodepool=ml-pool                               161m
nvidia-device-plugin-mps-control-daemon   0         0         0       0            0           accelerator=nvidia-gpu,karpenter.sh

kubectl get pods -n kube-system -l app.kubernetes.io/name=nvidia-device-plugin -o wide
No resources found in kube-system namespace.

Does it mean its KAI scheduler is not supported by EKS Auto Mode or am I missing something

I tried using same labels and taints in the EKS Node pool which is provisioning the nodes but no luck.

[enoodle]

Hi @msaeedevops 👋

I am not very familiar with EKS Auto Mode, but this seems to be an issue with the nvidia gpu operator and not with KAI.
I think that basic examples will probably be easy to setup for KAI on EKS Auto Mode but maybe there will be some issues between KAI's cluster auto scaler and karpenter.

I would try to use this aws guide to make sure that the pods can run on EKS and then continue to install KAI on the cluster.
Edit: The link to the AWS guide : https://docs.aws.amazon.com/eks/latest/userguide/auto-accelerated.html