GPU operator fails to start up. `gpu-operator-resources` errors.

[khteh]

clientVersion:
  buildDate: "2024-10-16T15:15:29Z"
  compiler: gc
  gitCommit: cbb86e0d7f4a049666fac0551e8b02ef3d6c3d9a
  gitTreeState: clean
  gitVersion: v1.27.16
  goVersion: go1.22.5
  major: "1"
  minor: "27"
  platform: linux/amd64
kustomizeVersion: v5.0.1
serverVersion:
  buildDate: "2024-10-16T15:16:32Z"
  compiler: gc
  gitCommit: cbb86e0d7f4a049666fac0551e8b02ef3d6c3d9a
  gitTreeState: clean
  gitVersion: v1.27.16
  goVersion: go1.22.5
  major: "1"
  minor: "27"
  platform: linux/amd64

Summary

gpu-operator-node-feature-discovery-worker-lchx9              1/1     Running                 0             2m19s
gpu-operator-76998cd846-cwfrv                                 1/1     Running                 0             2m19s
gpu-operator-node-feature-discovery-master-6fbc745786-l828r   1/1     Running                 0             2m19s
nvidia-operator-validator-hhgp7                               0/1     Init:CrashLoopBackOff   4 (34s ago)   119s
nvidia-dcgm-exporter-zsjdz                                    0/1     Init:CrashLoopBackOff   4 (29s ago)   118s
nvidia-container-toolkit-daemonset-rxrx9                      0/1     Init:CrashLoopBackOff   4 (29s ago)   119s
gpu-feature-discovery-s7lsd                                   0/1     Init:CrashLoopBackOff   4 (19s ago)   118s
nvidia-device-plugin-daemonset-qz9rq                          0/1     Init:CrashLoopBackOff   4 (19s ago)   119s

  Warning  Failed     3m5s (x4 over 3m45s)  kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli.real: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown
  Warning  BackOff         3m5s (x5 over 3m44s)  kubelet  Back-off restarting failed container toolkit-validation in pod nvidia-device-plugin-daemonset-mcn5z_gpu-operator-resources(73d22052-9175-4d74-9cbb-71367483b3f3)
  Warning  FailedMount     2m11s                 kubelet  MountVolume.SetUp failed for volume "run-nvidia" : hostPath type check failed: /run/nvidia is not a directory
  Normal   SandboxChanged  2m5s                  kubelet  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled          29s (x4 over 2m2s)    kubelet  Container image "nvcr.io/nvidia/cloud-native/gpu-operator-validator:v22.9.1" already present on machine
  Normal   Created         29s (x4 over 2m2s)    kubelet  Created container toolkit-validation
  Warning  Failed          29s (x4 over 2m1s)    kubelet  Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli.real: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown
  Warning  BackOff  6s (x12 over 2m1s)  kubelet  Back-off restarting failed container toolkit-validation in pod nvidia-device-plugin-daemonset-mcn5z_gpu-operator-resources(73d22052-9175-4d74-9cbb-71367483b3f3)

What Should Happen Instead?

No GPU operator resources error

Reproduction Steps

1. ...
2. ...

Introspection Report

inspection-report-20250923_152230.tar.gz

Can you suggest a fix?

Are you interested in contributing with a fix?

no

[khteh]

Hello!!! Any update on this!?!

[elezar]

Could you please provide the following:

• What is your microk8s configuration?
• How the NVIDIA GPU Drivers are installed on your system?
• Which steps are you following to install the GPU Operator?

[khteh]

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <secret>
    server: https://127.0.0.1:16443
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    token: <secret>

$ microk8s enable gpu

[khteh]

Any update?

[d-shehu]

I also have an issue with GPU add on. It prompted me to run microk8s enable nvidia instead of gpu. This also resulted in the operator not deploying correctly after a clean install of microk8s.

Status (indefinitely)

microk8s kubectl logs -n gpu-operator-resources -lapp=nvidia-operator-validator -c nvidia-operator-validator
Error from server (BadRequest): container "nvidia-operator-validator" in pod "nvidia-operator-validator-gslpn" is waiting to start: PodInitializing

Status of operators (indefinitely):

gpu-operator-resources   gpu-feature-discovery-stwkz                                   0/1     Init:0/1   0          2m15s
gpu-operator-resources   gpu-operator-6d5f7877bb-74bqm                                 1/1     Running    0          2m26s
gpu-operator-resources   gpu-operator-node-feature-discovery-gc-664cfcb6d9-hxrph       1/1     Running    0          2m26s
gpu-operator-resources   gpu-operator-node-feature-discovery-master-59755fc94f-wz6x9   1/1     Running    0          2m26s
gpu-operator-resources   gpu-operator-node-feature-discovery-worker-gmm5b              1/1     Running    0          2m26s
gpu-operator-resources   nvidia-container-toolkit-daemonset-2l4hq                      1/1     Running    0          2m15s
gpu-operator-resources   nvidia-dcgm-exporter-c74ld                                    0/1     Init:0/1   0          2m15s
gpu-operator-resources   nvidia-device-plugin-daemonset-jrwls                          0/1     Init:0/1   0          2m15s
gpu-operator-resources   nvidia-operator-validator-gslpn                               0/1     Init:0/4   0          2m15s

Status of pod (indefinitely):

Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               3m17s                 default-scheduler  Successfully assigned gpu-operator-resources/gpu-feature-discovery-stwkz to vienna
  Warning  FailedCreatePodSandBox  3m5s (x2 over 3m18s)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox runtime: no runtime for "nvidia" is configured
  Warning  FailedCreatePodSandBox  2m51s                 kubelet            Failed to create pod sandbox: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/snap/microk8s/common/run/containerd.sock: connect: connection refused"

Cuda drivers are working fine on my system and I can run nvidia-smi and various programs in dockerized python environments, Jupyter notebooks, etc. without issues.

[d-shehu]

Version 1.32 worked fine with a clean install of microk8s if I "enable nvidia". If I try 1.34 I get the same error. Likewise if I try to install the Nvidia operator directly.

sudo -E snap install microk8s --classic --channel=1.32/stable

I apologize if I missed this but if there's a place to check which version is stable (by plugin?) that would be a time saver. Thanks.