nvml error: driver/library version mismatch: unknown

[jaipreetnagpal]

failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown

We can observe that the nodes during the initial boot gives the error with the nvidia/k8s-device-plugin. But if we reboot the node the mismatch error disappears and we are able to observe that the nodes are able to register the GPU. what can be the issue here

cli-version: 1.17.8
lib-version: 1.17.8
nvidia-smi output
Failed to initialize NVML: Driver/library version mismatch
NVML library version: 570.195
nvidia/k8s-device-plugin:v0.16.2

[jaipreetnagpal]

@dims @aaronp24 ,Can someone look into this issue ??

[elezar]

Could you describe how the GPU Device Plugin and the NVIDIA GPU Driver are installed?