Can't make it work on kubernetes #13851
-
|
I have kubernetes with two nodes: control plane (doesn't have GPU) and worker node with 4 tesla p100. When I go directly to worker node and just submit my job directly on server it runs fine and uses GPU. The configuration below: However when I try to submit the job via control plane node: I am getting: I literally tried every possible version to build my container still getting the same error. Here is an example of my Dockerfile: My worker node: Also I can see that GPU is exposed to the cluster: Could you advise me what's the issue here and how to debug what exactly mismatch? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 7 replies
-
|
The issue is caused by a CUDA version mismatch between your Kubernetes executor container, the Spark RAPIDS plugin, and the GPU driver on your node. Your worker node reports:
Your container is built from:
But the RAPIDS plugin you use:
When running directly on the node, Spark loads the host CUDA libraries and everything works. The real fix is to rebuild the Spark executor image using a CUDA 11.8 base image, which matches the RAPIDS 23.10 release: Rebuild your image, redeploy the executors, and the GPU runtime will match RAPIDS and the host driver, resolving the error. |
Beta Was this translation helpful? Give feedback.
-
|
If your original container already used CUDA 11.8 and you still saw the same error, then the issue isn’t the base image — it’s which CUDA libraries Spark + RAPIDS actually load at runtime. Check the following: • Confirm the pod truly uses CUDA 11.8 and not a hidden CUDA 12.x layer: • Ensure the Kubernetes device plugin correctly mounts host driver libraries: • Verify no Spark configs are forcing CUDA 12.x: • RAPIDS must see a consistent major CUDA version across: Next step: inspect the executor pod’s runtime environment. Most issues come from an unexpected CUDA library shadowing the correct one. |
Beta Was this translation helpful? Give feedback.
-
|
This is my output within the container: Feels like that's where my problem is: Could you direct me how to configure it correctly? |
Beta Was this translation helpful? Give feedback.
-
|
Maybe you can provide a direction on how to: |
Beta Was this translation helpful? Give feedback.
-
|
You're very close — the root cause is now visible from your diagnostics. Even though the node has driver 580.105.08 (CUDA 13.0), the executor pod fails to load This means the device plugin is not correctly passing GPUs / driver libraries / CDI spec to your pods. What you need to fixBelow is a checklist of what to adjust. 1. Ensure your container is not overriding host CUDA driversYour container currently contains: /usr/local/cuda-11.8This is fine only if you allow the device plugin to mount the host driver libs: Expected mounts in the executor pod: /dev/nvidia0
/dev/nvidiactl
/dev/nvidia-uvm
/usr/lib/x86_64-linux-gnu/libcuda.so.X.Y <-- host driver
/usr/lib/x86_64-linux-gnu/libnvidia-ml.soIf those are missing, the container falls back to its own runtime libraries → mismatch → error. 2. Verify the newer NVIDIA Kubernetes stack (driver + toolkit + plugin)Inside your node, confirm: nvidia-ctk runtime configure --runtime=containerd
systemctl restart containerdThen reinstall the k8s device plugin (correct version for 580 driver): # deploy the daemonset
kubectl apply -f \
https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.15.0/nvidia-device-plugin.ymlVersion v0.15.0 is required for CDI-based injection, which your cluster is trying to use but failing. Your logs show exactly this failure. 3. Confirm CDI is enabled on the nodeCheck: nvidia-ctk cdi listYou must see CDI device definitions like: runtime.nvidia.com/gpu=0
runtime.nvidia.com/gpu=1If empty → the device plugin cannot inject drivers → your Spark pod gets no GPU. Generate CDI: sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
sudo systemctl restart containerd4. Validate inside a test podRun: kubectl run cuda-test --rm -it \
--image=nvidia/cuda:11.8.0-base-ubuntu20.04 \
--overrides='{"spec":{"runtimeClassName":"nvidia"}}' \
-- bashInside: nvidia-smi
ldconfig -p | grep libcudaIf this fails, Spark will fail too. Final directionYou do not have a CUDA mismatch anymore — you have a driver injection issue caused by a broken/old NVIDIA device plugin. Once you:
→ your CUDA runtime errors will disappear. |
Beta Was this translation helpful? Give feedback.
-
|
At the end I have reinstalled plugin with this config and everything seem to work fine. Thanks for your help. |
Beta Was this translation helpful? Give feedback.
-
|
Great news 👍🏻 |
Beta Was this translation helpful? Give feedback.
You're very close — the root cause is now visible from your diagnostics.
Your container correctly has CUDA 11.8, but inside Kubernetes your Spark executor is not seeing the host's NVIDIA driver libraries, leading to:
Even though the node has driver 580.105.08 (CUDA 13.0), the executor pod fails to load
libcuda.sofrom the host.The error in your
nvidia-device-pluginpod confirms this:This means the device plugin is not correctly passing GPUs / driver libraries / CDI spec to your pods.
What you need to fix
Below is a checklist of what to adjust.
1. Ensure your contai…