nvidia-cuda-validator Pod in nvidia-gpu-operator Namespace Stuck in Init:CrashLoopBackOff on OpenShift 4.18.22

[KothaHariChandana2000]

Important Note: NVIDIA AI Enterprise customers can get support from NVIDIA Enterprise support. Please open a case here.

Describe the bug
nvidia-cuda-validator pod in nvidia-gpu-operator is in Init:CrashLoopBackOff

% oc logs nvidia-cuda-validator-4bnbm -c cuda-validation -n nvidia-gpu-operator
Failed to allocate device vector A (error code initialization error)!
[Vector addition of 50000 elements]

To Reproduce
Install Nvidia GPU Operator 25.10.0 from Operator Hub on Redhat openshift container platform 4.18.22

Expected behavior
Install the NVIDIA GPU Operator v25.10.0 Successfully, with all pods in the nvidia-gpu-operator namespace in a Running state.

Environment (please provide the following information):
Bare metal OpenShift cluster 4.18.22
1 GPU nodes with L40S GPU cards
Node Feature Discovery operator installed 4.18.0
Nvidia gpu Operator installed 25.10.0

Information to attach (optional if deemed irrelevant)

• kubernetes pods status: kubectl get pods -n OPERATOR_NAMESPACE
• kubernetes daemonset status: kubectl get ds -n OPERATOR_NAMESPACE
• If a pod/ds is in an error state or pending state kubectl describe pod -n OPERATOR_NAMESPACE POD_NAME
• If a pod/ds is in an error state or pending state kubectl logs -n OPERATOR_NAMESPACE POD_NAME --all-containers
• Output from running nvidia-smi from the driver container: kubectl exec DRIVER_POD_NAME -n OPERATOR_NAMESPACE -c nvidia-driver-ctr -- nvidia-smi
• containerd logs journalctl -u containerd > containerd.log

Collecting full debug bundle (optional):

curl -o must-gather.sh -L https://raw.githubusercontent.com/NVIDIA/gpu-operator/main/hack/must-gather.sh
chmod +x must-gather.sh
./must-gather.sh

NOTE: please refer to the must-gather script for debug data collected.

This bundle can be submitted to us via email: operator_feedback@nvidia.com

[KothaHariChandana2000]

% oc get pods -n nvidia-gpu-operator
NAME                                                  READY   STATUS       RESTARTS        AGE
gpu-feature-discovery-6jhlr                           1/1     Running      0               5h48m
gpu-operator-5d7457b677-7ddw5                         1/1     Running      0               5h49m
nvidia-container-toolkit-daemonset-bbgqn              1/1     Running      0               5h48m
nvidia-cuda-validator-wgs28                           0/1     Init:Error   3 (39s ago)     56s
nvidia-dcgm-8f4p9                                     1/1     Running      0               5h48m
nvidia-dcgm-exporter-s8bfk                            1/1     Running      2 (5h46m ago)   5h48m
nvidia-device-plugin-daemonset-qd9sd                  1/1     Running      0               5h48m
nvidia-driver-daemonset-418.94.202508060022-0-hs6rw   2/2     Running      0               5h48m
nvidia-node-status-exporter-rjvg5                     1/1     Running      0               5h48m
nvidia-operator-validator-99l6b                       0/1     Init:2/4     41 (104s ago)   4h37m

% oc get ds -n  nvidia-gpu-operator
NAME                                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                                                                         AGE
gpu-feature-discovery                           1         1         1       1            1           nvidia.com/gpu.deploy.gpu-feature-discovery=true                                                                      5h48m
nvidia-container-toolkit-daemonset              1         1         1       1            1           nvidia.com/gpu.deploy.container-toolkit=true                                                                          5h48m
nvidia-dcgm                                     1         1         1       1            1           nvidia.com/gpu.deploy.dcgm=true                                                                                       5h48m
nvidia-dcgm-exporter                            1         1         1       1            1           nvidia.com/gpu.deploy.dcgm-exporter=true                                                                              5h48m
nvidia-device-plugin-daemonset                  1         1         1       1            1           nvidia.com/gpu.deploy.device-plugin=true                                                                              5h48m
nvidia-device-plugin-mps-control-daemon         0         0         0       0            0           nvidia.com/gpu.deploy.device-plugin=true,nvidia.com/mps.capable=true                                                  5h48m
nvidia-driver-daemonset-418.94.202508060022-0   1         1         1       1            1           feature.node.kubernetes.io/system-os_release.OSTREE_VERSION=418.94.202508060022-0,nvidia.com/gpu.deploy.driver=true   5h48m
nvidia-mig-manager                              0         0         0       0            0           nvidia.com/gpu.deploy.mig-manager=true                                                                                5h48m
nvidia-node-status-exporter                     1         1         1       1            1           nvidia.com/gpu.deploy.node-status-exporter=true                                                                       5h48m
nvidia-operator-validator                       1         1         0       1            0           nvidia.com/gpu.deploy.operator-validator=true                                                                         5h48m

% oc exec nvidia-driver-daemonset-418.94.202508060022-0-hs6rw -c nvidia-driver-ctr -- nvidia-smi
Fri Oct 31 11:40:38 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.82.07              Driver Version: 580.82.07      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L40S                    On  |   00000000:03:00.0 Off |                    0 |
| N/A   29C    P8             35W /  350W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA L40S                    On  |   00000000:04:00.0 Off |                    0 |
| N/A   28C    P8             34W /  350W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA L40S                    On  |   00000000:63:00.0 Off |                    0 |
| N/A   29C    P8             36W /  350W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA L40S                    On  |   00000000:64:00.0 Off |                    0 |
| N/A   29C    P8             35W /  350W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA L40S                    On  |   00000000:83:00.0 Off |                    0 |
| N/A   27C    P8             35W /  350W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA L40S                    On  |   00000000:84:00.0 Off |                    0 |
| N/A   26C    P8             35W /  350W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA L40S                    On  |   00000000:E3:00.0 Off |                    0 |
| N/A   27C    P8             35W /  350W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA L40S                    On  |   00000000:E4:00.0 Off |                    0 |
| N/A   26C    P8             35W /  350W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

[KothaHariChandana2000]

Must- gather logs:

nvidia_gpu_operator_logs.zip

[rahulait]

Hi @KothaHariChandana2000 , thank you for providing the logs. Can you try disabling kaslr? So something like:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 100-master-nokaslr
spec:
  config:
    ignition:
      version: 3.2.0
  kernelArguments:
      - nokaslr

Make sure you target the right machine master/worker to disable kaslr for that machine. Once disabled, you can check cat /proc/cmdline to see if kaslr is now disabled in it or not and see if that helps. Let us know how things go.

[KothaHariChandana2000]

Hi @rahulait ,

The suggested workaround worked — after applying the MachineConfig with nokaslr, all the GPU Operator pods are now in a Running state.
Could you please clarify what the implications would be if we document this workaround for production customers?
Also, is there any plan to address this issue in upcoming releases so that disabling KASLR isn’t required as a workaround?

Thank you for your quick response and guidance!

[rahulait]

KASLR is a linux security feature for KVM virtual machines which randomizes the memory address when kernel image is decompressed, and thus prevents guest security exploits to some extent. It seems like it is having issues in the kernel version which comes by default with openshift. Hopefully, as newer kernel fixes are released, there won't be need to disable kaslr.

Another alternative which can be tried is setting kernelModuleType: proprietary in helm values here when installing gpu-operator.

[KothaHariChandana2000]

@rahulait I've tried setting kernelModuleType: proprietary
GPU Driver version: 580.95.05
But I see the below logs in nvidia-operator-validator pod

Must-gather logs:

nvidia-gpu-operator_20251102_2127.tar.gz

[rahulait]

Re: kernelModuleType: proprietary I see all pods coming up fine in the logs. The validator pod logs might just be a red herring. Are you seeing issues in deploying workloads with gpu after that?

Re: kaslr: This is the kernel fix which was committed upstream: https://lkml.org/lkml/2025/2/22/327
Once redhat picks it and integrates it, you won't have to disable kaslr anymore. Since this is more kernel specific which is installed, you'll need to check with the distribution to see when they can get that fix in.

[KothaHariChandana2000]

I’ve deployed a sample GPU workload following the steps from the NVIDIA documentation:
https://docs.nvidia.com/datacenter/cloud-native/openshift/latest/install-gpu-ocp.html#running-a-sample-gpu-application

The sample application ran successfully, and I can see the following logs:

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

However, on one of our service pods running on the GPU node, I’m seeing the following error:

Error: container create failed: error executing hook `/usr/local/nvidia/toolkit/nvidia-cdi-hook` (exit code: 1)
error executing hook `/usr/local/nvidia/toolkit/nvidia-cdi-hook` (exit code: 1)

[rahulait]

Can you share the service pod definition which is getting applied?

[KothaHariChandana2000]

Here is the deployment and the pod
deployment-granite-3-2b-instruct-predictor-deployment.yaml
deployment-granite-3-2b-instruct-predictor-pod.yaml

This is the event