mps-control-daemon is restarting with GPU drivers installed through GKE daemonset

[kasia-kujawa]

mps-control-daemon is restarting when GPU drivers are installed through GKE daemonset, tested with example Pod from https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/main/demo/specs/quickstart/gpu-test-mps.yaml

Kuberentes version: 1.33.2-gke.1240000
Image type: UBUNTU_CONTAINERD

logs:

$ kubectl logs -n nvidia mps-control-daemon-d8478efa-5c09-461d-b7b9-f59f320396a8-5cv5nr8  -p
chroot: failed to run command 'sh': No such file or directory

GPU driver installation:

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/refs/heads/master/nvidia-driver-installer/ubuntu/daemonset-preloaded.yaml

DRA helm chart installation:

cat <<EOF > dra_values.yaml
resources.gpus.enabled: "true"
gpuResourcesEnabledOverride: "true"
nvidiaDriverRoot: /opt/nvidia

controller:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: "nvidia.com/gpu.present"
                operator: "Exists"
kubeletPlugin:
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  nodeSelector:
    nvidia.com/gpu.present: "true"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: "nvidia.com/gpu.present"
                operator: "Exists"
EOF

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
    && helm repo update

helm upgrade -i nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu --version="25.3.0-rc.4" \
  --namespace nvidia \
  -f dra_values.yaml

[klueska]

What image do you see in nvidia mps-control-daemon-d8478efa-5c09-461d-b7b9-f59f320396a8-5cv5nr8

[kasia-kujawa]

What image do you see in nvidia mps-control-daemon-d8478efa-5c09-461d-b7b9-f59f320396a8-5cv5nr8

nvcr.io/nvidia/k8s-dra-driver-gpu:v25.3.0-rc.4

full pod:

apiVersion: v1
kind: Pod
metadata:
  name: mps-control-daemon-13254a39-d421-4f12-8562-92b607b4ff04-34jc2df
  generateName: mps-control-daemon-13254a39-d421-4f12-8562-92b607b4ff04-3453d-55dcd747cc-
  namespace: nvidia
  uid: aaf571fb-3805-4cbe-ad94-b41348bf1c03
  resourceVersion: '1755546850227775008'
  generation: 1
  creationTimestamp: '2025-08-18T19:53:19Z'
  labels:
    app: mps-control-daemon-13254a39-d421-4f12-8562-92b607b4ff04-3453d
    pod-template-hash: 55dcd747cc
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: mps-control-daemon-13254a39-d421-4f12-8562-92b607b4ff04-3453d-55dcd747cc
      uid: 4629953e-fb1d-49ea-bde5-9a6ae6592ac6
      controller: true
      blockOwnerDeletion: true
  managedFields:
    - manager: kube-controller-manager
      operation: Update
      apiVersion: v1
      time: '2025-08-18T19:53:19Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:generateName: {}
          f:labels:
            .: {}
            f:app: {}
            f:pod-template-hash: {}
          f:ownerReferences:
            .: {}
            k:{"uid":"4629953e-fb1d-49ea-bde5-9a6ae6592ac6"}: {}
        f:spec:
          f:containers:
            k:{"name":"mps-control-daemon"}:
              .: {}
              f:args: {}
              f:command: {}
              f:env:
                .: {}
                k:{"name":"CUDA_VISIBLE_DEVICES"}:
                  .: {}
                  f:name: {}
                  f:value: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:name: {}
              f:resources: {}
              f:securityContext:
                .: {}
                f:privileged: {}
              f:startupProbe:
                .: {}
                f:exec:
                  .: {}
                  f:command: {}
                f:failureThreshold: {}
                f:initialDelaySeconds: {}
                f:periodSeconds: {}
                f:successThreshold: {}
                f:timeoutSeconds: {}
              f:terminationMessagePath: {}
              f:terminationMessagePolicy: {}
              f:volumeMounts:
                .: {}
                k:{"mountPath":"/driver-root"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/driver-root/dev/shm"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/driver-root/tmp/nvidia-mps"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/driver-root/var/log/nvidia-mps"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
          f:dnsPolicy: {}
          f:enableServiceLinks: {}
          f:hostPID: {}
          f:nodeName: {}
          f:restartPolicy: {}
          f:schedulerName: {}
          f:securityContext: {}
          f:terminationGracePeriodSeconds: {}
          f:volumes:
            .: {}
            k:{"name":"driver-root"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
            k:{"name":"mps-log-directory"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
            k:{"name":"mps-pipe-directory"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
            k:{"name":"mps-shm-directory"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
    - manager: kubelet
      operation: Update
      apiVersion: v1
      time: '2025-08-18T19:54:10Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:conditions:
            .: {}
            k:{"type":"ContainersReady"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Initialized"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"PodReadyToStartContainers"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"PodScheduled"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"Ready"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
          f:containerStatuses: {}
          f:hostIP: {}
          f:hostIPs: {}
          f:phase: {}
          f:podIP: {}
          f:podIPs:
            .: {}
            k:{"ip":"10.84.3.8"}:
              .: {}
              f:ip: {}
          f:startTime: {}
      subresource: status
  selfLink: >-
    /api/v1/namespaces/nvidia/pods/mps-control-daemon-13254a39-d421-4f12-8562-92b607b4ff04-34jc2df
status:
  phase: Running
  conditions:
    - type: PodReadyToStartContainers
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2025-08-18T19:53:21Z'
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2025-08-18T19:53:19Z'
    - type: Ready
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2025-08-18T19:53:19Z'
      reason: ContainersNotReady
      message: 'containers with unready status: [mps-control-daemon]'
    - type: ContainersReady
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2025-08-18T19:53:19Z'
      reason: ContainersNotReady
      message: 'containers with unready status: [mps-control-daemon]'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2025-08-18T19:53:19Z'
  hostIP: 10.142.0.30
  hostIPs:
    - ip: 10.142.0.30
  podIP: 10.84.3.8
  podIPs:
    - ip: 10.84.3.8
  startTime: '2025-08-18T19:53:19Z'
  containerStatuses:
    - name: mps-control-daemon
      state:
        waiting:
          reason: CrashLoopBackOff
          message: >-
            back-off 40s restarting failed container=mps-control-daemon
            pod=mps-control-daemon-13254a39-d421-4f12-8562-92b607b4ff04-34jc2df_nvidia(aaf571fb-3805-4cbe-ad94-b41348bf1c03)
      lastState:
        terminated:
          exitCode: 127
          reason: Error
          startedAt: '2025-08-18T19:54:08Z'
          finishedAt: '2025-08-18T19:54:08Z'
          containerID: >-
            containerd://8977c2e755432a89e35dddeba700a92f3d0ac1526d30272613fde7644ea9225c
      ready: false
      restartCount: 3
      image: nvcr.io/nvidia/k8s-dra-driver-gpu:v25.3.0-rc.4
      imageID: >-
        nvcr.io/nvidia/k8s-dra-driver-gpu@sha256:5dd583277c1f2825cb637c3c07d8208c6278b1e6ccb4231f0ac011dbf651d5a9
      containerID: >-
        containerd://8977c2e755432a89e35dddeba700a92f3d0ac1526d30272613fde7644ea9225c
      started: false
      resources: {}
      volumeMounts:
        - name: driver-root
          mountPath: /driver-root
        - name: mps-shm-directory
          mountPath: /driver-root/dev/shm
        - name: mps-pipe-directory
          mountPath: /driver-root/tmp/nvidia-mps
        - name: mps-log-directory
          mountPath: /driver-root/var/log/nvidia-mps
        - name: kube-api-access-h7qlh
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
          recursiveReadOnly: Disabled
      user:
        linux:
          uid: 0
          gid: 0
          supplementalGroups:
            - 0
  qosClass: BestEffort
spec:
  volumes:
    - name: driver-root
      hostPath:
        path: /opt/nvidia
        type: ''
    - name: mps-shm-directory
      hostPath:
        path: >-
          /var/lib/kubelet/plugins/gpu.nvidia.com/mps/13254a39-d421-4f12-8562-92b607b4ff04-3453d/shm
        type: ''
    - name: mps-pipe-directory
      hostPath:
        path: >-
          /var/lib/kubelet/plugins/gpu.nvidia.com/mps/13254a39-d421-4f12-8562-92b607b4ff04-3453d/pipe
        type: ''
    - name: mps-log-directory
      hostPath:
        path: >-
          /var/lib/kubelet/plugins/gpu.nvidia.com/mps/13254a39-d421-4f12-8562-92b607b4ff04-3453d/log
        type: ''
    - name: kube-api-access-h7qlh
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: mps-control-daemon
      image: nvcr.io/nvidia/k8s-dra-driver-gpu:v25.3.0-rc.4
      command:
        - chroot
        - /driver-root
        - sh
        - '-c'
      args:
        - >-
          set -e

          rm -f /var/log/nvidia-mps/startup.log


          nvidia-cuda-mps-control -d

          echo set_default_active_thread_percentage 50 | nvidia-cuda-mps-control

          echo set_default_device_pinned_mem_limit
          GPU-ab966f59-814f-63b9-35b2-b44c17c104d5 10240M |
          nvidia-cuda-mps-control


          echo "startup complete" > /var/log/nvidia-mps/startup.log


          tail -n +1 -f /var/log/nvidia-mps/control.log
      env:
        - name: CUDA_VISIBLE_DEVICES
          value: GPU-ab966f59-814f-63b9-35b2-b44c17c104d5
      resources: {}
      volumeMounts:
        - name: driver-root
          mountPath: /driver-root
        - name: mps-shm-directory
          mountPath: /driver-root/dev/shm
        - name: mps-pipe-directory
          mountPath: /driver-root/tmp/nvidia-mps
        - name: mps-log-directory
          mountPath: /driver-root/var/log/nvidia-mps
        - name: kube-api-access-h7qlh
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      startupProbe:
        exec:
          command:
            - cat
            - /driver-root/var/log/nvidia-mps/startup.log
        initialDelaySeconds: 1
        timeoutSeconds: 1
        periodSeconds: 1
        successThreshold: 1
        failureThreshold: 3
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        privileged: true
  restartPolicy: Always
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: default
  serviceAccount: default
  nodeName: gke-kasiak-08-18-a-gpu-pool-1-92ce6a89-3dj0
  hostPID: true
  securityContext: {}
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority

[klueska]

Ah yes, of course, since on GKE the driver mount is not a full file-sytstem, chroot will not work in this environment.

[asymingt]

@kasia-kujawa you might want to see if the small fix I made for Talos also happens to work for GKE. It basically just moves the chroot to wrap the nvidia-cuda-mps-control binary in stead of sh: hydazz#1