Create Container Fails: Auto-detected mode as 'legacy' nvidia-container-cli: ldcache error: process /sbin/ldconfig terminated with signal 9

[reefland]

Bare metal K3S v1.33.3+k3s1 on kernel 6.15.11-2-MANJARO.

Not a new install; this had been stable for many months. Rebooted node with the GPU, now POD crash loops with message:

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy' nvidia-container-cli: ldcache error: process /sbin/ldconfig terminated with signal 9

I'm confused by the OCI, Legacy and ldcache references.

Chart reference in ArgoCD:

- repoURL: https://nvidia.github.io/k8s-device-plugin
  chart: nvidia-device-plugin
  targetRevision: 0.17.3

Helm Values File:

---
# yaml-language-server: $schema=https://json.schemastore.org/helmfile

nodeSelector:
  nvidia.feature.node.kubernetes.io/gpu.3060: "true"

runtimeClassName: nvidia

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: nvidia.feature.node.kubernetes.io/gpu.3060
              operator: In
              values:
                - "true"

config:
  map:
    default: |-
      version: v1
      flags:
        migStrategy: none
      sharing:
        timeSlicing:
          renameByDefault: false
          failRequestsGreaterThanOne: false
          resources:
            - name: nvidia.com/gpu
              replicas: 6

# Subcharts
nfd: {}
gfd:
  enabled: false

• NFD is already installed via its own Helm Chart.

Current versions on host:

$ pacman -Q libnvidia-container
libnvidia-container 1.17.8-1

$ pacman -Q nvidia-container-toolkit
nvidia-container-toolkit 1.17.8-1

$ pacman -Q nvidia-utils
nvidia-utils 575.64.05-1

$ nvidia-smi
Tue Sep  2 15:33:10 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.64.05              Driver Version: 575.64.05      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060        Off |   00000000:09:00.0  On |                  N/A |
|  0%   54C    P3             30W /  170W |    1665MiB /  12288MiB |     37%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

$ nvidia-container-cli info
NVRM version:   575.64.05
CUDA version:   12.9

Device Index:   0
Device Minor:   0
Model:          NVIDIA GeForce RTX 3060
Brand:          GeForce
GPU UUID:       GPU-ace6a26d-6a78-9562-4fbc-69984c397347
Bus Location:   00000000:09:00.0
Architecture:   8.6

$ nvidia-container-cli list
/dev/nvidiactl
/dev/nvidia-uvm
/dev/nvidia-uvm-tools
/dev/nvidia-modeset
/dev/nvidia0
/usr/bin/nvidia-smi
/usr/bin/nvidia-debugdump
/usr/bin/nvidia-persistenced
/usr/bin/nvidia-cuda-mps-control
/usr/bin/nvidia-cuda-mps-server
/usr/lib/libnvidia-ml.so.575.64.05
/usr/lib/libnvidia-cfg.so.575.64.05
/usr/lib/libcuda.so.575.64.05
/usr/lib/libcudadebugger.so.575.64.05
/usr/lib/libnvidia-gpucomp.so.575.64.05
/usr/lib/libnvidia-ptxjitcompiler.so.575.64.05
/usr/lib/libnvidia-allocator.so.575.64.05
/usr/lib/libnvidia-pkcs11.so.575.64.05
/usr/lib/libnvidia-pkcs11-openssl3.so.575.64.05
/usr/lib/libnvidia-nvvm.so.575.64.05
/usr/lib/libnvidia-ngx.so.575.64.05
/usr/lib/libnvidia-encode.so.575.64.05
/usr/lib/libnvidia-opticalflow.so.575.64.05
/usr/lib/libnvcuvid.so.575.64.05
/usr/lib/libnvidia-eglcore.so.575.64.05
/usr/lib/libnvidia-glcore.so.575.64.05
/usr/lib/libnvidia-tls.so.575.64.05
/usr/lib/libnvidia-glsi.so.575.64.05
/usr/lib/libnvidia-fbc.so.575.64.05
/usr/lib/libnvidia-rtcore.so.575.64.05
/usr/lib/libnvoptix.so.575.64.05
/usr/lib/libGLX_nvidia.so.575.64.05
/usr/lib/libEGL_nvidia.so.575.64.05
/usr/lib/libGLESv2_nvidia.so.575.64.05
/usr/lib/libGLESv1_CM_nvidia.so.575.64.05
/usr/lib/libnvidia-glvkspirv.so.575.64.05
/usr/lib32/libnvidia-ml.so.575.64.05
/usr/lib32/libcuda.so.575.64.05
/usr/lib32/libnvidia-gpucomp.so.575.64.05
/usr/lib32/libnvidia-ptxjitcompiler.so.575.64.05
/usr/lib32/libnvidia-allocator.so.575.64.05
/usr/lib32/libnvidia-encode.so.575.64.05
/usr/lib32/libnvidia-opticalflow.so.575.64.05
/usr/lib32/libnvcuvid.so.575.64.05
/usr/lib32/libnvidia-eglcore.so.575.64.05
/usr/lib32/libnvidia-glcore.so.575.64.05
/usr/lib32/libnvidia-tls.so.575.64.05
/usr/lib32/libnvidia-glsi.so.575.64.05
/usr/lib32/libnvidia-fbc.so.575.64.05
/usr/lib32/libGLX_nvidia.so.575.64.05
/usr/lib32/libEGL_nvidia.so.575.64.05
/usr/lib32/libGLESv2_nvidia.so.575.64.05
/usr/lib32/libGLESv1_CM_nvidia.so.575.64.05
/usr/lib32/libnvidia-glvkspirv.so.575.64.05
/lib/firmware/nvidia/575.64.05/gsp_ga10x.bin
/lib/firmware/nvidia/575.64.05/gsp_tu10x.bin

From K3S config:

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia']
  runtime_type = "io.containerd.runc.v2"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia'.options]
  BinaryName = "/usr/bin/nvidia-container-runtime"
  SystemdCgroup = true

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia-cdi']
  runtime_type = "io.containerd.runc.v2"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia-cdi'.options]
  BinaryName = "/usr/bin/nvidia-container-runtime.cdi"
  SystemdCgroup = true

$ k get all -n nvidia
NAME                             READY   STATUS                  RESTARTS       AGE
pod/nvidia-device-plugin-268bb   0/2     Init:CrashLoopBackOff   16 (50s ago)   60m

NAME                                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                                 AGE
daemonset.apps/nvidia-device-plugin                      1         1         0       1            0           nvidia.feature.node.kubernetes.io/gpu.3060=true                               60m
daemonset.apps/nvidia-device-plugin-mps-control-daemon   0         0         0       0            0           nvidia.com/mps.capable=true,nvidia.feature.node.kubernetes.io/gpu.3060=true   60m

Nvidia packages have not been updated recently on the host:

$ ls -ltR /var/cache/pacman/pkg/nvidia*.zst
.rw-r--r-- root root  80 KB Mon Aug  4 11:36:53 2025  /var/cache/pacman/pkg/nvidia-driver-assistant-0.22.65.06-1-any.pkg.tar.zst
.rw-r--r-- root root 334 MB Tue Jul 22 13:48:56 2025  /var/cache/pacman/pkg/nvidia-utils-575.64.05-1-x86_64.pkg.tar.zst
.rw-r--r-- root root 334 MB Tue Jul  1 17:02:35 2025  /var/cache/pacman/pkg/nvidia-utils-575.64.03-1-x86_64.pkg.tar.zst
.rw-r--r-- root root 334 MB Tue Jun 17 14:26:14 2025  /var/cache/pacman/pkg/nvidia-utils-575.64-1-x86_64.pkg.tar.zst
.rw-r--r-- root root  79 KB Tue Jun  3 21:00:32 2025  /var/cache/pacman/pkg/nvidia-driver-assistant-0.21.57.08-1-any.pkg.tar.zst
.rw-r--r-- root root 4.3 MB Sun Jun  1 11:33:12 2025  /var/cache/pacman/pkg/nvidia-container-toolkit-1.17.8-1-x86_64.pkg.tar.zst
.rw-r--r-- root root  79 KB Thu May  1 23:38:55 2025  /var/cache/pacman/pkg/nvidia-driver-assistant-0.21.51.03-1-any.pkg.tar.zst
.rw-r--r-- root root 4.3 MB Sat Apr 26 11:27:21 2025  /var/cache/pacman/pkg/nvidia-container-toolkit-1.17.6-1-x86_64.pkg.tar.zst
.rw-r--r-- root root 4.2 MB Thu Mar 13 11:17:51 2025  /var/cache/pacman/pkg/nvidia-container-toolkit-1.17.5-1-x86_64.pkg.tar.zst

[TheDarkula]

I am running Arch, and the exact same thing is happening to me today after a system upgrade/reboot.
I have a RuntimeClass created, and the chart installed via this command:

helm upgrade -i nvdp nvdp/nvidia-device-plugin --namespace nvidia-device-plugin --create-namespace --version 0.17.3 --set-file config.map.config=config.yaml --set runtimeClassName=nvidia

Running nvidia-smi on the host shows the expected output.

[ballerburg9005]

On Arch my system upgrade was probably a month old, got the error also with Docker. Upgraded system still the same.

ChatGPT first suggested to remove the nvidia block in /etc/docker/daemon.json, then it suggested to snapshot the container and re-run the snapshot so it would not be configured with old runtime. Then it suggested to pacman -R nvidia-container-runtime, claiming it was legacy. None of this resolved the problem.

[TheDarkula]

My intuition was correct, and it is related to the latest driver updates in Arch.

Relevant nvidia-container-toolkit issue.
Relevant Arch Forum post.

This document shows what should work as a fix for most situations, but as I am running k3s, I am fairly certain that they do not apply.

If anyone can deduce how to remedy the situation for k3s, I would greatly appreciate it :)

[GruffyPuffy]

My intuition was correct, and it is related to the latest driver updates in Arch.

Relevant nvidia-container-toolkit issue. Relevant Arch Forum post.

This document shows what should work as a fix for most situations, but as I am running k3s, I am fairly certain that they do not apply.

If anyone can deduce how to remedy the situation for k3s, I would greatly appreciate it :)

@TheDarkula I know it is a long-shot. We have the bug/issue when running incus GPU-containers (using the nvidia-container-toolkit), however incus does not yet support CDI so that workaround is not possible, as it seems with docker containers. We did however find that the problem came when there was a glibc (and lib32-glibc) update a few weeks ago. It might not be the best solution, but it works for us when holding back this one version. Might be worth testing if you feel that you dare to hold back glibc... NVIDIA/nvidia-container-toolkit#1246 (comment)

[reefland]

I didn't downgrade anything, I did the steps I've seen other use and worked for my K3S install:

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
sudo nvidia-ctk config --in-place --set nvidia-container-runtime.mode=cdi && sudo systemctl restart docker

• This node did have docker installed so I included the docker restart (I don't think K3S actually uses it).

Pods restarted and came back online.

$ k get pods -n nvidia

NAME                         READY   STATUS    RESTARTS   AGE
nvidia-dcgm-exporter-2b8wl   1/1     Running   0          60s
nvidia-dcgm-exporter-7cjrk   1/1     Running   0          5d15h
nvidia-device-plugin-66vsv   2/2     Running   0          3m13s
nvidia-device-plugin-qzc6j   2/2     Running   0          5d18h

Showed up in Grafana again.

[TheDarkula]

@GruffyPuffy Thank you for the information!

@reefland I tried the commands you provided, but the helm chart still fails to bring pods up.

Running this:

helm upgrade --install nvdp nvdp/nvidia-device-plugin --namespace nvidia-device-plugin --create-namespace --version 0.17.3 --set-file config.map.config=config.yaml

The pod/container logs show a failure here: kubectl -n nvidia-device-plugin logs nvdp-nvidia-device-plugin-bw7tw -c nvidia-device-plugin-ctr

I0914 17:43:05.845907      62 main.go:235] "Starting NVIDIA Device Plugin" version=<
	e0a461e1
	commit: e0a461e1e7ad1d239d4708c954f08c3038e2654a
 >
I0914 17:43:05.845947      62 main.go:238] Starting FS watcher for /var/lib/kubelet/device-plugins
I0914 17:43:05.845984      62 main.go:245] Starting OS watcher.
I0914 17:43:05.846045      62 main.go:260] Starting Plugins.
I0914 17:43:05.846055      62 main.go:317] Loading configuration.
I0914 17:43:05.846686      62 main.go:342] Updating config with default resource matching patterns.
I0914 17:43:05.846740      62 main.go:353] 
Running with config:
{
  "version": "v1",
  "flags": {
    "migStrategy": "none",
    "failOnInitError": true,
    "mpsRoot": "/run/nvidia/mps",
    "nvidiaDriverRoot": "/",
    "nvidiaDevRoot": "/",
    "gdsEnabled": false,
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": false,
      "deviceListStrategy": [
        "envvar"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/bin/nvidia-ctk",
      "containerDriverRoot": "/driver-root"
    }
  },
  "resources": {
    "gpus": [
      {
        "pattern": "*",
        "name": "nvidia.com/gpu"
      }
    ]
  },
  "sharing": {
    "timeSlicing": {
      "resources": [
        {
          "name": "nvidia.com/gpu",
          "devices": "all",
          "replicas": 10
        }
      ]
    }
  },
  "imex": {}
}
I0914 17:43:05.846747      62 main.go:356] Retrieving plugins.
E0914 17:43:05.846868      62 factory.go:112] Incompatible strategy detected auto
E0914 17:43:05.846875      62 factory.go:113] If this is a GPU node, did you configure the NVIDIA Container Toolkit?
E0914 17:43:05.846880      62 factory.go:114] You can check the prerequisites at: https://github.com/NVIDIA/k8s-device-plugin#prerequisites
E0914 17:43:05.846885      62 factory.go:115] You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start
E0914 17:43:05.846889      62 factory.go:116] If this is not a GPU node, you should set up a toleration or nodeSelector to only deploy this plugin on GPU nodes
E0914 17:43:05.847195      62 main.go:173] error starting plugins: error getting plugins: unable to create plugins: failed to construct resource managers: invalid device discovery strategy

[reefland]

@TheDarkula - The 2nd command I had above sets --set nvidia-container-runtime.mode=cdi mode should now be cdi; yet your error message still says its "auto" (previous default):

E0914 17:43:05.846868 62 factory.go:112] Incompatible strategy detected auto

Your log snip is exactly the same as mine, except for the ending:

I0915 13:50:57.186923     125 main.go:356] Retrieving plugins.
I0915 13:50:57.214579     125 server.go:195] Starting GRPC server for 'nvidia.com/gpu'
I0915 13:50:57.214989     125 server.go:139] Starting to serve 'nvidia.com/gpu' on /var/lib/kubelet/device-plugins/nvidia-gpu.sock
I0915 13:50:57.216355     125 server.go:146] Registered device plugin for 'nvidia.com/gpu' with Kubelet

[TheDarkula]

@reefland I tried copy/pasting your exact code block, but it still has the same error.

[TheDarkula]

I found a solution!

For anyone running k3s, to /etc/rancher/k3s/config.yaml, add:

default-runtime: nvidia

Then run

sudo systemctl restart containerd
sudo systemctl restart k3s