Leftover containerd-shim-runc-v2 processes

[xhejtman]

Hello,

I noticed, that if (probably) a static pod is restarted (like kube-proxy pod on rke2 agent/server restart), the containerd-shim-runc-v2 process is not terminated and is left running.

Also it is filling the kubelet logs with messages like:

I0608 00:49:17.851008   38441 fs.go:465] unable to determine file system type, partition mountpoint does not exist: /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/c60cd0d54ab280216f286cd4051f4feb371d3e3134aa47262bf9789d0f9f04db/rootfs
I0608 00:49:17.851985   38441 fs.go:465] unable to determine file system type, partition mountpoint does not exist: /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/c60cd0d54ab280216f286cd4051f4feb371d3e3134aa47262bf9789d0f9f04db/shm
I0608 00:49:27.937706   38441 fs.go:465] unable to determine file system type, partition mountpoint does not exist: /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/c60cd0d54ab280216f286cd4051f4feb371d3e3134aa47262bf9789d0f9f04db/shm
I0608 00:49:27.940669   38441 fs.go:465] unable to determine file system type, partition mountpoint does not exist: /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/c60cd0d54ab280216f286cd4051f4feb371d3e3134aa47262bf9789d0f9f04db/rootfs

It does not seem to be happening for normal pods.

ps faxww can output (the output is not filtered, those shims do not have any child processes):

 724227 ?        Sl     3:56 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace k8s.io -id 32946168876b68c9866bf00fae6dde76b65cd89d77a45206aac89b936e72df9b -address /run/k3s/containerd/containerd.sock
1731109 ?        Sl     2:22 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace k8s.io -id d4687985cba2862519644a47bfd4bf20e45e8f1c15e6c866865cea3ab18fce6d -address /run/k3s/containerd/containerd.sock
1749227 ?        Sl     2:49 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace k8s.io -id 0579ad16e5ae85fc8b871423161b64b7aaad70e57e1a160df20fa7c5c2e85479 -address /run/k3s/containerd/containerd.sock
1777900 ?        Sl     2:05 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace k8s.io -id 6012c098c61345fa29a609aae68b804254cdfb92225428ea433bf4126325c30a -address /run/k3s/containerd/containerd.sock
1791751 ?        Sl     2:17 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5d9293fff9a7268e76b68948beae06e0a518fa0a102bea99675f063ef59b99b1 -address /run/k3s/containerd/containerd.sock
1864021 ?        Sl     3:57 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace k8s.io -id 321538b52d926f84d0c64446c31a6768fa67098464de05a178df61f033e6273e -address /run/k3s/containerd/containerd.sock

I can see it on 1.31.7 and 1.32.5 versions. Is it something known like containerd/containerd#5708?

[brandond]

It is normal for the shims to continue running when RKE2 is stopped or restarted. The shims are how containerd resumes managing the containers across restarts.

The "mountpoint does not exist" errors seem unrelated.

Are there any specific steps you can provide to reproduce this, if it is an error?

[xhejtman]

It seems, that if you e.g., upgrade rke2 verstion, like from 1.32.4 to 1.32.5, you will see leftover shims still running after kube-api, kube-proxy, and other pods are re-created. It did not happen with older containerd version (I believe 1.7, and it happens with 2.0).