[QUESTION/HELP] Cluster fails to start with error "Failed to run kubelet"

[sayjeyhi]

Question / Where do you need Help?

The rancher server container is restarting continuously.

bb48fa89ebb3 rancher/k3s:v1.20.6-k3s1 "/bin/k3s server --n…" 15 minutes ago Restarting (255) 9 seconds ago k3d-hoory-server-0

and if I get k3d node list I can see the server has a restart loop.

NAME                 ROLE           CLUSTER   STATUS
k3d-cname-registry   registry       cname     running
k3d-cname-server-0   server         cname     restarting
k3d-cname-serverlb   loadbalancer   cname     running

when I get docker container logs bb48fa89ebb3 it gives me this error log:

I0630 12:17:33.589220       7 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
I0630 12:17:33.589254       7 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=false
I0630 12:17:33.623341       7 request.go:668] Waited for 1.04835598s due to client-side throttling, not priority and fairness, request: GET:https://127.0.0.1:6444/apis/certificates.k8s.io/v1beta1?timeout=32s
I0630 12:17:33.790306       7 server.go:777] "Failed to ApplyOOMScoreAdj" err="write /proc/self/oom_score_adj: permission denied"
I0630 12:17:33.790719       7 kubelet.go:404] "Attempting to sync node with API server"
I0630 12:17:33.790792       7 kubelet.go:272] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
I0630 12:17:33.790857       7 kubelet.go:283] "Adding apiserver pod source"
I0630 12:17:33.790912       7 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
E0630 12:17:33.791243       7 server.go:288] "Failed to run kubelet" err="failed to run Kubelet: failed to create kubelet: open /dev/kmsg: no such file or directory"

It was working fine and suddenly today stopped working.

Scope of your Question

• Is your question related to a specific version of k3d (or k3s)?
  I think no, I was on k3d v4.4.2 and updated but it is there
• Please paste the output of k3d version here
  k3d version v4.4.6
  k3s version v1.21.1-k3s1 (default)

docker version: Docker version 20.10.7, build f0df350
os: Linux 5.8.0-59-generic #66~20.04.1-Ubuntu

[sayjeyhi]

if you face this issue, I solved it by:

• keep your Linux up to date
• don't trust docker rootless : |
• if you want to have a fully pruned docker, remove whole the docker from the system and install it again

containers/podman#3024

[iwilltry42]

Thanks @sayjeyhi for asking and directly answering 😁
So you were running Docker in rootless mode, but not doing that anymore now, right?
FWIW: k3d runs containers in privileged mode, but K3s has an experimental --rootless mode.