[BUG] starting Node 'k3d-mycluster-server-0' hanging on RPi4 Debian bookworm/sid (64bits)

[Utopiah]

What did you do

• How was the cluster created?

  • k3d cluster create mycluster as per https://k3d.io/v5.6.0/#install-specific-release
  • because rootless failed I tried, after removing rooltess, as root sudo k3d cluster create mycluster

• What did you do afterwards?

  • waited few minutes and checked with top for any activity

What did you expect to happen

The command to terminate and proceed with kubectl get nodes after doing k3d kubeconfig merge mycluster --kubeconfig-switch-context

Screenshots or terminal output

If applicable, add screenshots or terminal output (code block) to help explain your problem.

Which OS & Architecture

arch: aarch64
cgroupdriver: systemd
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: raspberrypi
name: docker
os: Debian GNU/Linux 12 (bookworm)
ostype: linux
version: 24.0.7

Which version of k3d

k3d version v5.6.0
k3s version v1.27.4-k3s1 (default)p

Which version of docker

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:15 2023
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:15 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 4
  Running: 1
  Paused: 0
  Stopped: 3
 Images: 4
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-rpi7-rpi-v8
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.628GiB
 Name: raspberrypi
 ID: 89be8d37-2347-4948-a561-09d478e07720
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support

[iwilltry42]

Hi @Utopiah
Can you please run docker ps -a and docker logs k3d-mycluster-server-0 and paste the output here?

[Utopiah]

Sure, and thanks for helping already :

fabien@raspberrypi:~ $ sudo docker ps -a
CONTAINER ID   IMAGE                            COMMAND                  CREATED          STATUS                   PORTS     NAMES
5c770a763796   ghcr.io/k3d-io/k3d-proxy:5.6.0   "/bin/sh -c nginx-pr…"   20 seconds ago   Created                            k3d-mycluster-serverlb
1277e1961b82   rancher/k3s:v1.27.4-k3s1         "/bin/k3d-entrypoint…"   21 seconds ago   Up 19 seconds                      k3d-mycluster-server-0

and

fabien@raspberrypi:~ $ sudo docker logs k3d-mycluster-server-0
time="2024-01-19T15:50:08Z" level=info msg="Starting k3s v1.27.4+k3s1 (36645e73)"
E0119 15:50:08.699248      24 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0119 15:50:08.701125      24 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0119 15:50:08.702811      24 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0119 15:50:08.704250      24 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?
time="2024-01-19T15:50:08Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2024-01-19T15:50:08Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2024-01-19T15:50:08Z" level=info msg="Database tables and indexes are up to date"
time="2024-01-19T15:50:08Z" level=info msg="Kine available at unix://kine.sock"
time="2024-01-19T15:50:08Z" level=info msg="Bootstrap key locked for initial create"
time="2024-01-19T15:50:08Z" level=info msg="generated self-signed CA certificate CN=k3s-client-ca@1705679408: notBefore=2024-01-19 15:50:08.769168857 +0000 UTC notAfter=2034-01-16 15:50:08.769168857 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=system:k3s-supervisor,O=system:masters signed by CN=k3s-client-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=system:apiserver,O=system:masters signed by CN=k3s-client-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=k3s-cloud-controller-manager signed by CN=k3s-client-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="generated self-signed CA certificate CN=k3s-server-ca@1705679408: notBefore=2024-01-19 15:50:08.787659503 +0000 UTC notAfter=2034-01-16 15:50:08.787659503 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-server-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="generated self-signed CA certificate CN=k3s-request-header-ca@1705679408: notBefore=2024-01-19 15:50:08.791709508 +0000 UTC notAfter=2034-01-16 15:50:08.791709508 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="generated self-signed CA certificate CN=etcd-server-ca@1705679408: notBefore=2024-01-19 15:50:08.795378978 +0000 UTC notAfter=2034-01-16 15:50:08.795378978 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="generated self-signed CA certificate CN=etcd-peer-ca@1705679408: notBefore=2024-01-19 15:50:08.798916801 +0000 UTC notAfter=2034-01-16 15:50:08.798916801 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:08Z" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:08 +0000 UTC"
time="2024-01-19T15:50:10Z" level=info msg="Saving cluster bootstrap data to datastore"
time="2024-01-19T15:50:10Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1705679408: notBefore=2024-01-19 15:50:08 +0000 UTC notAfter=2025-01-18 15:50:10 +0000 UTC"
time="2024-01-19T15:50:10Z" level=warning msg="dynamiclistener [::]:6443: no cached certificate available for preload - deferring certificate load until storage initialization or first client request"
time="2024-01-19T15:50:10Z" level=info msg="Active TLS secret / (ver=) (count 12): map[listener.cattle.io/cn-0.0.0.0:0.0.0.0 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.20.0.2:172.20.0.2 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-k3d-mycluster-server-0:k3d-mycluster-server-0 listener.cattle.io/cn-k3d-mycluster-serverlb:k3d-mycluster-serverlb listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=51925B0D697002223F07E975122DFC3EA01CAF6D]"
time="2024-01-19T15:50:10Z" level=info msg="Bootstrap key lock is held"
time="2024-01-19T15:50:10Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --etcd-servers=unix://kine.sock --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
time="2024-01-19T15:50:10Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259"
time="2024-01-19T15:50:10Z" level=info msg="Waiting for API server to become available"
time="2024-01-19T15:50:10Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true"
time="2024-01-19T15:50:10Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false"
time="2024-01-19T15:50:10Z" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token"
time="2024-01-19T15:50:10Z" level=info msg="To join server node to cluster: k3s server -s https://172.20.0.2:6443 -t ${SERVER_NODE_TOKEN}"
time="2024-01-19T15:50:10Z" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token"
time="2024-01-19T15:50:10Z" level=info msg="To join agent node to cluster: k3s agent -s https://172.20.0.2:6443 -t ${AGENT_NODE_TOKEN}"
W0119 15:50:10.751549      23 feature_gate.go:241] Setting GA feature gate JobTrackingWithFinalizers=true. It will be removed in a future release.
time="2024-01-19T15:50:10Z" level=info msg="Wrote kubeconfig /output/kubeconfig.yaml"
I0119 15:50:10.753124      23 server.go:567] external host was not specified, using 172.20.0.2
time="2024-01-19T15:50:10Z" level=info msg="Run: k3s kubectl"
time="2024-01-19T15:50:10Z" level=fatal msg="failed to find memory cgroup (v2)"
E0119 15:50:11.945124      91 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:6443/api?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused
E0119 15:50:11.946802      91 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:6443/api?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused
E0119 15:50:11.948905      91 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:6443/api?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused
E0119 15:50:11.950494      91 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:6443/api?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

[Utopiah]

Note that

E0119 15:51:50.073213     395 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:6443/api?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused
E0119 15:51:50.074234     395 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:6443/api?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused
E0119 15:51:50.075238     395 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:6443/api?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused
E0119 15:51:50.077209     395 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:6443/api?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

repeats dozens of times, basically until I kill the cluster create command.

[Utopiah]

I doubt it's relevant here but just in case, said RPi4 has Internet access, just ping a known working server.

[iwilltry42]

time="2024-01-19T15:50:10Z" level=fatal msg="failed to find memory cgroup (v2)"

That's the line I was looking for.
Add cgroup_memory=1 cgroup_enable=memory to the file /boot/cmdline.txt and reboot... that should do the trick 👍