Reboots leaves pods in Terminating and Nodes in NotReady

[nmajin]

Is this expected behavior?

kairos@kubedoom-0d54:~> sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml get all --namespace kubedoom
NAME                                                       READY   STATUS        RESTARTS   AGE
pod/kubedoom-kubedoom-helmchart-kubedoom-55f9c8d7d-bwfqf   1/1     Terminating   0          20d
pod/kubedoom-kubedoom-helmchart-novnc-64dbd5d77c-ps8rd     1/1     Terminating   0          22d
pod/kubedoom-kubedoom-helmchart-kubedoom-55f9c8d7d-gtfrv   1/1     Terminating   0          20d
pod/kubedoom-kubedoom-helmchart-novnc-64dbd5d77c-fd2jt     1/1     Terminating   0          20d
pod/kubedoom-kubedoom-helmchart-novnc-64dbd5d77c-8t6kv     1/1     Running       0          20d
pod/kubedoom-kubedoom-helmchart-kubedoom-55f9c8d7d-ksxq5   1/1     Running       0          20d
pod/helm-install-kubedoom-hz9p6                            1/1     Running       0          26s

kairos@kubedoom-0d54:~> sudo kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml get node
NAME            STATUS                        ROLES                  AGE   VERSION
kubedoom-5b5f   NotReady                      control-plane,master   20d   v1.25.4+k3s1
kubedoom-7982   NotReady                      control-plane,master   22d   v1.25.4+k3s1
kubedoom-d3c0   NotReady                      control-plane,master   20d   v1.25.4+k3s1
kubedoom-0d54   Ready                         control-plane,master   23h   v1.26.0+k3s1

I did a recent upgrade, which is why the nodes are on an older version, but I can confirm this happens with reboots for sure.

[mudler]

This doesn't look good. Seems the node is changing hostname in between - can you tell the versions you are testing against?

Can you try to set a static hostname in the cloud config?

Here's at my box:

❯ ssh kairos@xxx
Last login: Thu Dec 29 09:22:28 2022 from xxx
Welcome to Kairos!

Refer to https://kairos.io for documentation.
kairos@kairoslab-41e5:~> sudo reboot
Terminated
kairos@kairoslab-41e5:~> client_loop: send disconnect: Broken pipe

❯ ssh kairos@xxx
Welcome to Kairos!

Refer to https://kairos.io for documentation.
kairos@kairoslab-41e5:~> uptime
 08:42:17  up   0:00,  1 user,  load average: 1.21, 0.30, 0.10
kairos@kairoslab-41e5:~> 

And this is the hostname set on the cloud config:

kairoslab-41e5:~ # cat /oem/90_custom.yaml
#cloud-config

hostname: kairoslab-{{ trunc 4 .MachineID }}

Try to set the hostname to something fixed, e.g. 'kubedoom'. However it should just persist. The machine-id is stored in /usr/local/etc/machine-id and should update /etc/machine-id at the earliest stages. Can you check what you have in /usr/local/etc/machine-id and if it does persist?

[nmajin]

So not sure why I was having this before but when I upgraded to v1.4.0-k3sv1.26.0-k3s1 it looks like my issue is resolved. I find it weird though that k3s service is disabled after the upgrade is done and not enabled again...that seems to obviously break everything coming up and issues finishing the upgrade.

[nmajin]

Upgrade to v1.4.0-k3sv1.26.0-k3s1 seemed to resolve this issue