Resilience issue when losing a node in a P2P HA setup

[jnamdar]

Kairos version:

PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

KAIROS_ARCH="arm64"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_FAMILY="debian"
KAIROS_FIPS="false"
KAIROS_FLAVOR="ubuntu"
KAIROS_FLAVOR_RELEASE="22.04"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_ID="kairos"
KAIROS_ID_LIKE="kairos-standard-ubuntu-22.04"
KAIROS_INIT_VERSION="v0.5.7"
KAIROS_MODEL="rpi4"
KAIROS_NAME="kairos-standard-ubuntu-22.04"
KAIROS_RELEASE="v3.5.0"
KAIROS_SOFTWARE_VERSION="v1.33.2+k3s1"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_TARGETARCH="arm64"
KAIROS_TRUSTED_BOOT="false"
KAIROS_VARIANT="standard"
KAIROS_VERSION="v3.5.0"

CPU architecture, OS, and Version:

Linux kairos-be24 5.15.0-1080-raspi #83-Ubuntu SMP PREEMPT Fri May 30 13:44:46 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
Describe the bug

I wanted to test the overall resiliency of a multi node setup in KairOS as mentioned here : https://kairos.io/docs/architecture/network/, "Resilient: Kairos ensures that the cluster remains resilient, even in the face of network disruptions or failures. By using VirtualIPs, nodes can communicate with each other without the need for static IPs, and the cluster’s etcd database remains unaffected by any disruptions."

Basically I set up a 4 nodes cluster using 4 identical Raspberry Pi 4B and configured my p2p section (fitting this example https://kairos.io/docs/examples/multi-node-p2p-ha/) as such :

p2p:
  # Enforce the discovery of nodes on local network
  disable_dht: true
  network_token: "<TOKEN_HERE>"
  # Automatic cluster deployment configuration
  auto:
    # Enables Automatic node configuration (self-coordination)
    # for role assignment
    enable: true
    # HA enables automatic HA roles assignment.
    # A master cluster init is always required,
    # Any additional master_node is configured as part of the
    # HA control plane.
    # If auto is disabled, HA has no effect.
    ha:
      # Enables HA control-plane
      enable: true
      # Number of HA additional master nodes.
      # A master node is always required for creating the cluster and is implied.
      # The setting below adds 2 additional master nodes, for a total of 3.
      master_nodes: 2

At first, everything goes according to plan. I get a HA controlplane with 3 nodes elected as masters (including one as the clusterinit) :

kairos@kairos-be24:~$ sudo kairos role list
Node                                             Role                            IP             
-----------------------------------------------  ------------------------------  ---------------
690b186899e44cf3b7139659271d9d5f-kairos-690b     worker                          10.1.0.4       
0e34593a16cc4eab96b149dfa653ac79-kairos-0e34     master/ha                       10.1.0.2       
be249c08f7f043f0a16243985875afc2-kairos-be24     master/clusterinit              10.1.0.1       
dd24ec0920764fec8748213c161b9079-kairos-dd24     master/ha                       10.1.0.3       

kairos@kairos-be24:~$ sudo kubectl get node -o wide
NAME                   STATUS   ROLES                       AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
kairos-0e34            Ready    control-plane,etcd,master   19h   v1.33.2+k3s1   10.1.0.2      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1
kairos-690b-d939c31f   Ready    <none>                      19h   v1.33.2+k3s1   10.1.0.4      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1
kairos-be24            Ready    control-plane,etcd,master   19h   v1.33.2+k3s1   10.1.0.1      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1
kairos-dd24            Ready    control-plane,etcd,master   19h   v1.33.2+k3s1   10.1.0.3      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1

Then, I shut down one of my nodes and monitor the resiliency : I chose kairos-be24. I can see logs and events indicating the node lost becomes "NotReady" from k3s point of view, and I still have a connection to my cluster, which is good so far. Note that, at this point, I haven't checked the output of sudo kairos role list again, but I suspect it chose to change one of my worker roles into a master role at this point.

I then plug the node back online, but this time something odd happens. I can confirm KairOS is updating the roles to try and fit the HA control plane I requested in my configuration (3 nodes) when checking the output :

kairos@kairos-be24:~$ sudo kairos role list
Node                                             Role                            IP             
-----------------------------------------------  ------------------------------  ---------------
690b186899e44cf3b7139659271d9d5f-kairos-690b     master/ha                       10.1.0.4       
0e34593a16cc4eab96b149dfa653ac79-kairos-0e34     master/ha                       10.1.0.2       
be249c08f7f043f0a16243985875afc2-kairos-be24     worker                                         
dd24ec0920764fec8748213c161b9079-kairos-dd24     master/clusterinit              10.1.0.3       

But the missing IP address on the worker is odd. Finally, the output of kubectl get nodes hasn't changed :

kairos@kairos-be24:~$ sudo kubectl get node -o wide
NAME                   STATUS   ROLES                       AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
kairos-0e34            Ready    control-plane,etcd,master   19h   v1.33.2+k3s1   10.1.0.2      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1
kairos-690b-d939c31f   Ready    <none>                      19h   v1.33.2+k3s1   10.1.0.4      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1
kairos-be24            Ready    control-plane,etcd,master   19h   v1.33.2+k3s1   10.1.0.1      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1
kairos-dd24            Ready    control-plane,etcd,master   19h   v1.33.2+k3s1   10.1.0.3      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1

which is weird, since there is now a mismatch between KairOS roles' choice, and k3s nodes roles.

Logs from kairos-agent confirm the node-be24 does not seem to update its configuration regarding the role change :

Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Active: '12D3KooWB1d4UmjUL97qi4N9W7QpzjGGNnmbu7LWTyM3daz47gK7'
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Active: '12D3KooWGTFxGnmimku1gZe8MREqvvC97u8NczZXj39pEMvDYHeX'
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Active: '12D3KooWLHQpgrmtvjPTEmV93wY7XyoX1qXszdiwJx6fokWzHfKc'
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Active: '12D3KooWNAdx5HXJpyuid4uXcKh6QfjntMvVE6bvnL7xxNu2KMTG'
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Applying role 'auto'
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Role loaded. Applying auto
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Active nodes:[12D3KooWB1d4UmjUL97qi4N9W7QpzjGGNnmbu7LWTyM3daz47gK7 12D3KooWGTFxGnmimku1gZe8MREqvvC97u8NczZXj39pEMvDYHeX 12D3KooWLHQpgrmtvjPTEmV93wY7XyoX1qXszdiwJx6fokWzHfKc 12D3KooWNAdx5HXJpyuid4uXcKh6QfjntMvVE6bvnL7xxNu2KMTG]
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Advertizing nodes:[dd24ec0920764fec8748213c161b9079-kairos-dd24 be249c08f7f043f0a16243985875afc2-kairos-be24 0e34593a16cc4eab96b149dfa653ac79-kairos-0e34 690b186899e44cf3b7139659271d9d5f-kairos-690b]
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: <be249c08f7f043f0a16243985875afc2-kairos-be24> not a leader, leader is '690b186899e44cf3b7139659271d9d5f-kairos-690b', sleeping
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Roles assigned
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Applying role 'worker'
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Role loaded. Applying worker
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Starting Worker
Dec 18 13:50:03 kairos-be24 kairos-provider[1657]: Node already configured, backing off

I am guessing since the node has already been configured once with the master role, a sentinel file prevents it from reconfiguring (https://github.com/kairos-io/provider-kairos/blob/0f254b788f23c65360bbdd0fd86d522107a5b9df/internal/role/p2p/worker.go#L28).

The same logs from the node who went worker -> master show the same thing in reverse :

Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Active: '12D3KooWNAdx5HXJpyuid4uXcKh6QfjntMvVE6bvnL7xxNu2KMTG'
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Active: '12D3KooWB1d4UmjUL97qi4N9W7QpzjGGNnmbu7LWTyM3daz47gK7'
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Active: '12D3KooWGTFxGnmimku1gZe8MREqvvC97u8NczZXj39pEMvDYHeX'
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Active: '12D3KooWLHQpgrmtvjPTEmV93wY7XyoX1qXszdiwJx6fokWzHfKc'
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Applying role 'auto'
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Role loaded. Applying auto
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Active nodes:[12D3KooWB1d4UmjUL97qi4N9W7QpzjGGNnmbu7LWTyM3daz47gK7 12D3KooWGTFxGnmimku1gZe8MREqvvC97u8NczZXj39pEMvDYHeX 12D3KooWLHQpgrmtvjPTEmV93wY7XyoX1qXszdiwJx6fokWzHfKc 12D3KooWNAdx5HXJpyuid4uXcKh6QfjntMvVE6bvnL7xxNu2KMTG]
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Advertizing nodes:[be249c08f7f043f0a16243985875afc2-kairos-be24 0e34593a16cc4eab96b149dfa653ac79-kairos-0e34 690b186899e44cf3b7139659271d9d5f-kairos-690b dd24ec0920764fec8748213c161b9079-kairos-dd24]
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: I'm the leader. My UUID is: 690b186899e44cf3b7139659271d9d5f-kairos-690b.
                                                    Current assigned roles: map[0e34593a16cc4eab96b149dfa653ac79-kairos-0e34:master/ha 690b186899e44cf3b7139659271d9d5f-kairos-690b:master/ha be249c08f7f043f0a16243985875afc2-kairos-be24:worker dd24ec0920764fec8748213c161b9079-kairos-dd24:master/clusterinit]
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Master already present: true
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Unassigned nodes: []
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Done scheduling
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Roles assigned
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Applying role 'master/ha'
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Role loaded. Applying master/ha
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Starting Master(master/ha)
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Checking role assignment
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Determining K8s distro
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Verifying sentinel file
Dec 18 14:18:06 kairos-690b kairos-provider[1607]: Node already configured, propagating master data and backing off

Ultimately my question would be: is that use case supported ? Can KairOS withstand node failure ? Should I have completely reset my node before plugging it back online ? Even if I did that, the node KairOS elected at first as a worker and then changed to a master did not reconfigure, and is still a worker from k3s point of view, which will probably cause issues at some point.

All in all I think the documentation around the resiliency could be upgraded to further explain the resiliency aspect promoted in the p2p docs.

[jnamdar]

Update: I now wanted to try another use case. I reset my cluster to start fresh. Then, I reset only one node (it was a master/ha node). When it comes back up, the p2p network does pick up the node with a new ID, but the role worker was assigned to the node !
Logs from the node I reset:

Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Active: '12D3KooWAeCS6Rw84eKWsz5qsiNu9x7RiUsNBERnAGi6NDq6M4>
Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Active: '12D3KooWE7PiaeU5bieGUbiL57hx8mgtgk5SKT2YYWRAySb9wM>
Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Active: '12D3KooWKcKLYUaaYt5rouKNpuPSazTicG6uWiTuMyWgHLdWjj>
Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Active: '12D3KooWMpuwmpCUmQLv4KAkjkUsZjxmc3VoKvqEVyjPLvYMPo>
Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Applying role 'auto'
Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Role loaded. Applying auto
Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Active nodes:[12D3KooWAeCS6Rw84eKWsz5qsiNu9x7RiUsNBERnAGi6N>
Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Advertizing nodes:[e9b4e260d0044ba2ad2f1dc2bc96c496-kairos->
Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Backing off, as we are not currently flagged as leader
Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Roles assigned
Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Applying role 'worker'
Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Role loaded. Applying worker
Dec 18 16:15:53 kairos-e9b4 kairos-provider[1625]: Starting Worker

Logs from the leader node:

Dec 18 16:15:44 kairos-a14c kairos-provider[1576]: Active: '12D3KooWAeCS6Rw84eKWsz5qsiNu9x7RiUsNBERnAGi6NDq6M49p'
Dec 18 16:15:44 kairos-a14c kairos-provider[1576]: Active: '12D3KooWE7PiaeU5bieGUbiL57hx8mgtgk5SKT2YYWRAySb9wMPo'
Dec 18 16:15:44 kairos-a14c kairos-provider[1576]: Active: '12D3KooWKcKLYUaaYt5rouKNpuPSazTicG6uWiTuMyWgHLdWjjsm'
Dec 18 16:15:44 kairos-a14c kairos-provider[1576]: Active: '12D3KooWMpuwmpCUmQLv4KAkjkUsZjxmc3VoKvqEVyjPLvYMPo8q'
Dec 18 16:15:44 kairos-a14c kairos-provider[1576]: Applying role 'auto'
Dec 18 16:15:44 kairos-a14c kairos-provider[1576]: Role loaded. Applying auto
Dec 18 16:15:44 kairos-a14c kairos-provider[1576]: Active nodes:[12D3KooWAeCS6Rw84eKWsz5qsiNu9x7RiUsNBERnAGi6NDq6M49p 12D3KooWE7PiaeU5bieGUbiL57hx8mgtgk5SKT2YYWRAySb9w>
Dec 18 16:15:44 kairos-a14c kairos-provider[1576]: Advertizing nodes:[648a58ffadf243c59595bc5cdd77ea6c-kairos-648a e9b4e260d0044ba2ad2f1dc2bc96c496-kairos-e9b4 fc0b247>
Dec 18 16:15:44 kairos-a14c kairos-provider[1576]: I'm the leader. My UUID is: a14c2f21ede14acab2ff3501d26fc845-kairos-a14c.
                                                    Current assigned roles: map[3b9907e206334b2f997e4504639702d3-kairos-3b99:master/clusterinit 648a58ffadf243c59595bc5>
Dec 18 16:15:44 kairos-a14c kairos-provider[1576]: Master already present: true
Dec 18 16:15:44 kairos-a14c kairos-provider[1576]: Unassigned nodes: [e9b4e260d0044ba2ad2f1dc2bc96c496-kairos-e9b4]
Dec 18 16:15:44 kairos-a14c kairos-provider[1576]: -> Set worker to e9b4e260d0044ba2ad2f1dc2bc96c496-kairos-e9b4

I don't really understand why the worker role was chosen, and actually a few minutes later, the leader complains it doesn't have enough members to form the requested control plane (I had 3 nodes in the control plane, which is now down to 2):

Dec 18 16:20:24 kairos-a14c kairos-provider[1576]: Active: '12D3KooWKcKLYUaaYt5rouKNpuPSazTicG6uWiTuMyWgHLdWjjsm'
Dec 18 16:20:24 kairos-a14c kairos-provider[1576]: Active: '12D3KooWMpuwmpCUmQLv4KAkjkUsZjxmc3VoKvqEVyjPLvYMPo8q'
Dec 18 16:20:24 kairos-a14c kairos-provider[1576]: Active: '12D3KooWAeCS6Rw84eKWsz5qsiNu9x7RiUsNBERnAGi6NDq6M49p'
Dec 18 16:20:24 kairos-a14c kairos-provider[1576]: Active: '12D3KooWE7PiaeU5bieGUbiL57hx8mgtgk5SKT2YYWRAySb9wMPo'
Dec 18 16:20:24 kairos-a14c kairos-provider[1576]: Applying role 'auto'
Dec 18 16:20:24 kairos-a14c kairos-provider[1576]: Role loaded. Applying auto
Dec 18 16:20:24 kairos-a14c kairos-provider[1576]: Active nodes:[12D3KooWE7PiaeU5bieGUbiL57hx8mgtgk5SKT2YYWRAySb9wMPo 12D3KooWKcKLYUaaYt5rouKNpuPSazTicG6uWiTuMyWgHLdWj>
Dec 18 16:20:24 kairos-a14c kairos-provider[1576]: Advertizing nodes:[3b9907e206334b2f997e4504639702d3-kairos-3b99 e9b4e260d0044ba2ad2f1dc2bc96c496-kairos-e9b4 a14c2f2>
Dec 18 16:20:24 kairos-a14c kairos-provider[1576]: I'm the leader. My UUID is: a14c2f21ede14acab2ff3501d26fc845-kairos-a14c.
                                                    Current assigned roles: map[3b9907e206334b2f997e4504639702d3-kairos-3b99:master/clusterinit 648a58ffadf243c59595bc5>
Dec 18 16:20:24 kairos-a14c kairos-provider[1576]: Master already present: true
Dec 18 16:20:24 kairos-a14c kairos-provider[1576]: Unassigned nodes: []
Dec 18 16:20:24 kairos-a14c kairos-provider[1576]: Failed applying roleautonot enough nodes to create HA control plane

Finally, the old node still shows from k3s point of view, even though KairOS did remove it from its members list.

kairos@kairos-a14c:~$ sudo kubectl get node -o wide
NAME                   STATUS     ROLES                       AGE     VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
kairos-3b99            Ready      control-plane,etcd,master   3h15m   v1.33.2+k3s1   10.1.0.3      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1
kairos-648a-8fe7f405   Ready      <none>                      3h15m   v1.33.2+k3s1   10.1.0.4      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1
kairos-a14c            Ready      control-plane,etcd,master   3h14m   v1.33.2+k3s1   10.1.0.1      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1
kairos-e9b4-6d84ffa9   Ready      <none>                      101m    v1.33.2+k3s1   10.1.0.5      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1
kairos-fc0b            NotReady   control-plane,etcd,master   3h14m   v1.33.2+k3s1   10.1.0.2      <none>        Ubuntu 22.04.5 LTS   5.15.0-1080-raspi   containerd://2.0.5-k3s1

[jimmykarily]

@jnamdar I edited the description to remove the token. The token allows anybody to connect to your cluster and thus should be treated like a passsword. I assume you don't use that token anymore but if you do, better change to a new one and never share it.

[jnamdar]

@jnamdar I edited the description to remove the token. The token allows anybody to connect to your cluster and thus should be treated like a passsword. I assume you don't use that token anymore but if you do, better change to a new one and never share it.

Yeah thanks, to be honest I thought about it but didn't bother since I'm on a LAN, so noone else can connect 😊

[jimmykarily]

@jnamdar I edited the description to remove the token. The token allows anybody to connect to your cluster and thus should be treated like a passsword. I assume you don't use that token anymore but if you do, better change to a new one and never share it.

Yeah thanks, to be honest I thought about it but didn't bother since I'm on a LAN, so noone else can connect 😊

hm, don't be so sure about that :D . I mean, with edgevpn you get NAT traversal. I use it to connect to my home computer when I'm on the road.

[jimmykarily]

@jnamdar I had a look now on the situation you describe above. First of all, nice write up and debugging, helps a lot! I don't know if these scenarios are supported, I'm not very familiar with this feature, maybe @mudler can chime in here.

That said, in your second test, while you reset your node, based on the findings of your first test, I would expect another node (the worker) to be promoted to control-plane, although unsuccessfully (similarly to what happened in your first test). Which might explain why the node you reset got the worker role: the control-plane role was already taken by the previously-worker node.

Even if this is the case, it still doesn't answer the original question of whether the scenario is supported or not. The sentinel seems to be very intentional there to avoid re-configuring so I'm not sure what the expected flow is. I'll let others chime in otherwise we can look for traces of intention in the commit messages.

[jimmykarily]

btw, since you found the sentinel file, why don't you try to delete it to see if that works?