multiple k8s clusters and etcd clusters in one k8s app when network is unstable

[nobuto-m]

Bug Description

In an extreme circumstance with network instability or disruption in jujud communication, juju deploy k8s -n3 creates two Kubernetes clusters and two separate etcd clusters in one application.

There are two improvements we can make probably:

1. the initial cluster creation can be more robust even in an extreme circumstance such as network disruptions
2. juju status shouldn't say all green when things go wrong such as multiple clusters in one app, or number of units != number of cluster members

$ juju status
Model     Controller  Cloud/Region         Version  SLA          Timestamp
k8s-test  localhost   localhost/localhost  3.6.21   unsupported  11:45:16Z

App         Version  Status  Scale  Charm       Channel       Rev  Exposed  Message
k8s         1.35.0   active      3  k8s         1.35/beta    1846  no       Ready
keepalived           active      3  keepalived  1.32/stable    59  no       VIP ready

Unit             Workload  Agent  Machine  Public address  Ports     Message
k8s/0*           active    idle   0        10.0.9.68       6443/tcp  Ready
  keepalived/0   active    idle            10.0.9.68                 VIP ready
k8s/1            active    idle   1        10.0.9.66       6443/tcp  Ready
  keepalived/2   active    idle            10.0.9.66                 VIP ready
k8s/2            active    idle   2        10.0.9.167      6443/tcp  Ready
  keepalived/1*  active    idle            10.0.9.167                VIP ready

Machine  State    Address     Inst id        Base          AZ       Message
0        started  10.0.9.68   juju-18f332-0  ubuntu@24.04  darkbox  Running
1        started  10.0.9.66   juju-18f332-1  ubuntu@24.04  darkbox  Running
2        started  10.0.9.167  juju-18f332-2  ubuntu@24.04  darkbox  Running

^^^ All green

$ juju exec -a k8s -- k8s status
k8s/0:
cluster status:           ready
control plane nodes:      10.0.9.68:6400 (voter)
high availability:        no
datastore:                etcd
network:                  enabled
dns:                      enabled at 10.152.183.128
ingress:                  disabled
load-balancer:            enabled, L2 mode
local-storage:            enabled at /var/snap/k8s/common/rawfile-storage
gateway                   disabled

k8s/1:
cluster status:           ready
control plane nodes:      10.0.9.21:6400 (spare), 10.0.9.167:6400 (voter)
high availability:        no
datastore:                etcd
network:                  enabled
dns:                      enabled at 10.152.183.247
ingress:                  disabled
load-balancer:            enabled, L2 mode
local-storage:            enabled at /var/snap/k8s/common/rawfile-storage
gateway                   disabled

k8s/2:
cluster status:           ready
control plane nodes:      10.0.9.21:6400 (spare), 10.0.9.167:6400 (voter)
high availability:        no
datastore:                etcd
network:                  enabled
dns:                      enabled at 10.152.183.247
ingress:                  disabled
load-balancer:            enabled, L2 mode
local-storage:            enabled at /var/snap/k8s/common/rawfile-storage
gateway                   disabled

^^^ Split brain, one unit sees only themselves, other two see each other. So there are two clusters in one app.

$ juju exec -a k8s '
    etcdctl \
        --endpoints=https://127.0.0.1:2379 \
        --cert=/etc/kubernetes/pki/etcd/server.crt \
        --key=/etc/kubernetes/pki/etcd/server.key \
        --cacert=/etc/kubernetes/pki/etcd/ca.crt \
        member list -w table
'
k8s/0:
+------------------+---------+---------------+------------------------+------------------------+------------+
|        ID        | STATUS  |     NAME      |       PEER ADDRS       |      CLIENT ADDRS      | IS LEARNER |
+------------------+---------+---------------+------------------------+------------------------+------------+
| fd95843cce776408 | started | juju-18f332-0 | https://10.0.9.68:2380 | https://10.0.9.68:2379 |      false |
+------------------+---------+---------------+------------------------+------------------------+------------+

k8s/1:
+------------------+---------+---------------+-------------------------+-------------------------+------------+
|        ID        | STATUS  |     NAME      |       PEER ADDRS        |      CLIENT ADDRS       | IS LEARNER |
+------------------+---------+---------------+-------------------------+-------------------------+------------+
| 426542836e0cfceb | started | juju-18f332-1 |  https://10.0.9.21:2380 |  https://10.0.9.21:2379 |      false |
| fb2f3d7b31443261 | started | juju-18f332-2 | https://10.0.9.167:2380 | https://10.0.9.167:2379 |      false |
+------------------+---------+---------------+-------------------------+-------------------------+------------+

k8s/2:
+------------------+---------+---------------+-------------------------+-------------------------+------------+
|        ID        | STATUS  |     NAME      |       PEER ADDRS        |      CLIENT ADDRS       | IS LEARNER |
+------------------+---------+---------------+-------------------------+-------------------------+------------+
| 426542836e0cfceb | started | juju-18f332-1 |  https://10.0.9.21:2380 |  https://10.0.9.21:2379 |      false |
| fb2f3d7b31443261 | started | juju-18f332-2 | https://10.0.9.167:2380 | https://10.0.9.167:2379 |      false |
+------------------+---------+---------------+-------------------------+-------------------------+------------+

^^^ etcd has also a split brain in the same way as the k8sd.

To Reproduce

Run the attached script or follow the steps below:

1. juju deploy k8s -n 3
2. juju deploy keepalived
3. juju relate keepalived k8s

From my testing, it cannot be reproducible with normal deployments, but network instability caused by the keepalived subordinate can make it reliably reproducible.

k8s-etcd-keepalived.sh

Environment

k8s 1.35/edge 1846

Relevant log output

k8s-debug.log

machine-2: 2026-05-07 08:52:09 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-2: 2026-05-07 08:52:09 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-2: 2026-05-07 08:52:09 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-2: 2026-05-07 08:52:09 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-2: 2026-05-07 08:52:09 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-2: 2026-05-07 08:52:09 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-0: 2026-05-07 08:52:46 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-0: 2026-05-07 08:52:46 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-0: 2026-05-07 08:52:46 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-0: 2026-05-07 08:52:46 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-0: 2026-05-07 08:52:46 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-0: 2026-05-07 08:52:46 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-0: 2026-05-07 08:52:46 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-0: 2026-05-07 08:52:46 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
machine-0: 2026-05-07 08:52:46 ERROR juju.api.watcher error trying to stop watcher: connection is shutdown before send: connection is shut down
...

Additional context

keepalived is used as a workaround for missing HA for K8s API in the k8s snap or charm in the first place so that we can proceed with HA testing for Juju K8s applications on top of K8s.

[nobuto-m]

sosreport-juju-18f332-0-2026-05-07-agdplzi.tar.gz

sosreport-juju-18f332-1-2026-05-07-czhrjpf.tar.gz

sosreport-juju-18f332-2-2026-05-07-mzaiouk.tar.gz