etcdmember objects wiped upon join of a new controller node

[openvdro]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 6.8.0-137-generic #137-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 17 20:28:23 UTC 2026 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 24.04.4 LTS"

Version

v1.33.13+k0s.1

Sysinfo

`k0s sysinfo`

Total memory: 21.8 GiB (pass)                                                                                                                                                                                                         [33/144]
File system of /var/lib/k0s: ext4 (pass)
Disk space available for /var/lib/k0s: 96.5 GiB (pass)
Relative disk space available for /var/lib/k0s: 83% (pass)
Name resolution: localhost: [::1 127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.8.0-137-generic (pass)
  Max. file descriptors per process: current: 1048575 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

After deployment of the second controller node, etcdmember objects disappear. We rely on these objects to determine whether it is safe to proceed with the deployment of a third controller.

$ kubectl get etcdmembers -A
No resources found
$ k0s etcd member-list
{"members":{"master-0":"https://10.100.91.52:2380","master-1":"https://10.100.91.50:2380"}}

Steps to reproduce

1. deploy the first k0s controller node
2. deploy the second one

Expected behavior

both etcdmember objects exist in kubernetes, the list corresponds to the etcd member-list output.

Actual behavior

It seems that etcdmember CRD is being recreated upon the join of master-1:

$ kubectl get crd etcdmembers.etcd.k0sproject.io -o jsonpath='{.metadata.creationTimestamp}'
2026-08-17T01:54:01Z

master-0 has fully initialized around this timeframe

TASK [k0s : Wait for k8s-node availability] ************************************
task path: /root/lcm-ansible-0.31.0-59-812d58b5/roles/k0s/tasks/main.yml:234
ok: [master-0] => {
    "attempts": 3,
    "changed": false,
    "cmd": [
        "/usr/local/bin/kubectl",
        "get",
        "node",
        "master-0"
    ],
    "delta": "0:00:00.341634",
    "end": "2026-08-17 01:45:04.961757",
    "invocation": {
        "module_args": {
            "_raw_params": "/usr/local/bin/kubectl get node master-0",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "expand_argument_vars": true,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true
        }
    },
    "rc": 0,
    "start": "2026-08-17 01:45:04.620123"
}

STDOUT:

NAME       STATUS     ROLES    AGE   VERSION
master-0   NotReady   <none>   0s    v1.33.13+k0s

master-1 has fully initialized around this timeframe

TASK [k0s : Wait for k8s-node availability] ************************************
task path: /root/lcm-ansible-0.31.0-59-812d58b5/roles/k0s/tasks/main.yml:234
ok: [master-1] => {
    "attempts": 8,
    "changed": false,
    "cmd": [
        "/usr/local/bin/kubectl",
        "get",
        "node",
        "master-1"
    ],
    "delta": "0:00:00.294705",
    "end": "2026-08-17 01:54:05.150578",
    "invocation": {
        "module_args": {
            "_raw_params": "/usr/local/bin/kubectl get node master-1",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "expand_argument_vars": true,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true
        }
    },
    "rc": 0,
    "start": "2026-08-17 01:54:04.855873"
}

STDOUT:

NAME       STATUS     ROLES    AGE   VERSION
master-1   NotReady   <none>   3s    v1.33.13+k0s

Screenshots and logs

Attaching the k0scontroller logs from both machines

master-0-k0scontroller.stripped.log
master-1-k0scontroller.log

master-0 controller log is stripped to the first 40000 lines to allow upload.

Additional context

Claude suggests it might be a race between starting up applier-manager and creating etcdmembers manifest during CRD.Start().

Restart of k0scontroller service solves the problem.

[twz123]

Can you repro this with a newer k0s version? K0s v1.33 is no longer actively maintained. There've been quite some changes in the etcd member management code in recent releases.

[twz123]

Closed by a misplaced "Fixes" comment. Reopening.

[openvdro]

Hey Tom, we hit this issue pretty rarely in our CI and I have not seen it during manual deployments. We are going to be switching our CI to newer versions in about half a year time, so until then I might not have a repro with a newer version. Though I'll give it a try next week anyway.