etcd member reconciler: a restarting controller silently clears its own spec.leave, revoking an in-flight leave

[i-prudnikov]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

boot loose
/# uname -srvmo; cat /etc/os-release || lsb_release -a
Linux 6.12.76-linuxkit #1 SMP Tue Jul 21 14:38:37 UTC 2026 aarch64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Version

main @ dff90ce (found on
v1.35.5-0.20260508195609-b42a7462a089, where it is the common case; #7649 narrowed it but did
not close it).

Sysinfo

Total memory: 20.2 GiB (pass)
File system of /var/lib/k0s: ext4 (pass)
Disk space available for /var/lib/k0s: 735.3 GiB (pass)
Relative disk space available for /var/lib/k0s: 80% (pass)
Name resolution: localhost: [::1 127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.12.76-linuxkit (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: unavailable (pass)
  Executable in PATH: modprobe: exec: "modprobe": executable file not found in $PATH (warning)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: built-in (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: built-in (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: built-in (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: built-in (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: built-in (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: built-in (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: built-in (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: built-in (pass)
      CONFIG_NETFILTER_NETLINK: built-in (pass)
      CONFIG_NF_NAT: built-in (pass)
      CONFIG_IP_SET: IP set support: built-in (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: built-in (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: built-in (pass)
      CONFIG_IP_VS: IP virtual server support: built-in (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: built-in (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: built-in (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: built-in (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: built-in (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: built-in (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: built-in (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: built-in (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: built-in (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: built-in (pass)
      CONFIG_NF_DEFRAG_IPV4: built-in (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: built-in (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: built-in (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: built-in (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: built-in (pass)
      CONFIG_NF_DEFRAG_IPV6: built-in (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: built-in (pass)
      CONFIG_LLC: built-in (pass)
      CONFIG_STP: built-in (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

CreateMemberObject sets em.Spec.Leave = false on every start when the CR already exists (L494).
reconcileMember then returns early on !member.Spec.Leave (L567). So if the target's k0s comes
back before the removal lands, the leave request is discarded — with no status, event, or log to
say so.

Since #7649 the self-leave normally lands first. It does not always: Etcd.Stop() collects the
leave error and stops regardless (etcd.go L273-288), e.g. when etcd is unreachable or quorum is
briefly lost. The unit is Restart=always with RestartSec=10 (pkg/install/systemd_linux.go:22,42
— kardianos/service v1.2.4 default), so the target is back ~10 s later, still a member, and
clears the flag.

Steps to reproduce

1. Three systemd-managed controllers, stacked etcd.
2. kubectl patch etcdmember node-b --type=merge -p '{"spec":{"leave":true}}'
3. Make the self-leave fail (e.g. drop node-b's peer traffic just before it stops).
4. kubectl get etcdmember node-b -o yaml -w

Expected behavior

The removal either completes or is reported as failed; spec.leave is not silently dropped.

Actual behavior

spec.leave flips back to false, Joined stays True, the member stays in etcdctl member list. Leader logs member not marked for leave, no action needed. An automated caller has no way to distinguish a revoked leave from a slow one and blocks until its own timeout.

Suggested fix:

clear the flag only when the leave actually completed — i.e. when Joined is absent or False:

  // createMemberObject, replacing the unconditional em.Spec.Leave = false
  if joined := em.Status.GetCondition(etcdv1beta1.ConditionTypeJoined); joined == nil ||
  joined.Status == etcdv1beta1.ConditionFalse {
        em.Spec.Leave = false
  } else if em.Spec.Leave {
        log.Info("spec.leave is set and this member has not left yet; preserving the leave request
  }
  // createMemberObject, replacing the unconditional em.Spec.Leave = false
  if joined := em.Status.GetCondition(etcdv1beta1.ConditionTypeJoined); joined == nil ||
  joined.Status == etcdv1beta1.ConditionFalse {
        em.Spec.Leave = false
  } else if em.Spec.Leave {
        log.Info("spec.leave is set and this member has not left yet; preserving the leave request
  }

A Joined=True member with spec.leave=true is someone else's in-flight operation; overwriting it
cancels that operation without telling anyone. Narrower alternative: keep the clear, skip it
while this member is still in the local etcd member list, and log at warn.

Workaround: stop the k0s service on the target before setting spec.leave=true. Don't k0s reset
first — the k0s-ctrl- lease object must still exist or the reconciler bails with "No k0s
controller lease found" (L624-634).

[twz123]

Flipping back the leave field to false unconditionally on each k0s start was a deliberate decision back in the day. Not sure if we want to change that.

Maybe we can retry harder to leave the cluster. IIUC the retry logic correctly, we're doing it 10 times every 3 to 6 seconds. Are you able to share the logs of a k0s shutdown that failed to leave the cluster?

Any thoughts @jnummelin?