Node status incorrect after host fail [1.34]

[kcarson77]

Summary

I'm running 3 nodes on 1.34. When I take a host down I get 1 of 3 outcomes:

1. All nodes remain reported as Ready even though a host is not accessible
2. The node I put offline will go NotReady as will another healthy one, ie 2 nodes reported as NotReady
3. The node I put offline will be marked NotReady

This is reproducible on different systems on both Red Hat and Ubuntu
Seeing other undesirable behaviour like pods not being (re)scheduled

dmhost1 taken offline:

labuser@dmhost2:~$ kubectl get no
NAME STATUS ROLES AGE VERSION
dmhost1 NotReady 9d v1.34.1
dmhost2 Ready 9d v1.34.1
dmhost3 NotReady 9d v1.34.1

When I brought host1 back online, it returned to Ready, however host3 did not (should not have been NotReady)

dmhost3
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message

---

NetworkUnavailable False Thu, 23 Oct 2025 10:45:20 +0000 Thu, 23 Oct 2025 10:45:20 +0000 CalicoIsUp Calico is running on this node
MemoryPressure Unknown Thu, 23 Oct 2025 10:51:25 +0000 Thu, 23 Oct 2025 10:50:53 +0000 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Thu, 23 Oct 2025 10:51:25 +0000 Thu, 23 Oct 2025 10:50:53 +0000 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Thu, 23 Oct 2025 10:51:25 +0000 Thu, 23 Oct 2025 10:50:53 +0000 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Thu, 23 Oct 2025 10:51:25 +0000 Thu, 23 Oct 2025 10:50:53 +0000 NodeStatusUnknown Kubelet stopped posting node status.

Seen other strange behaviour like not reacting to the NotReady

What Should Happen Instead?

Failed node, and only failed node, should be marked NotReady

Reproduction Steps

1. 3 node system
2. Take host down

Introspection Report

dmhost1 offline

inspection-report-20251023_113929.tar.gz

Can you suggest a fix?

Are you interested in contributing with a fix?

[kcarson77]

Other scenario |(different cluster with just mK8s installed)

Host 1 offline. All nodes reporting Ready

labuser@dailybuild4-host2:$ kubectl get no
NAME STATUS ROLES AGE VERSION
dailybuild4-host1 Ready 5h13m v1.34.1
dailybuild4-host2 Ready 4h53m v1.34.1
dailybuild4-host3 Ready 4h53m v1.34.1
labuser@dailybuild4-host2:$ date
Thu 23 Oct 18:12:15 BST 2025
labuser@dailybuild4-host2:~$ ping host1
PING dailybuild4-host1 (10.173.130.211) 56(84) bytes of data.
From dailybuild4-host2 (10.173.130.212) icmp_seq=1 Destination Host Unreachable
From dailybuild4-host2 (10.173.130.212) icmp_seq=2 Destination Host Unreachable
From dailybuild4-host2 (10.173.130.212) icmp_seq=3 Destination Host Unreachable
From dailybuild4-host2 (10.173.130.212) icmp_seq=4 Destination Host Unreachable

Describe node shows everything OK and ready status being posted, even though the host is offline and not contactable. Looks stale.

CreationTimestamp: Thu, 23 Oct 2025 12:59:03 +0100
Taints:
Unschedulable: false
Lease:
HolderIdentity: dailybuild4-host1
AcquireTime:
RenewTime: Thu, 23 Oct 2025 18:09:12 +0100
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message

---

NetworkUnavailable False Thu, 23 Oct 2025 13:30:24 +0100 Thu, 23 Oct 2025 13:30:24 +0100 CalicoIsUp Calico is running on this node
MemoryPressure False Thu, 23 Oct 2025 18:08:30 +0100 Thu, 23 Oct 2025 12:59:03 +0100 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 23 Oct 2025 18:08:30 +0100 Thu, 23 Oct 2025 12:59:03 +0100 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 23 Oct 2025 18:08:30 +0100 Thu, 23 Oct 2025 12:59:03 +0100 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 23 Oct 2025 18:08:30 +0100 Thu, 23 Oct 2025 13:31:52 +0100 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.173.130.211
Hostname: dailybuild4-host1

Lease had long since expired (1.5 hours ago)

labuser@dailybuild4-host2:~$ kubectl describe lease dailybuild4-host1 -n kube-node-lease
Name: dailybuild4-host1
Namespace: kube-node-lease
Labels:
Annotations:
API Version: coordination.k8s.io/v1
Kind: Lease
Metadata:
Creation Timestamp: 2025-10-23T11:59:13Z
Owner References:
API Version: v1
Kind: Node
Name: dailybuild4-host1
UID: f55127a1-9289-4c32-a4d6-7269011337ff
Resource Version: 56235
UID: 6ccc28c3-0864-4203-9f5b-7be363cba0bf
Spec:
Holder Identity: dailybuild4-host1
Lease Duration Seconds: 40
Renew Time: 2025-10-23T17:09:12.684880Z
Events:

inspect attached

3 Ready (1 host down)

inspection-report-20251023_181809.tar.gz

[sonianuj287]

Hi @kcarson77 , I've raised a PR. please review it and suggest me any changes.
Thanks :)

[kcarson77]

thanks - I've done some testing. If I manually add those parameters to the files I get API server instability and flapping leading to cyclic connection failures.
I noticed the parameters are not present in the 1.32 files but the defaults seem stable and I don't have these issues, just the known issue around the DB lock (which is why I need 1.34)
Feels like some dqlite issue or control plane coordination issue in 1.34

[kcarson77]

Hi - are there any updates here? If I add the suggested config I get API server instability, in that it continually restarts.

We've had to rollback to 1.32, but that release also has issues.
Is the recommendation to rollback to 1.29?

thanks

[cs-dsmyth]

Also seeing this issue and the fix as suggested (adding the params manually to config) causes a restart as above. Any further guidance would be appreciated as we are stuck here.