Flannel pod keeps restarting on new Kubernetes node, other pods like kube-proxy also crash repeatedly despite identical configuration

[ptite-ratte]

Hello,

I have a Kubernetes cluster where I recently added a new node (srvvkubernetes06). Flannel is used as the CNI across all nodes. However, on this new node, the Flannel pod is stuck in a crash loop (CrashLoopBackOff). Other pods on the same node, such as kube-proxy, also exhibit similar behavior — they start, then stop and restart repeatedly.

What I’ve checked so far:

The CNI config files (/etc/cni/net.d/10-flannel.conflist) are exactly the same on the new node and on a working node (srvvkubernetes05).

The vxlan kernel module is loaded on both nodes.

The flannel.1 network interface exists and is UP on both nodes, with node-specific IP addresses.

The Flannel pod’s init containers complete successfully, but the main container keeps restarting.

The pod is deployed via a DaemonSet with the same image version on all nodes.

Other system pods like kube-proxy on the new node also keep restarting, suggesting a broader node-related issue.

I suspect a network or node configuration problem specific to the new node but haven’t identified the root cause yet.

If anyone has faced similar issues or can suggest debugging steps, I would greatly appreciate it!

I can share full pod logs, network interface info, and Flannel config if needed.

Thanks in advance!

[rbrtbnfgl]

Are the CNI binaries installed?

[ptite-ratte]

Yes, the CNI binaries are installed on the node.
Here is the output of ls -l /opt/cni/bin:

-rwxr-xr-x 1 root root 4151672 Feb 5 2021 bandwidth*
-rwxr-xr-x 1 root root 4536104 Feb 5 2021 bridge*
-rwxr-xr-x 1 root root 10270090 Feb 5 2021 dhcp*
-rwxr-xr-x 1 root root 4297556 Jan 10 2024 dummy*
-rwxr-xr-x 1 root root 4767801 Feb 5 2021 firewall*
-rwxr-xr-x 1 root root 2835118 Oct 6 12:01 flannel*
-rwxr-xr-x 1 root root 4144106 Feb 5 2021 host-device*
-rwxr-xr-x 1 root root 3565330 Feb 5 2021 host-local*
-rwxr-xr-x 1 root root 4288339 Feb 5 2021 ipvlan*
-rwxr-xr-x 1 root root 3530531 Feb 5 2021 loopback*
-rwxr-xr-x 1 root root 4367216 Feb 5 2021 macvlan*
-rwxr-xr-x 1 root root 3966455 Feb 5 2021 portmap*
-rwxr-xr-x 1 root root 4467317 Feb 5 2021 ptp*
-rwxr-xr-x 1 root root 3701138 Feb 5 2021 sbr*
-rwxr-xr-x 1 root root 3153330 Feb 5 2021 static*
-rwxr-xr-x 1 root root 4371897 Jan 10 2024 tap*
-rwxr-xr-x 1 root root 3668289 Feb 5 2021 tuning*
-rwxr-xr-x 1 root root 4287972 Feb 5 2021 vlan*
-rwxr-xr-x 1 root root 3759977 Feb 5 2021 vrf*

Also, here are some logs from the kube-flannel container.

They look completely fine, showing successful VXLAN setup, correct annotations, and no major errors throughout the runtime.
However, the pod has already restarted over 200 times, and while the logs show a clean init, it eventually crashes — but without any clear error message leading to it:

I1006 12:00:39.216471 1 match.go:286] Defaulting external address to interface address (10.1.10.11)
I1006 12:00:39.216580 1 vxlan.go:141] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
I1006 12:00:39.222131 1 kube.go:636] List of node(srvvkubernetes06) annotations: {...}
I1006 12:00:39.260074 1 main.go:416] Running backend.
I1006 12:00:39.262771 1 vxlan_network.go:100] Received Subnet Event...
...
I1006 12:01:56.786266 1 main.go:452] shutdownHandler sent cancel signal...
E1006 12:01:56.786306 1 subnet.go:137] could not watch leases: context canceled
I1006 12:01:57.859041 1 main.go:440] Exiting cleanly..

So far, no visible configuration issues, no obvious permission errors, and everything appears properly initialized.

Any insights would be appreciated — especially on what could silently trigger this crash loop despite clean logs.

[rbrtbnfgl]

Are you sure that there aren't any errors on the log? If you do a kubectl describe on the pod did you get any error message?

[ptite-ratte]

Yes, I checked both the logs and the pod description, but unfortunately there’s nothing obviously useful or indicative of the problem.

Example:

kubectl describe pod kube-flannel-ds-ttkqq -n kube-flannel shows the container is in CrashLoopBackOff, but no errors are reported — just normal lifecycle messages.

The logs (kubectl logs) don’t show any actual error either — the flannel process seems to start correctly, completes its setup (iptables, subnet manager, etc.), and then shuts down cleanly.

No stack traces or specific error messages are visible. Here's an excerpt of the logs:

I1006 14:18:24.818524 1 vxlan_network.go:79] evts chan closed
I1006 14:18:25.822847 1 iptables.go:81] iptables (ipv4): cleaning-up before exiting flannel...
I1006 14:18:25.889909 1 main.go:440] Exiting cleanly...

So for now, nothing really useful stands out in the logs or the describe output.

[rbrtbnfgl]

something it's sending a termination signal to flannel. Did you check kubelet logs?

[ptite-ratte]

Hi, thanks for your help.

I’ve checked the kubelet logs on the affected node, and here are some recurring messages. All the crashing pods (like Flannel and Node Exporter) show similar errors:

E1006 19:40:03.843921 83646 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to "StartContainer" for "kube-flannel" with CrashLoopBackOff: "back-off 5m0s restarting failed container=kube-flannel pod=kube-flannel-ds-ttkqq_kube-flannel(...)""
...
E1006 19:40:12.821646 83646 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to "StartContainer" for "node-exporter" with CrashLoopBackOff: "back-off 5m0s restarting failed container=node-exporter pod=kube-prometheus-stack-prometheus-node-exporter-bz6nk(...)""
...

These logs repeat in a loop, with the same CrashLoopBackOff errors for all pods on this node.

Let me know if you'd like logs from the containers themselves or from kubectl describe pod. Right now, I'm trying to understand what could be causing all containers to crash like this on that specific node.

Thanks in advance!

[rbrtbnfgl]

It doesn't say which is the error. It seems that there is something on your node that keeps the pods failing to start and it seems unrelated to flannel.

[ptite-ratte]

I have a Kubernetes cluster with 3 servers, and one of them is not functioning properly within the cluster. When checking the routing table (ip route) on each server, I noticed a significant difference.

Here are the ip route outputs for each:

Server 1 (not functional):

default via 10.1.10.1 dev ens192 proto static
10.1.10.0/24 dev ens192 proto kernel scope link src 10.1.10.11
10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink
10.244.3.0/24 via 10.244.3.0 dev flannel.1 onlink
10.244.5.0/24 via 10.244.5.0 dev flannel.1 onlink
10.244.6.0/24 dev cni0 proto kernel scope link src 10.244.6.1 linkdown

On this server, the cni0 interface is in a linkdown state.

Server 2 (functional):

default via 10.1.10.1 dev ens160 proto static
10.1.10.0/24 dev ens160 proto kernel scope link src 10.1.10.9
10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink
10.244.3.0/24 via 10.244.3.0 dev flannel.1 onlink
10.244.5.0/24 dev cni0 proto kernel scope link src 10.244.5.1
10.244.6.0/24 via 10.244.6.0 dev flannel.1 onlink

Here, the cni0 interface is up and operational.

Server 3 (functional):

default via 10.1.10.1 dev ens160 proto static
10.1.10.0/24 dev ens160 proto kernel scope link src 10.1.10.8
10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink
10.244.2.0/24 dev cni0 proto kernel scope link src 10.244.2.1
10.244.3.0/24 via 10.244.3.0 dev flannel.1 onlink
10.244.5.0/24 via 10.244.5.0 dev flannel.1 onlink
10.244.6.0/24 via 10.244.6.0 dev flannel.1 onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown

The cni0 interface is also active here.

The key point is that on the non-functional server, the cni0 interface is linkdown, unlike the functional servers where it is up.

Has anyone encountered this issue before? Any ideas on how to force or fix the cni0 interface state so that the server becomes fully operational in the cluster again?

Thanks in advance for your help!

[rbrtbnfgl]

How the route could be related on your issue?

[ptite-ratte]

I’ve found the cause of my issue — it was related to the containerd configuration. I needed to set the systemd cgroup driver in /etc/containerd/config.toml for runc like this:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true

Thanks a lot for your help and suggestions!