calico-kube-controllers Single Point of Failure and Automatic Remediation

[Aslan-Liu]

Environment

• Kubernetes Version: v1.26.5
• Calico Version: v3.25.1
• Installation Method: Kubrspray
• CNI Plugin: Calico
• Datastore: Kubernetes API

Problem Description

We encountered a cascading failure scenario where a calico-node pod malfunction caused the calico-kube-controllers pod to become stuck in CrashLoopBackOff on the same node, effectively breaking the Calico control plane.

Detailed Scenario

1. Initial Failure: The calico-node pod on a specific node experiences network state corruption (veth interfaces, iptables, or routing issues)

2. Symptom: All container-based networking on that node fails:

   • Pods cannot reach the Kubernetes API server (https://192.168.0.1:443)
   • Host machine networking remains functional (bypasses container network)
   • Error logs from calico-node:

   [ERROR] startup/startup.go 173: failed to query Rancher's cluster state config map
   error=Get "https://192.168.0.1:443/api/v1/namespaces/kube-system/configmaps/full-cluster-state?timeout=2s":
   net/http: request canceled (Client.Timeout exceeded while awaiting headers)
   Calico node failed to start
   

3. Cascading Impact:

   • calico-kube-controllers (single replica) happens to be scheduled on this node
   • It enters CrashLoopBackOff because it cannot reach the API server
   • Since the node itself appears "Ready" to Kubernetes, the pod is not rescheduled
   • The Calico control plane becomes degraded/unavailable

4. Current Workaround: Manually restarting the calico-node pod restores networking and allows calico-kube-controllers to function again.

Root Cause Analysis

• calico-node network objects (veth, routing, iptables) become abnormal, breaking all container networking
• calico-kube-controllers runs as a single replica (to avoid race conditions)
• Kubernetes does not reschedule the pod because the node status is "Ready"
• No automatic remediation mechanism exists to detect and fix this scenario

Questions

1. High Availability for calico-kube-controllers

Is there (or will there be) support for running multiple replicas of calico-kube-controllers?

2. Automatic Remediation Strategies

What are the recommended approaches for automatically detecting and remediating calico-node failures?

3. Best Practices

What is the recommended production deployment strategy to handle this scenario?

[caseydavenport]

Calico Version: v3.25.1

FWIW, this is pretty old version of Calico. I'd recommend upgrading.

Is there (or will there be) support for running multiple replicas of calico-kube-controllers?

kube-controllers is not a critical path component, and it crashlooping shouldn't impact cluster functionality. We don't currently have plans to support HA for it, and several of the controllers are intended to be run as singleton and would otherwise fight with each other. The more pressing issue is why calico/node is failing and not self-healing.

Manually restarting the calico-node pod restores networking
Calico node failed to start

I'm a little bit confused about this - when calico/node fails to start, it automatically gets restarted. So I wonder why manually restarting behaves any differently?

definitely worth upgrading (even if we diagnose the fix here, v3.25 is way out of patch release support) and then collecting logs from a failing calico/node, which should help me diagnose what might be going on.

[Aslan-Liu]

Thank you for your clarification.

In our situation, the calico/node pod did not crash or restart—it was running, but networking issues occurred and could only be resolved by manually restarting the pod. We are planning to upgrade Calico, but I am still concerned about the possibility of similar silent failures happening in the future, even on newer versions.

Is there any recommended way to automatically detect and remediate these issues, where the pod remains healthy according to Kubernetes but network functionality is affected? If no standard solution exists, I am also open to implementing custom detection and remediation.

Any suggestions or references would be greatly appreciated!

[caseydavenport]

I think it's really going to depend on what the specific issue is. Would love to dig in to what actually was causing the networking issues if possible, that's probably the best way of ensuring it doesn't happen again (and potentially enhancing our readiness / liveness probe code to be aware of that situation)

[Aslan-Liu]

Thank you for your response.
Unfortunately, the issue occurred very suddenly and I wasn't able to identify the root cause or reproduce it reliably. At the time, I had to troubleshoot step by step, and eventually found it was related to Calico; restarting the pod was the quickest way to restore the environment.

Right now, I'm not sure how to pinpoint the exact reason behind the problem, which is why I'm thinking about setting up an automated remediation mechanism. At least, if the same issue happens again in the future, we can minimize downtime and recover services more quickly.

[mazdakn]

@Aslan-Liu next time when the issue happens, you should dump calico-node logs from the faulty node, and include it here, so we can pin point the issue. Also there are many improvements, and bug fixes introduced to Calico since 3.25.1, so upgrading would help from different aspects.