|
| 1 | +# CNI Readiness |
| 2 | + |
| 3 | +In many Kubernetes clusters, the CNI plugin runs as a DaemonSet. When a new node joins the cluster, there is a race condition: |
| 4 | +1. The Node object is created and marked `Ready` by the Kubelet. |
| 5 | +2. The Scheduler sees the node as `Ready` and schedules application pods. |
| 6 | +3. However, the CNI DaemonSet might still be initializing networking on that node. |
| 7 | + |
| 8 | +This guide demonstrates how to use the Node Readiness Controller to prevent pods from being scheduled on a node until the Container Network Interface (CNI) plugin (e.g., Calico) is fully initialized and ready. |
| 9 | + |
| 10 | +The high-level steps are: |
| 11 | +1. Node is bootstrapped with a [startup taint](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) `readiness.k8s.io/NetworkReady=pending:NoSchedule` immediately upon joining. |
| 12 | +2. A sidecar is patched to the cni-agent to monitor the CNI's health and report it to the API server as node-condition (`network.k8s.io/CalicoReady`). |
| 13 | +3. Node Readiness Controller will untaint the node only when the CNI reports it is ready. |
| 14 | + |
| 15 | +## Step-by-Step Guide |
| 16 | + |
| 17 | +This example uses **Calico**, but the pattern applies to any CNI. |
| 18 | + |
| 19 | +> **Note**: You can find all the manifests used in this guide in the [`examples/cni-readiness`](https://github.com/kubernetes-sigs/node-readiness-controller/tree/main/examples/cni-readiness) directory. |
| 20 | +
|
| 21 | +### 1. Deploy the Readiness Condition Reporter |
| 22 | + |
| 23 | +We need to bridge Calico's internal health status to a Kubernetes Node Condition. We will add a **sidecar container** to the Calico DaemonSet. |
| 24 | + |
| 25 | +This sidecar checks Calico's local health endpoint (`http://localhost:9099/readiness`) and updates a node condition `network.k8s.io/CalicoReady`. |
| 26 | + |
| 27 | +**Patch your Calico DaemonSet:** |
| 28 | + |
| 29 | +```yaml |
| 30 | +# cni-patcher-sidecar.yaml |
| 31 | +- name: cni-status-patcher |
| 32 | + image: registry.k8s.io/node-readiness-controller/node-readiness-reporter:v0.1.1 |
| 33 | + imagePullPolicy: IfNotPresent |
| 34 | + env: |
| 35 | + - name: NODE_NAME |
| 36 | + valueFrom: |
| 37 | + fieldRef: |
| 38 | + fieldPath: spec.nodeName |
| 39 | + - name: CHECK_ENDPOINT |
| 40 | + value: "http://localhost:9099/readiness" # update to your CNI health endpoint |
| 41 | + - name: CONDITION_TYPE |
| 42 | + value: "network.k8s.io/CalicoReady" # update this node condition |
| 43 | + - name: CHECK_INTERVAL |
| 44 | + value: "15s" |
| 45 | + resources: |
| 46 | + limits: |
| 47 | + cpu: "10m" |
| 48 | + memory: "32Mi" |
| 49 | + requests: |
| 50 | + cpu: "10m" |
| 51 | + memory: "32Mi" |
| 52 | +``` |
| 53 | +
|
| 54 | + > Note: In this example, the CNI pod health is monitored by a side-car, so watcher's lifecycle is same as the pod lifecycle. |
| 55 | + If the Calico pod is crashlooping, the sidecar will not run and cannot report readiness. For robust 'continuous' readiness reporting, the watcher should be 'external' to the pod. |
| 56 | +
|
| 57 | +### 2. Grant Permissions (RBAC) |
| 58 | +
|
| 59 | +The sidecar needs permission to update the Node object's status. |
| 60 | +
|
| 61 | +```yaml |
| 62 | +# calico-rbac-node-status-patch-role.yaml |
| 63 | +apiVersion: rbac.authorization.k8s.io/v1 |
| 64 | +kind: ClusterRole |
| 65 | +metadata: |
| 66 | + name: node-status-patch-role |
| 67 | +rules: |
| 68 | +- apiGroups: [""] |
| 69 | + resources: ["nodes/status"] |
| 70 | + verbs: ["patch", "update"] |
| 71 | +--- |
| 72 | +apiVersion: rbac.authorization.k8s.io/v1 |
| 73 | +kind: ClusterRoleBinding |
| 74 | +metadata: |
| 75 | + name: calico-node-status-patch-binding |
| 76 | +roleRef: |
| 77 | + apiGroup: rbac.authorization.k8s.io |
| 78 | + kind: ClusterRole |
| 79 | + name: node-status-patch-role |
| 80 | +subjects: |
| 81 | +# Bind to CNI's ServiceAccount |
| 82 | +- kind: ServiceAccount |
| 83 | + name: calico-node |
| 84 | + namespace: kube-system |
| 85 | +``` |
| 86 | +
|
| 87 | +### 3. Create the Node Readiness Rule |
| 88 | +
|
| 89 | +Now define the rule that enforces the requirement. This tells the controller: *"Keep the `readiness.k8s.io/NetworkReady` taint on the node until `network.k8s.io/CalicoReady` is True."* |
| 90 | + |
| 91 | +```yaml |
| 92 | +# network-readiness-rule.yaml |
| 93 | +apiVersion: readiness.node.x-k8s.io/v1alpha1 |
| 94 | +kind: NodeReadinessRule |
| 95 | +metadata: |
| 96 | + name: network-readiness-rule |
| 97 | +spec: |
| 98 | + # The condition(s) to monitor |
| 99 | + conditions: |
| 100 | + - type: "network.k8s.io/CalicoReady" |
| 101 | + requiredStatus: "True" |
| 102 | + |
| 103 | + # The taint to manage |
| 104 | + taint: |
| 105 | + key: "readiness.k8s.io/NetworkReady" |
| 106 | + effect: "NoSchedule" |
| 107 | + value: "pending" |
| 108 | + |
| 109 | + # "bootstrap-only" means: once the CNI is ready once, we stop enforcing. |
| 110 | + enforcementMode: "bootstrap-only" |
| 111 | + |
| 112 | + # Update to target only the nodes that need to be protected by this guardrail |
| 113 | + nodeSelector: |
| 114 | + matchLabels: |
| 115 | + node-role.kubernetes.io/worker: "" |
| 116 | +``` |
| 117 | + |
| 118 | +## Test scripts |
| 119 | + |
| 120 | +1. **Create the Readiness Rule**: |
| 121 | + ```sh |
| 122 | + cd examples/cni-readiness |
| 123 | + kubectl apply -f network-readiness-rule.yaml |
| 124 | + ``` |
| 125 | + |
| 126 | +2. **Install Calico CNI and Apply the RBAC**: |
| 127 | + ```sh |
| 128 | + chmod +x apply-calico.sh |
| 129 | + sh apply-calico.sh |
| 130 | + ``` |
| 131 | + |
| 132 | + |
| 133 | +## Verification |
| 134 | + |
| 135 | +To test this, add a new node to the cluster. |
| 136 | + |
| 137 | +1. **Check the Node Taints**: |
| 138 | + Immediately upon joining, the node should have the taint: |
| 139 | + `readiness.k8s.io/NetworkReady=pending:NoSchedule`. |
| 140 | + |
| 141 | +2. **Check Node Conditions**: |
| 142 | + Watch the node conditions. You will initially see `network.k8s.io/CalicoReady` as `False` or missing. |
| 143 | + Once Calico starts, the sidecar will update it to `True`. |
| 144 | + |
| 145 | +3. **Check Taint Removal**: |
| 146 | + As soon as the condition becomes `True`, the Node Readiness Controller will remove the taint, and workloads will be scheduled. |
0 commit comments