You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: designs/termination.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,7 +20,7 @@ The termination controller is responsible for gracefully terminating instances.
20
20
The current termination workflow finds nodes with the label `karpenter.sh/lifecycle-phase: terminable` then cordons, drains, and deletes them. This naively handles a few error cases, has no user safeguards, and makes no effort to rate limit. The new workflow will improve these but continue to monitor nodes with the same labels.
21
21
22
22
The new termination process will begin with a node that receives a delete request. After Karpenter validates the request, a Karpenter mutating webhook will add the Karpenter [finalizer](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#finalizers) to the node. Then, we cordon and begin draining the node. After no pods remain, we terminate the instance in the cloud provider, then remove the Karpenter finalizer. This will result in the APIServer deleting the node object.
The current termination process acts on a reconcile loop. We will change the termination controller to watch nodes and manage the Karpenter [finalizer](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#finalizers), making it responsible for all node termination and pod eviction logic.
Copy file name to clipboardExpand all lines: website/content/en/preview/concepts/disruption.md
+116-7Lines changed: 116 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -28,13 +28,122 @@ Karpenter automatically discovers disruptable nodes and spins up replacements wh
28
28
29
29
### Termination Controller
30
30
31
-
When a Karpenter node is deleted, the Karpenter finalizer will block deletion and the APIServer will set the `DeletionTimestamp` on the node, allowing Karpenter to gracefully shutdown the node, modeled after [Kubernetes Graceful Node Shutdown](https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/#graceful-node-shutdown). Karpenter's graceful shutdown process will:
32
-
1. Add the `karpenter.sh/disrupted:NoSchedule` taint to the node to prevent pods from scheduling to it.
33
-
2. Begin evicting the pods on the node with the [Kubernetes Eviction API](https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/) to respect PDBs, while ignoring all [static pods](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/), pods tolerating the `karpenter.sh/disrupted:NoSchedule` taint, and succeeded/failed pods. Wait for the node to be fully drained before proceeding to Step (3).
34
-
* While waiting, if the underlying NodeClaim for the node no longer exists, remove the finalizer to allow the APIServer to delete the node, completing termination.
35
-
3. Verify that all [VolumeAttachment](https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/volume-attachment-v1/) resources for drain-able pods are deleted.
36
-
4. Terminate the NodeClaim in the Cloud Provider.
37
-
5. Remove the finalizer from the node to allow the APIServer to delete the node, completing termination.
31
+
When a Karpenter node is deleted, the `karpenter.sh/termination` finalizer blocks deletion while the Termination Controller gracefully drains the node, waits for its volumes to detach, and terminates the underlying instance. This is modeled after [Kubernetes Graceful Node Shutdown](https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/#graceful-node-shutdown).
32
+
33
+
#### Actors
34
+
35
+
Node termination is coordinated by several controllers, each with a distinct responsibility:
36
+
37
+
| Actor | Responsibility |
38
+
| ----- | -------------- |
39
+
|**NodeClaim Lifecycle Controller**| Watches NodeClaims. When a NodeClaim gets a `DeletionTimestamp`, stamps the `karpenter.sh/nodeclaim-termination-timestamp` annotation on the NodeClaim (equal to `DeletionTimestamp + spec.terminationGracePeriod`, if set) and deletes the associated Node so termination can begin. After the Node's finalizer is removed, ensures the cloud instance is deleted and removes the NodeClaim's finalizer. |
40
+
|**Node Termination Controller**| Watches Nodes with a `DeletionTimestamp`. Adds the `karpenter.sh/disrupted:NoSchedule` taint, then progresses the node through drain, volume detachment, and instance termination. Publishes progress as NodeClaim status conditions (`Drained`, `VolumesDetached`, `InstanceTerminating`). |
41
+
|**Terminator**| Helper used by the Node Termination Controller. Applies taints, groups pods by [eviction priority](#pod-eviction-priority), and hands evictable pods to the Eviction Queue. Also proactively deletes pods whose `terminationGracePeriodSeconds` would otherwise overrun the Node's `terminationGracePeriod`. |
42
+
|**Eviction Queue**| Rate-limited worker queue that calls the [Kubernetes Eviction API](https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/) for each pod with exponential backoff. `429 Too Many Requests` (a blocking PDB) and pods with multiple PDBs on them cause the pod to be re-queued; `404` and `409` responses drop the pod from the queue. Karpenter keeps retrying until the pod evicts or the Node's `terminationGracePeriod` elapses, at which point the pod is force-deleted. |
43
+
|**Node Health Controller**| Runs when [Node Auto Repair]({{<ref "#node-auto-repair" >}}) is enabled. For unhealthy nodes, overwrites the `karpenter.sh/nodeclaim-termination-timestamp` annotation with the current time, forcing the Termination Controller to skip drain and volume-detachment waits. |
44
+
45
+
#### Workflow
46
+
47
+
For a node with a `DeletionTimestamp`, the Termination Controller runs the following steps in order. Each phase is reflected as a status condition on the owning NodeClaim.
48
+
49
+
1.**Taint.** Add the `karpenter.sh/disrupted:NoSchedule` taint to prevent new pods from scheduling to the node, and add the `node.kubernetes.io/exclude-from-external-load-balancers=karpenter` label so the node is removed from load balancer target groups before it stops serving traffic.
50
+
2.**Fast path for missing instances.** If the Node's `Ready` condition is not `True`**and** the underlying instance is no longer visible to the cloud provider, remove the finalizer immediately and skip the remaining phases. See [Fast path when the instance disappears](#fast-path-when-the-instance-disappears).
51
+
3.**Drain** (`Drained: Unknown` with reason `Draining` → `Drained: True`). Evict all drainable pods via the [Kubernetes Eviction API](https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/), respecting PDBs. Pods are grouped by [priority](#pod-eviction-priority) and evicted one group at a time. Pods tolerating the `karpenter.sh/disrupted:NoSchedule` taint, [static pods](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/), and succeeded/failed pods are skipped. If the underlying NodeClaim disappears mid-drain, the Node's finalizer is removed and termination completes.
52
+
4.**Volume detachment** (`VolumesDetached: Unknown` with reason `AwaitingVolumeDetachment` → `VolumesDetached: True`, or `VolumesDetached: False` with reason `TerminationGracePeriodElapsed`). Wait for all [VolumeAttachment](https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/volume-attachment-v1/) resources belonging to drainable pods to be deleted by the Kubernetes attach-detach controller. Volumes belonging to non-drainable pods are not waited on. See [Volume detachment](#volume-detachment).
53
+
5.**Instance termination** (`InstanceTerminating: True`). Call the cloud provider to terminate the instance, requeuing every 5 seconds until the cloud provider reports the instance as gone.
54
+
6.**Finalizer removal.** Remove the `karpenter.sh/termination` finalizer from the Node so the API server can delete it. The NodeClaim Lifecycle Controller then removes the finalizer from the NodeClaim.
55
+
56
+
#### State diagram
57
+
58
+
NodeClaim status conditions progress through the following states during termination. Reason strings shown are the values set by the controller. Transitions also emit Kubernetes events on the Node.
(Reason: TerminationGracePeriodElapsed) with pending attachments)
89
+
│
90
+
▼
91
+
InstanceTerminating: True
92
+
│ cloud provider reports instance gone
93
+
▼
94
+
finalizer removed
95
+
```
96
+
97
+
#### Pod eviction priority
98
+
99
+
The Terminator drains pods in strict priority order, matching the [Kubernetes Graceful Node Shutdown](https://kubernetes.io/docs/concepts/architecture/nodes/#graceful-node-shutdown) ordering. Pods are placed into one of four groups, and a group with any remaining pods blocks all lower-priority groups until it fully drains:
100
+
101
+
1.**Non-critical, non-DaemonSet** — regular workload pods without a system priority class.
102
+
2.**Non-critical, DaemonSet** — DaemonSet pods without a system priority class.
103
+
3.**Critical, non-DaemonSet** — pods with `priorityClassName` of `system-cluster-critical` or `system-node-critical` that are not owned by a DaemonSet.
104
+
4.**Critical, DaemonSet** — DaemonSet pods with `system-cluster-critical` or `system-node-critical`.
105
+
106
+
A pod is considered evictable if it is not a static pod, does not tolerate the `karpenter.sh/disrupted:NoSchedule` taint, is not already terminating, and has not succeeded or failed. Pods with blocking PDBs remain in the Eviction Queue until the PDB allows the eviction or the Node's `terminationGracePeriod` elapses, at which point they are force-deleted; see [`terminationGracePeriod`]({{<ref "#terminationgraceperiod" >}}).
107
+
108
+
#### Volume detachment
109
+
110
+
After drain completes, Karpenter waits for the [VolumeAttachment](https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/volume-attachment-v1/) resources for drainable pods to be deleted by the Kubernetes attach-detach controller before terminating the instance. This gives workloads that use PersistentVolumes a chance to migrate cleanly: the volume can be attached to a replacement node without waiting for the (up to several minute) forced-detach timeout that would otherwise apply after the instance disappears.
111
+
112
+
Karpenter only waits on volumes belonging to drainable pods. Volumes used by pods that were ignored during drain (static, taint-tolerating, succeeded, or failed) do not block termination.
113
+
114
+
If the NodeClaim's `terminationGracePeriod` elapses before all volumes detach, the wait is skipped, the `VolumesDetached` condition is set to `False` with reason `TerminationGracePeriodElapsed`, and termination proceeds. Any remaining volumes are force-detached by the cloud provider once the instance is gone.
115
+
116
+
#### Fast path when the instance disappears
117
+
118
+
If the underlying instance disappears before drain completes — for example, because it was manually terminated in the cloud provider console or reclaimed via spot interruption — Karpenter takes a fast path to avoid waiting on a kubelet that can no longer report pod status. When both of the following are true, the Node's finalizer is removed immediately without draining or waiting on volumes:
119
+
120
+
* The Node's `Ready` condition is not `True`.
121
+
* The cloud provider reports the instance as not found.
122
+
123
+
Karpenter checks `Ready` before consulting the cloud provider because eventual consistency in the cloud provider's API can briefly report an instance as gone while the kubelet is still healthy.
124
+
125
+
#### How terminationGracePeriod is determined
126
+
127
+
The `terminationGracePeriod` used at termination time is stored on the NodeClaim as the `karpenter.sh/nodeclaim-termination-timestamp` annotation (an RFC3339 timestamp). The annotation is set by:
128
+
129
+
***NodePool inheritance.** When a NodeClaim is created, `spec.template.spec.terminationGracePeriod` on the owning NodePool is copied to the NodeClaim's `spec.terminationGracePeriod`. When the NodeClaim gets a `DeletionTimestamp`, the NodeClaim Lifecycle Controller stamps the annotation as `DeletionTimestamp + spec.terminationGracePeriod`. See [`terminationGracePeriod`]({{<ref "#terminationgraceperiod" >}}) for the user-facing configuration.
130
+
***Node Auto Repair override.** When [Node Auto Repair]({{<ref "#node-auto-repair" >}}) determines a node is unhealthy, the Health Controller overwrites the annotation with the current time, forcing the Termination Controller to skip drain and volume-detachment waits. If the annotation is already in the past, it is not overwritten.
131
+
***Unset.** If neither the NodePool nor the Health Controller sets a `terminationGracePeriod`, the annotation is absent and Karpenter waits indefinitely for drain and volume detachment.
132
+
133
+
#### terminationGracePeriod timeline
134
+
135
+
The table below traces how the Node's `terminationGracePeriod`, a pod's `terminationGracePeriodSeconds`, and the Node's `DeletionTimestamp` interact. In this example the Node has `terminationGracePeriod: 1h` and a pod on it has `terminationGracePeriodSeconds: 900` (15 minutes).
136
+
137
+
| Time | Event |
138
+
| ---- | ----- |
139
+
|`t=0`|`Node.DeletionTimestamp` set. NodeClaim Lifecycle Controller stamps the `karpenter.sh/nodeclaim-termination-timestamp` annotation as `t + 1h`. Node Termination Controller taints the node, sets `Drained: Unknown` (`Reason: Draining`), and begins evicting pods via the Eviction API. |
140
+
|`t=45m`| The Terminator sees that the pod is 15 minutes away from the annotation (`900s` before `t + 1h`), and calls `Delete` on the pod with `gracePeriodSeconds=900`. The pod's kubelet now has its full grace period to shut down before the node dies. |
141
+
|`t=60m`| The Node's `terminationGracePeriod` elapses. The volume-detach wait is skipped if pending (`VolumesDetached: False` with `TerminationGracePeriodElapsed`); the cloud provider is called to terminate the instance even if pods or volumes remain. Node finalizer removed once the instance is gone. |
142
+
143
+
Two important properties follow from this design:
144
+
145
+
***Pod grace periods are respected up to the Node's grace period.** If a pod's `terminationGracePeriodSeconds` is shorter than the Node's remaining time, the pod gets its full grace period. If it is longer, Karpenter clamps the pod's grace period to the Node's remaining time (minimum 1 second, to preserve at-most-one-pod semantics).
146
+
***Blocking pods do not extend the Node's lifetime.** Pods with `karpenter.sh/do-not-disrupt` or blocking PDBs are proactively deleted before the Node's grace period expires, so they cannot indefinitely delay termination. See [`terminationGracePeriod`]({{<ref "#terminationgraceperiod" >}}) for examples.
38
147
39
148
## Manual Methods
40
149
***Node Deletion**: You can use `kubectl` to manually remove a single Karpenter node or nodeclaim. Since each Karpenter node is owned by a NodeClaim, deleting either the node or the nodeclaim will cause cascade deletion of the other:
0 commit comments