Question: Drift, Disruption and Pods That Cannot Be Evicted

[jcstanaway]

As I read through the disruption documentation, it was my impression that even if a nodeClaim had drifted, karpenter would not disrupt the node if there were pods which cannot be evicted.

Karpenter will mark nodes as drifted and disrupt nodes that have drifted from their desired specification.

If there are pods that cannot be evicted on the node, Karpenter will ignore the node and try disrupting it later.

However, when looking at the description for terminationGracePeriod in the nodepools CRD, it states:

When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached.

The CRD description appears to be in conflict with the documentation as the documentation indicates that the node would NOT be drained if there are pods blocking eviction.

For our use case, we do not want such nodes to be drained at that point (the behavior described in the documentation and not the CRD is what we want). We want to wait for the full expireAfter + terminationGracePeriod duration to elapse (as described in the example in TerminationGracePeriod).

Effectively, we want a nodePool where drifted nodes are blocked from being replaced by pods which cannot be evicted until the node expires.

So, am I misreading the doc and/or the CRD or is either the doc or CRD not in alignment with the actual behavior?

[jmdeal]

If a terminationGracePeriod is set, the behavior of drift changes. The default behavior (w/o TGP) is what you described first, where Karpenter will ignore nodes if any scheduled pods have blocking PDBs or the karpenter.sh/do-not-disrupt annotation. If TGP is configured, those nodes will be candidates for disruption via drift. Karpenter would begin to drain those nodes, but pods with blocking PDBs or the do-not-disrupt annotation would not be evicted from the node until the TGP is about to expire. The termination grace period section of the docs should be updated to call out this behavior explicitly.

For our use case, we do not want such nodes to be drained at that point (the behavior described in the documentation and not the CRD is what we want). We want to wait for the full expireAfter + terminationGracePeriod duration to elapse (as described in the example in TerminationGracePeriod).

It sounds like the behavior you're after is the behavior described in the CRD. Without this behavior, Karpenter would never disrupt the node in the first place. With this behavior in place, Karpenter will begin draining the node, but it will be unable to drain pods with blocking PDBs or the do-not-disrupt annotation until the TGP is about to expire, as described in the TGP example.

/kind support
/triage needs-information

[jcstanaway]

Thanks much for the explanation.

Regarding

The termination grace period section of the docs should be updated to call out this behavior explicitly.

That would be good. I don't know if you want to keep this issue open to track the need for that update, but as far as I'm concerned, my issue is resolved.

[jcstanaway]

I wanted to follow-up and make sure my understanding is correct...

• expiresAfter set to something long (30 - 90 days).
• terminationGracePeriod not set.
• At least some of the pods on the node can not be disrupted.

Prior to expiresAfter drift occurs. Per the description for the Disruption Controller my understanding is that Karpenter will ignore this node. Is that correct?

Later, expiration occurs. The description in the Expiration section simply says that Karpenter will disrupt the node. It is unclear precisely which steps are executed and if there are any blocking conditions. I'm looking at both the Disruption Controller and Termination Controller sections and it seems like Karpenter will initiate draining but since the node can not be fully drained that the termination of the node will be blocked until those blocking pods go away (assume they either self terminate or something else eventually deletes them). Is that correct?

[jmdeal]

Prior to expiresAfter drift occurs. Per the description for the Disruption Controller my understanding is that Karpenter will ignore this node. Is that correct?

Yes, without TGP configured, pods with the karpenter.sh/do-not-disrupt annotation would block drift.

The description in the Expiration section simply says that Karpenter will disrupt the node. It is unclear precisely which steps are executed and if there are any blocking conditions.

Disruption refers to the initiation of the termination process. When a node is disrupted, the karpenter.sh/disrupted:NoSchedule taint is applied and drain is initiated. You're correct that pods with the karpenter.sh/do-not-disrupt annotation will prevent drain from completing until they are terminated or go into a terminal state (e.g. Succeeded or Failed).

I've put up a PR to update the docs with the details around TGP and drift: aws/karpenter-provider-aws#7710

[jmdeal]

/kind documentation
/triage accepted

[jmdeal]

/remove-triage needs-information

[rschalo]

/remove-triage accepted
/triage solved

[jcstanaway]

Clarifications added by aws/karpenter-provider-aws#7710.