Add a gracePeriod for the `do-not-disrupt` pod annotation

[stevehipwell]

Tell us about your request
I'd like the ability to taint a node after a given period via ttlSecondsUntilTainted to allow running pods to finish before the node is terminated, this should still respect the ttlSecondsUntilExpired and ttlSecondsAfterEmpty.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
When running jobs such as CI/CD pipelines in K8s there are long running jobs which shouldn't be terminated due to the high cost of re-running them, by adding ttlSecondsUntilTainted we can have nodes that expire and are replaced without the cost of killing potentially long running jobs.

Are you currently working around this issue?
Not using Karpenter.

Additional context
n/a

Attachments
n/a

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ellistarn]

Does annotating the pods of your longrunning jobs with karpenter.sh/do-not-evict work for you?

[stevehipwell]

@ellistarn wouldn't this allow more jobs to schedule onto the node and basically make it unterminatable?

[ellistarn]

The node will still expire and cordon, but it won't start draining until the pod with those annotations are done. @njtran can you confirm?

[njtran]

Yep! If you have a pod with a do-not-evict annotation, Karpenter will not drain any of the pods on a node, but it will still cordon the node.

A node is not drained if it has a do-not-evict pod, and we will not terminate a node until that node is considered drained:

• https://github.com/aws/karpenter/blob/bb65375ae4f1a6ca1a4a30f217162abf71bd5dcc/pkg/controllers/termination/controller.go#L85-L91
• https://github.com/aws/karpenter/blob/bb65375ae4f1a6ca1a4a30f217162abf71bd5dcc/pkg/controllers/termination/terminate.go#L68-L70

Does this fit your use case?

[stevehipwell]

@njtran this fits my use case happy path perfectly, but it opens up a separate concern that the do-not-evict annotation could keep a node running indefinitely? If there isn't a way to do this already I'm happy to close this issue as being a "didn't read the docs correctly" and open a new issue regarding a do-not-evict grace period?

[njtran]

Sorry for the delay! You're correct in that if a do-not-evict pod exists, then Karpenter would never try to scale down the node.

open a new issue regarding a do-not-evict grace period?

Could you expand on what you mean here? What's the use-case for having an additional grace period for a do-not-evict pod? You could programmatically delete these pods once they've completed, which in that case, the node would then be able to be deleted if there were no other do-not-evict pods.

[stevehipwell]

@njtran fundamentally it comes down to reducing the blast radius. If anyone who can start a container can keep a node running that starts to look like a potential problem. I'd like to know that after a given time, no matter what a user has done, that a node will terminate and be replaced.

[ellistarn]

additional grace period for a do-not-evict

Is there a difference between this and terminationGracePeriodSeconds?

[tzneal]

I'd like to know that after a given time, no matter what a user has done, that a node will terminate and be replaced.

Even if it would violate a pod disruption budget?

[stevehipwell]

Is there a difference between this and terminationGracePeriodSeconds?

@ellistarn this would terminate the node even if there was a pod with a karpenter.sh/do-not-evict annotation running. Effectively after the TTL expires the annotation would be ignored.

Even if it would violate a pod disruption budget?

@tzneal I think that's a separate concern, but pretty important. A TTL which guarantees the node is terminated would make sense, having pods stuck due to user error and blocking node maintenance is a common problem when running a platform.

[njtran]

Renamed this issue to be more accurate of the discussion above