Do not evict running jobs

[runningman84]

Tell us about your request

Right now the consolidation replaces nodes with running jobs. This breaks a lot of deployments (doing db upgrades or other stuff). One way to handle this issue is putting do not eviction annotations on the corresponding pods. But it is difficult to ensure that if you have dozens of teams using a given cluster.

Could this be handled in a more generic way?

All these jobs look like this:
Controlled By: Job/xxx-yyy-zzz-28031590

Can karpenter optionally just use the do not evict behavior for running jobs?

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

It is difficult to ensure all pods have the do not evict annotation.

Are you currently working around this issue?

Manually set do not evict to all corresponding pods.

Additional Context

No response

Attachments

No response

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[engedaam]

Can you describe your use-case more? Would it not be possible to pass the annotation using the spec.template in the deployment?

[tzneal]

If you use Kyverno, there's an existing Kyverno policy to add the do-not-evict annotation to Job/CronJob pods automatically. https://kyverno.io/policies/karpenter/add-karpenter-donot-evict/add-karpenter-donot-evict/

[ellistarn]

Were considering this request as part of a batch of eviction based features. @njtran to follow up.

[Almenon]

Just ran into a similar issue, but with self-hosted CircleCI pods. They basically act the same as jobs (should only be run once). When Karpenter kills the node the pods are killed and don't come back. This issue was took a long time to figure out because I saw exit code 137 events from containerd, which normally indicates OOM, but in this case the culprit was actually Karpenter.

This is worth a warning in the docs, IMO

[ellistarn]

Related: aws/karpenter-provider-aws#2391 (comment)

[johngmyers]

We create a PodDisruptionBudget which allows for zero disruptions and has all such jobs in its selector.

[ellistarn]

@johngmyers , can you post an example of the PDB you're using? I'm a big fan of driving eviction behavior through the PDB API, if at all possible.

[johngmyers]

I don't have access to it at the moment. It has a large spec.minAvailable and has a selector for a label that we put on all such jobs. It basically makes the label equivalent to the do-not-evict annotation, except being recognized by the voluntary eviction API.

[HansBraun]

We have the same problem with tekton pipline taskrun pods. They get evicted after karpenter has cordoned a node. The PipeLine has the annotation "karpenter.sh/do-not-evict: true" and passes it to the taskruns, but some taskrun pods got still evicted, after karapenter consolidates a node.

[lzyli]

We have the same problem with cronjobs. We're running jobs on some dedicated nodes, if we use "karpenter.sh/do-not-evict: true" or PDB, Karpenter just skips nodes and does nothing. Nodes are never deprovisioned. The deprovisioning flow for cronjobs should be something like this: cordon nodes -> waiting for cronjobs finish -> drain nodes.

[yangwwei]

We have the same problem with cronjobs. We're running jobs on some dedicated nodes, if we use "karpenter.sh/do-not-evict: true" or PDB, Karpenter just skips nodes and does nothing. Nodes are never deprovisioned. The deprovisioning flow for cronjobs should be something like this: cordon nodes -> waiting for cronjobs finish -> drain nodes.

100% agree with this