kube api server getting rate limited due to karpenter not respecting the do not evict annotations

[doverk96]

Description

Observed Behavior:
I have a nodepool which is set to expireAfter 720h, consolidate : whenEmpty. Still, I see that karpenter is giving error that it can not consolidate the node because PDB prevents it. Then, i went and put annotation 'karpenter.sh/do-not-disrupt: "true"' on all pods which are related to PDB. Still, i can see in events and karpenter logs that it is giving message cannot consolidate.
Expected Behavior:
Expected behaviour is if consolidate : whenEmpty, then it should just ignore it, instead of trying to evict the running workloads(other than daemonset pods) from the node. In case of annotation, again it should not try to disrupt, should not go for consolidation. By the definition, nodes with 'karpenter.sh/do-not-disrupt: "true"' annotations are not to be considered for consolidation.
Reproduction Steps (Please include YAML):
Just place a 'karpenter.sh/do-not-disrupt: "true"' annotation on a pod of any nodepool and check the events and logs of karpenter.
Versions:

• Chart Version: 1.1.0
• Kubernetes Version (kubectl version): 1.29 (EKS)

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[rknightion]

The docs were recently updated I think to make it clearer that do-not-disrupt will not prevent a pod from evicting when a node is set to expire with expireAfter https://github.com/aws/karpenter-provider-aws/pull/7474/files ?

[doverk96]

But why is it doing consolidation on nodes which are 10 days old only, when expireAfter is set to 720h and consolidateWhen set to whenEmpty?
Even I did set consolidateAfter to never, even then it is giving error in kubectl events : not able to disrupt the nodes.
Nodes without expiry should respect the annotation

[jmdeal]

Could you provide an example of the event? This sounds like expected behavior, as you've called out if there's a pod scheduled with a do-not-disrupt annotation Karpenter won't be able to consolidation the node. That's what that event indicates.

[doverk96]

Could you provide an example of the event? This sounds like expected behavior, as you've called out if there's a pod scheduled with a do-not-disrupt annotation Karpenter won't be able to consolidation the node. That's what that event indicates.

But in that case, it should not make call to eviction endpoint. And if this is the expected behaviour, then is there any way we can configure Karpenter not to hog the api server with these calls ? Like, something that can be done on a specific nodepool or ec2nodeclass so that those nodes are consolidated only when they are at expiry else just do not consider them, do not make any calls for eviction to kube api server. Right now, challenge we are facing is that we are getting too many 429 requests error when we checked in kube-apiserver-audit-logs. Due to these repeated eviction calls, number of calls are increasing and some of our automation around kubectl is getting hampered.
Sharing the sample entry from audit logs.
[ { "@timestamp": "2025-01-28 06:26:18.329", "responseStatus.code": "429", "requestURI": "/api/v1/namespaces/namespace1/pods/redis-cluster-4/eviction", "userAgent": "karpenter/1.1.0" }, { "@timestamp": "2025-01-28 06:26:18.079", "responseStatus.code": "429", "requestURI": "/api/v1/namespaces/namespace2/pods/zk-0/eviction", "userAgent": "karpenter/1.1.0" } ]

[jmdeal]

If Karpenter is creating eviction requests, that means the node has already been disrupted. Changes to your consolidation policy or adding do-not-disrupt annotations won't "undisrupt" the node - that's a one way transition. Also, 429 is the expected response from the API server when eviction is blocked by PDBs, this doesn't actually mean Karpenter is hammering the API server (ref). This does seem like an unfortunate choice if you're trying to check for 429s to detect rate limiting.

some of our automation around kubectl is getting hampered

Can you elaborate on this? It's not clear to me how this relates to Karpenter's behavior.

[tatyana12345]

Hi! It seems we also encountered a similar situation, many of our pods have the annotation karpenter.sh/do-not-disrupt: "true", but nevertheless karpenter deleted the node on which these pods were launched. Reason: AMIDrift
Example log:

{"level":"INFO","time":"2025-02-05T06:01:03.248Z","logger":"controller","caller":"disruption/controller.go:183","message":"disrupting nodeclaim(s) via replace, terminating 1 nodes (7 pods) ip-xx-xx-xx-58.ec2.internal/m5.xlarge/on-demand and replacing with on-demand node from types m5.xlarge, m5.2xlarge","commit":"6174c75","controller":"disruption","namespace":"","name":"","reconcileID":"bebd3e56-c70c-4d3c-9a01-6de66e1e6d78","command-id":"c346a6c4-f550-4cba-ae2e-ebde75f4f8f9","reason":"drifted"}

{"level":"DEBUG","time":"2025-02-05T05:27:37.777Z","logger":"controller","caller":"disruption/controller.go:91","message":"marking drifted","commit":"6174c75","controller":"nodeclaim.disruption","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"ondemand-pt85t"},"na
```mespace":"","name":"ondemand-pt85t","reconcileID":"962b44d0-4ee5-48d0-b649-083bbee95d7c","reason":"AMIDrift"}

If I understand the official documentation correctly, then this annotation should block nodes from being deleted during drift - but this does not happen.

EKS version 1.31, karpenter/controller version:1.0.6