Karpenter issues with node NOT getting cleaned up properly and consistently

[chaitanya-blrone]

Description

Observed Behavior:
We are seeing cases where lets say a pod completes execution and the node utiilzation drops to about 10%, karpenter keeps the node idle for almost an hour, again some pod gets scheduled on that node and this iteration repeats till we see the node being cleaned up. So Why is karpenter not cleaning up the nodes immediately when it detects underutilization? I can confirm that NEITHER there are pods on that node with the annotation karpenter.sh/do-not-disrupt: true NOR any pdbs set which can prevent the node cleanup. This is our consolidationPolicy

consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 1m

Expected Behavior:
Nodes should get cleaned up within a few minutes of underutilization
Reproduction Steps (Please include YAML):

Versions:

• Chart Version: 1.5.2
• Kubernetes Version (kubectl version): 1.32

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[saurav-agarwalla]

Are there any disruption budgets that you have set on the nodepool which could cause this delay? If not, can you share the logs? That should tell us why Karpenter isn't consolidating this node.