Description
Description
What problem are you trying to solve?
I am trying to reduce the frequency of consolidation on clusters that have frequent but insignificant resource request changes.
An active cluster can cause frequent consolidation events.
For example, if a deploy with HPA scales up and down one replica every 10 minutes, it's very likely that a new node will be spun up and then spun down every 10 minutes, such that cost is optimized. This could even result in a packed node getting deleted, if Karpenter decides that a different node type or number of nodes would be more cost efficient.
That can be really disruptive. PDBs help, but in order for them to guard against users experiencing slowness you'd need to set a PDB of practically 1% maxUnavailable.
Once a consolidationPolicy of WhenUnderutilized works alongside consolidateAfter, that will help out greatly, but it would still result in consolidation likely happening every (for example) 2 hours, even with very low net resource changes.
I think a way of configuring "consolidation tolerance" would help here. One implementation could be a way of specifying cost tolerance.
In pseudo-configuration, there could be a consolidationCostTolerance field that I might set as "$50 per hour".
If an HPA decides a deploy needs a new replica and there's no space, it would spin up a new combination of nodes that has enough space for all desired pods but is still cost effective. Later on, the HPA might decrement desired replicas. Karpenter would normally want to consolidate now since there's now a more cost effective combination of nodes for requested resources.
The idea is, consolidation would not happen unless currentCostPerHour - consolidatedCostPerHour is greater than $50.
This way, until there is a significant amount of unused resources on nodes, consolidation would not trigger.
How important is this feature to you?
This feature is fairly important. Even when all the features described in disruption controls become stable, existing solutions only reduce the frequency of consolidation, slow down consolidation, or block consolidation during certain hours.
We could set a PDB on all deploys of 1% maxUnavailable, but that feels like a pretty extreme demand.
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Activity