Does Not Consolidate On-Demand Nodes Back to Spot When Capacity Stabilizes

[sai9920]

Description

Observed Behavior:

Once on-demand nodes are created due to capacity constraints, they remain in the cluster even after spot capacity becomes available. Karpenter does not automatically replace on-demand nodes with spot instances when conditions improve. Instead, manual intervention (draining the nodes) is required to trigger the transition back to spot instances.

As a result, when spot capacity runs out, the workload shifts to on-demand nodes, but it never moves back to spot instances even after the spot capacity is restored.

`  - name: default
    expireAfter: 720h
    terminationGracePeriod: 5m
    nodeClass: default
    taints:
      - key: default
        effect: NoSchedule
    labels:
      usage: default
    instanceRequirements:
    - key: "node.kubernetes.io/instance-type"
      operator: In
      values:
      - c6i.8xlarge
      - c6i.12xlarge
      - c6i.16xlarge
      - c7i.8xlarge
      - c7i.12xlarge
      - c7i.16xlarge
      - c7a.8xlarge
      - c7a.12xlarge
      - c7a.16xlarge
      - c6a.8xlarge
      - c6a.12xlarge
      - c6a.16xlarge
      - c6in.8xlarge
      - c6id.8xlarge
      - c6id.12xlarge
      - c6id.12xlarge
    - key: "topology.kubernetes.io/zone"
      operator: In
      values:
      - ap-southeast-1b
      - ap-southeast-1a
      - ap-southeast-1c
    - key: "kubernetes.io/arch"
      operator: In
      values:
      - amd64
    - key: "karpenter.sh/capacity-type"
      operator: In
      values:
      - spot
      - on-demand
    disruptionConfig:
      consolidationPolicy: WhenEmptyOrUnderutilized
      consolidateAfter: 300s
      budgets:
      - nodes: "50%"
    weight: 12
    limits:
      cpu: 5k
      memory: 4Ti`

Expected Behavior:

Karpenter should automatically consolidate and replace on-demand nodes with spot instances when they become available, optimizing cost efficiency.

Reproduction Steps (Please include YAML):

1.	Deploy Karpenter with the following node pool configuration, which supports both on-demand and spot instances.
2.	Simulate a scenario where spot capacity runs out, causing Karpenter to provision on-demand instances.
3.	Allow spot capacity to become available again.
4.	Observe that Karpenter does not consolidate on-demand instances and replace them with spot instances automatically.

Versions: 1.0.6

• Chart Version: 1.0.6
• Kubernetes Version (kubectl version): 1.31

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

Can you enable debug logging and share your logs? I'd be interested to see if you have any logs that say marking consolidatable or removing consolidatable status condition since that last one is an indication that you have too much cluster churn which is part of the reason that Karpenter isn't able to make a consolidation decision.

You should also be able to look at events on your NodeClaims/Nodes to see why Karpenter thinks that it can't consolidate.

[sai9920]

@jonathan-innis

I see below events

Events:
  Type    Reason            Age                  From       Message
  ----    ------            ----                 ----       -------
  Normal  Unconsolidatable  60m (x31 over 10h)   karpenter  Can't remove without creating 5 candidates
  Normal  Unconsolidatable  45m (x2 over 9h)     karpenter  Can't remove without creating 7 candidates
  Normal  Unconsolidatable  8m21s (x2 over 24m)  karpenter  Can't remove without creating 5 candidates

Logs:

  level: DEBUG
   logger: controller
   message: marking consolidatable

i also see quite a few

  caller: disruption/controller.go:91
   commit: 6174c75
   controller: nodeclaim.disruption
   controllerGroup: karpenter.sh
   controllerKind: NodeClaim
   level: DEBUG
   logger: controller
   message: removing consolidatable status condition

Any pointers on how to handle them so these can complete a successful consolidation and move to spot

[jmdeal]

Do you have any preferred constraints (e.g. preferred anti-affinity or topology spread constraints)? Since Karpenter respects these preferences if they are currently violated Karpenter won't be able to consolidate even if the node is underutilized. That could explain the events you've seen.