Does Not Consolidate On-Demand Nodes Back to Spot When Capacity Stabilizes

[sai9920]

Description

Observed Behavior:

Once on-demand nodes are created due to capacity constraints, they remain in the cluster even after spot capacity becomes available. Karpenter does not automatically replace on-demand nodes with spot instances when conditions improve. Instead, manual intervention (draining the nodes) is required to trigger the transition back to spot instances.

As a result, when spot capacity runs out, the workload shifts to on-demand nodes, but it never moves back to spot instances even after the spot capacity is restored.

`  - name: default
    expireAfter: 720h
    terminationGracePeriod: 5m
    nodeClass: default
    taints:
      - key: default
        effect: NoSchedule
    labels:
      usage: default
    instanceRequirements:
    - key: "node.kubernetes.io/instance-type"
      operator: In
      values:
      - c6i.8xlarge
      - c6i.12xlarge
      - c6i.16xlarge
      - c7i.8xlarge
      - c7i.12xlarge
      - c7i.16xlarge
      - c7a.8xlarge
      - c7a.12xlarge
      - c7a.16xlarge
      - c6a.8xlarge
      - c6a.12xlarge
      - c6a.16xlarge
      - c6in.8xlarge
      - c6id.8xlarge
      - c6id.12xlarge
      - c6id.12xlarge
    - key: "topology.kubernetes.io/zone"
      operator: In
      values:
      - ap-southeast-1b
      - ap-southeast-1a
      - ap-southeast-1c
    - key: "kubernetes.io/arch"
      operator: In
      values:
      - amd64
    - key: "karpenter.sh/capacity-type"
      operator: In
      values:
      - spot
      - on-demand
    disruptionConfig:
      consolidationPolicy: WhenEmptyOrUnderutilized
      consolidateAfter: 300s
      budgets:
      - nodes: "50%"
    weight: 12
    limits:
      cpu: 5k
      memory: 4Ti`

Expected Behavior:

Karpenter should automatically consolidate and replace on-demand nodes with spot instances when they become available, optimizing cost efficiency.

Reproduction Steps (Please include YAML):

1.	Deploy Karpenter with the following node pool configuration, which supports both on-demand and spot instances.
2.	Simulate a scenario where spot capacity runs out, causing Karpenter to provision on-demand instances.
3.	Allow spot capacity to become available again.
4.	Observe that Karpenter does not consolidate on-demand instances and replace them with spot instances automatically.

Versions: 1.0.6

• Chart Version: 1.0.6
• Kubernetes Version (kubectl version): 1.31

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

Can you enable debug logging and share your logs? I'd be interested to see if you have any logs that say marking consolidatable or removing consolidatable status condition since that last one is an indication that you have too much cluster churn which is part of the reason that Karpenter isn't able to make a consolidation decision.

You should also be able to look at events on your NodeClaims/Nodes to see why Karpenter thinks that it can't consolidate.

[sai9920]

@jonathan-innis

I see below events

Events:
  Type    Reason            Age                  From       Message
  ----    ------            ----                 ----       -------
  Normal  Unconsolidatable  60m (x31 over 10h)   karpenter  Can't remove without creating 5 candidates
  Normal  Unconsolidatable  45m (x2 over 9h)     karpenter  Can't remove without creating 7 candidates
  Normal  Unconsolidatable  8m21s (x2 over 24m)  karpenter  Can't remove without creating 5 candidates

Logs:

  level: DEBUG
   logger: controller
   message: marking consolidatable

i also see quite a few

  caller: disruption/controller.go:91
   commit: 6174c75
   controller: nodeclaim.disruption
   controllerGroup: karpenter.sh
   controllerKind: NodeClaim
   level: DEBUG
   logger: controller
   message: removing consolidatable status condition

Any pointers on how to handle them so these can complete a successful consolidation and move to spot

[jmdeal]

Do you have any preferred constraints (e.g. preferred anti-affinity or topology spread constraints)? Since Karpenter respects these preferences if they are currently violated Karpenter won't be able to consolidate even if the node is underutilized. That could explain the events you've seen.

[jonathan-innis]

Any pointers on how to handle them so these can complete a successful consolidation and move to spot

Removing the status condition is going to be associated with pod churn that you see on your nodes. Basically, if you have pods removed or added to a given node within the last 5m (based on your config), then it's not going to be eligible for consolidation.

Still, the fact that you are getting events like the ones you shared above indicate that you are seeing Karpenter attempt a scheduling simulation and attempt to consolidate

Can you share some of the current NodeClaims (that are on-demand) and some application manifests from your cluster? That will give us a better idea what might be happening. At present, the best guess that I have is that you have large OD capacity and there isn't equivalent spot capacity sizing, meaning that Karpenter would have to split the large node into smaller ones to go to spot.

[sai9920]

At present, the best guess that I have is that you have large OD capacity and there isn't equivalent spot capacity sizing, meaning that Karpenter would have to split the large node into smaller ones to go to spot.

If this is true,even on forceful draining of on-demand node should not result in spot instance. But on forceful termination i can see a new spot instance of same size.

Below is my node pool configuration:

apiVersion: karpenter.sh/v1
kind: NodeClaim
metadata:
  annotations:
    compatibility.karpenter.k8s.aws/cluster-name-tagged: 'true'
    compatibility.karpenter.k8s.aws/kubelet-drift-hash: '15379597991425564585'
    karpenter.k8s.aws/ec2nodeclass-hash: '10535655070762040088'
    karpenter.k8s.aws/ec2nodeclass-hash-version: v3
    karpenter.k8s.aws/tagged: 'true'
    karpenter.sh/nodepool-hash: '13729567759945582771'
    karpenter.sh/nodepool-hash-version: v3
    karpenter.sh/stored-version-migrated: 'true'
  creationTimestamp: '2025-03-21T06:21:43Z'
  finalizers:
    - karpenter.sh/termination
  generateName: dev-
  generation: 1
  labels:
    karpenter.k8s.aws/instance-category: c
    karpenter.k8s.aws/instance-cpu: '64'
    karpenter.k8s.aws/instance-cpu-manufacturer: amd
    karpenter.k8s.aws/instance-ebs-bandwidth: '20000'
    karpenter.k8s.aws/instance-encryption-in-transit-supported: 'true'
    karpenter.k8s.aws/instance-family: c6a
    karpenter.k8s.aws/instance-generation: '6'
    karpenter.k8s.aws/instance-hypervisor: nitro
    karpenter.k8s.aws/instance-memory: '131072'
    karpenter.k8s.aws/instance-network-bandwidth: '25000'
    karpenter.k8s.aws/instance-size: 16xlarge
    karpenter.sh/capacity-type: on-demand
    karpenter.sh/nodepool: global-dev
    kubernetes.io/arch: amd64
    kubernetes.io/os: linux
    node.kubernetes.io/instance-type: c6a.16xlarge
    provisioned-by: karpenter
    topology.k8s.aws/zone-id: apse1-az1
    topology.kubernetes.io/region: ap-southeast-1
    topology.kubernetes.io/zone: ap-southeast-1b
    usage:dev
  name: dev-gk7fn
  ownerReferences:
    - apiVersion: karpenter.sh/v1
      blockOwnerDeletion: true
      kind: NodePool
      name: dev
      uid: 05b43274-8546-4497-aca8-5350e53af6ae
  resourceVersion: '122791972'
  uid: 19edce8a-9eff-4ccf-ae4d-e1d481d39963
spec:
  expireAfter: 720h
  nodeClassRef:
    group: karpenter.k8s.aws
    kind: EC2NodeClass
    name:dev
  requirements:
    - key: kubernetes.io/arch
      operator: In
      values:
        - amd64
    - key: karpenter.sh/capacity-type
      operator: In
      values:
        - on-demand
        - spot
    - key: karpenter.sh/nodepool
      operator: In
      values:
        - global-dev
    - key: provisioned-by
      operator: In
      values:
        - karpenter
    - key: usage
      operator: In
      values:
        - global-dev
    - key: node.kubernetes.io/instance-type
      operator: In
      values:
        - c6a.16xlarge
        - c6i.16xlarge
        - c6id.16xlarge
        - c7i.16xlarge
        - m6a.16xlarge
        - m6i.16xlarge
        - m7i.16xlarge
        - r6a.16xlarge
        - r6i.16xlarge
        - r7i.16xlarge
    - key: topology.kubernetes.io/zone
      operator: In
      values:
        - ap-southeast-1a
        - ap-southeast-1b
        - ap-southeast-1c
  resources:
    requests:
      cpu: 51450m
      memory: 78839128Ki
      pods: '58'
  taints:
    - effect: NoSchedule
      key: amstack-sre/global-dev
  terminationGracePeriod: 5m0s
status:
  allocatable:
    cpu: 63770m
    ephemeral-storage: 35Gi
    memory: 112779Mi
    pods: '737'
    vpc.amazonaws.com/pod-eni: '107'
  capacity:
    cpu: '64'
    ephemeral-storage: 40Gi
    memory: 121241Mi
    pods: '737'
    vpc.amazonaws.com/pod-eni: '107'
  conditions:
    - lastTransitionTime: '2025-03-21T06:31:45Z'
      message: ''
      reason: ConsistentStateFound
      status: 'True'
      type: ConsistentStateFound
    - lastTransitionTime: '2025-03-21T06:22:38Z'
      message: ''
      reason: Initialized
      status: 'True'
      type: Initialized
    - lastTransitionTime: '2025-03-21T06:21:45Z'
      message: ''
      reason: Launched
      status: 'True'
      type: Launched
    - lastTransitionTime: '2025-03-21T06:22:38Z'
      message: ''
      reason: Ready
      status: 'True'
      type: Ready
    - lastTransitionTime: '2025-03-21T06:22:22Z'
      message: ''
      reason: Registered
      status: 'True'
      type: Registered
  imageID: ami-0a4d4869c365301f4
  lastPodEventTime: '2025-03-21T08:33:08Z'
  nodeName: ip-10-129-179-21.ap-southeast-1.compute.internal
  providerID: aws:///ap-southeast-1b/i-0870cee9fc0c62ccf

application rollout:
`apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    configmap.reloader.stakater.com/reload: test-dev-app-config
    kubectl.kubernetes.io/last-applied-configuration: >
    rollout.argoproj.io/revision: '63'
    secret.reloader.stakater.com/reload: test-secret
  creationTimestamp: '2025-01-21T07:24:10Z'
  generation: 136
  labels:
    app: deployment-appservice
    app.kubernetes.io/instance: test-deployment
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: test
    app.kubernetes.io/version: 1.0.0
    argocd.argoproj.io/instance: test-deployment--dev-blue
    env: -dev
    helm.sh/chart: microservice-1.6.23
    product: deployment
    project: test
    tier: app
    version: 1.0.0
  name: test
  namespace: deployment-dev
  resourceVersion: '122756053'
  uid: a15b4775-eb4c-431a-82fd-c6b51c92436b
spec:
  replicas: 1
  restartAt: '2025-03-21T05:54:07Z'
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: deployment-appservice
      app.kubernetes.io/instance: test-deployment
      app.kubernetes.io/name: test

      env: dev
      product: deployment
      project: test
      tier: app
  strategy:
    blueGreen:
      activeService: test
      autoPromotionEnabled: true
      autoPromotionSeconds: 5
      previewService: test-preview
  template:
    metadata:
      annotations:
        prometheus.io/path: /test/actuator/prometheus
        prometheus.io/port: '8080'
        reloader.stakater.com/last-reloaded-from: >-
          {"type":"CONFIGMAP","name":"test--dev-app-config","namespace":"deployment--dev","hash":"b28f0313e6fcbd5b7aa527744abdd44050ecbe0a","containerRefs":["test"],"observedAt":1742535510}
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: usage
                    operator: In
                    values:
                      - dev
      containers:
        - env:
            - name: MY_NODE_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
            - name: MY_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: APP_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.labels['app.kubernetes.io/name']
            - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
              value: >-
                http://otel-collector:4317
          image: >-
            123.dkr.ecr.ap-southeast-1.amazonaws.com/ap-southeast-1/teest:f60b62a8ceb823b4eaab095
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: test/actuator/health
              port: http
            periodSeconds: 10
            timeoutSeconds: 5
          name: test
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            failureThreshold: 5
            httpGet:
              path: test/actuator/health
              port: http
            periodSeconds: 10
            timeoutSeconds: 5
          resources:
            limits:
              cpu: '2'
              memory: 2Gi
            requests:
              cpu: '2'
              memory: 2Gi
          securityContext: {}
          startupProbe:
            failureThreshold: 12
            httpGet:
              path: test/actuator/health
              port: http
            periodSeconds: 10
            timeoutSeconds: 5
          volumeMounts:
            - mountPath: /etc/app/secrets/
              name: test-secrets
              readOnly: true
            - mountPath: /etc/app/config/
              name: extconfig-volume
              readOnly: true
      securityContext: {}
      serviceAccountName: test-sa
      tolerations:
        - effect: NoSchedule
          key: sre/-dev
          operator: Exists
      volumes:
        - csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: test-secretproviderclass
          name: test-secrets
        - configMap:
            name: test--dev-app-config
          name: extconfig-volume
status:
  HPAReplicas: 1
  availableReplicas: 1
  blueGreen:
    activeSelector: 5c99768b54
    previewSelector: 77c4bfb486
  canary: {}
  conditions:
    - lastTransitionTime: '2025-03-19T15:51:16Z'
      lastUpdateTime: '2025-03-19T15:51:16Z'
      message: Rollout is paused
      reason: RolloutPaused
      status: 'False'
      type: Paused
    - lastTransitionTime: '2025-03-20T04:32:44Z'
      lastUpdateTime: '2025-03-20T04:32:44Z'
      message: Rollout is not healthy
      reason: RolloutHealthy
      status: 'False'
      type: Healthy
    - lastTransitionTime: '2025-03-20T04:32:44Z'
      lastUpdateTime: '2025-03-20T04:32:44Z'
      message: RolloutCompleted
      reason: RolloutCompleted
      status: 'False'
      type: Completed
    - lastTransitionTime: '2025-03-21T06:25:36Z'
      lastUpdateTime: '2025-03-21T06:25:36Z'
      message: Rollout has minimum availability
      reason: AvailableReason
      status: 'True'
      type: Available
    - lastTransitionTime: '2025-03-21T08:11:52Z'
      lastUpdateTime: '2025-03-21T08:11:52Z'
      message: ReplicaSet "test-77c4bfb486" has timed out progressing.
      reason: ProgressDeadlineExceeded
      status: 'False'
      type: Progressing
  currentPodHash: 77c4bfb486
  message: >-
    ProgressDeadlineExceeded: ReplicaSet "test-77c4bfb486" has timed
    out progressing.
  observedGeneration: '136'
  phase: Degraded
  readyReplicas: 1
  replicas: 2
  restartedAt: '2025-03-21T05:54:07Z'
  selector: >-
  stableRS: 5c99768b54
  updatedReplicas: 1

`

[sai9920]

Do you have any preferred constraints (e.g. preferred anti-affinity or topology spread constraints)? Since Karpenter respects these preferences if they are currently violated Karpenter won't be able to consolidate even if the node is underutilized. That could explain the events you've seen.

But the manual drain of nodes are resulting in spot instances.

[dmarkhas]

We are seeing the same behavior - Karpenter launched multiple on-demand 32xlarge nodes, and is not replacing them with cheaper spot nodes - but if we terminate the nodes manually, Karpenter provisions new spot nodes.

I believe this is because Karpenter does not allow creating 2 nodes instead of 1 (even if it's cheaper).

https://karpenter.sh/docs/concepts/disruption/#consolidation

"Replace - A node can be replaced if all of its pods can run on a combination of free capacity of other nodes in the cluster and a single lower price replacement node."

[felipewnp]

I'm seeing this as well, and it's pretty annoying.

Once Karpenter deploys an on-demand instance, it will never be replaced by a spot instance again unless it is drained and removed manually.

[misterek]

Saw the same in the last week or so. New spot nodes came online (so, clearly there was capacity), but the on-demand nodes never got replaced.

[github-actions]

This issue has been inactive for 14 days. StaleBot will close this stale issue after 14 more days of inactivity.