Karpenter does not launch new nodes on pending pods when pods in cluster is over 40,000

[ofekifergan]

Description

Observed Behavior:
New pods stuck in Pending state when our cluster reach ~40,000 pods, no logs regarding the error

I can see that karpenter is computing pod scheduling and ites taking a lot of time.

{"level":"INFO","time":"2025-01-29T13:46:56.910Z","logger":"controller","message":"computing pod scheduling...","commit":"0a85efb","controller":"provisioner","namespace":"","name":"","reconcileID":"xxx","pods-scheduled":19769,"pods-remaining":13163,"duration":"33m11s","scheduling-id":"xxx"}

Expected Behavior:
New nodes should be launched on pending pods regardless the amount of pods.
Eks should support up to 150,000 pods

Reproduction Steps (Please include YAML):

Versions: 1.1

• Chart Version:
• Kubernetes Version (kubectl version): 1.30

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ofekifergan]

Update: Karpenter is stuck on "pods-remaining":13163, this is really critical issue, its not updating anymore and force me to remove pods in production!

Please fix this issue, why is this so slow as well

[jmdeal]

Are you able to share your logs, the resources allocated to the Karpenter deployment, and the instance type Karpenter is running on?

[rschalo]

Also, can you share your nodepool and nodeclass configs?

[ofekifergan]

@jmdeal @rschalo
Karpenter is running on m6a.2xlarge on these resources:

resources:
        limits:
          cpu: '1'
          memory: 8Gi
        requests:
          cpu: '1'
          memory: 8Gi

It crashed on OOM, becuase of the amount of pods so I had to increase the memory.

The nodegroup config:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gp-ondemand
spec:
  disruption:
    budgets:
      - nodes: '1'
    consolidateAfter: 0s
    consolidationPolicy: WhenEmptyOrUnderutilized
  limits:
    memory: 1500Gi
  template:
    metadata:
      labels:
        nodeGroup: gp-ondemand
    spec:
      expireAfter: Never
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: karpenter-gp-ondemand-template
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values:
            - on-demand
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
            - m6a.2xlarge
      taints:
        - effect: NoSchedule
          key: nodeGroup
          value: gp-ondemand

Nodeclass:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: karpenter-gp-ondemand-template
spec:
  amiFamily: AL2
  amiSelectorTerms:
    - id: xxx
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        deleteOnTermination: true
        encrypted: true
        volumeSize: 100Gi
        volumeType: gp3
  metadataOptions:
    httpEndpoint: enabled
    httpTokens: required
  role: eks-worker-node-role-prod-eu-west
  securityGroupSelectorTerms:
    - id: x
    - id: y
    - id: z
  subnetSelectorTerms:
    - id: x
    - id: y
    - id: z
  tags:
    ClusterName: <env>
    Environment: <env>
    Name: karpenter-eks-<env>-gp-ondemand-gp
    nodeGroup: gp-ondemand

Currently I dont have the logs because I restarted Karpenter but the recent logs was:

{"level":"INFO","time":"2025-01-29T13:46:56.910Z","logger":"controller","message":"computing pod scheduling...","commit":"0a85efb","controller":"provisioner","namespace":"","name":"","reconcileID":"xxx","pods-scheduled":19769,"pods-remaining":13163,"duration":"33m11s","scheduling-id":"xxx"}

and it got stuck on pods-remaining:13163 and basiclly hanged forever.
Only after reducing the amount of pods it worked again.

This happend to me at production and it must be fixed as we rely on karpenter to be able to handle this kind of load.

[jmdeal]

Karpenter is running on m6a.2xlarge on these resources

Just confirming because it matches the instance type specified on the NodePool, but this is the node that the Karpenter controller pods, not just the instance type it's provisioning?

1 vCPU sounds pretty limited given the number of pods in your cluster, though it's hard to give a one-size fit's all number since scheduling complexity is more than just the number of nodes and pods (e.g. TSC, anti-affinity, etc). I would check if Karpenter is being CPU throttled, I suspect it likely is with those requests.

Currently I dont have the logs because I restarted Karpenter but the recent logs was

Without logs, we don't have much to go on. If you've retained metrics, we can look at karpenter_scheduler_queue_depth to see how scheduling progressed over time. This could give a clue as to what is causing performance bottlenecks. I highly recommend retaining both Karpenter controller logs and metrics in production environments, since they are invaluable when it comes to root causing these types of issues.

Since the information we have to work with is limited, I would recommend increasing the CPU requests for the Karpenter controller pods as a starting point since that's the most obvious bottleneck. If you're still running into issues, the following information would help root cause:

• Karpenter controller logs
• CPU and memory utilization metrics
• All NodePools (if there's more than the one you already provided)
• All deployment manifests
  • Focus on scheduling constraints (node selectors, pod affinity / anti-affinity terms, topology spread constraints, etc)
• Other Karpenter metrics (e.g. karpenter_scheduler_queue_depth)

[ofekifergan]

Karpenter is running on m6a.2xlarge on these resources

Just confirming because it matches the instance type specified on the NodePool, but this is the node that the Karpenter controller pods, not just the instance type it's provisioning?

1 vCPU sounds pretty limited given the number of pods in your cluster, though it's hard to give a one-size fit's all number since scheduling complexity is more than just the number of nodes and pods (e.g. TSC, anti-affinity, etc). I would check if Karpenter is being CPU throttled, I suspect it likely is with those requests.

Currently I dont have the logs because I restarted Karpenter but the recent logs was

Without logs, we don't have much to go on. If you've retained metrics, we can look at karpenter_scheduler_queue_depth to see how scheduling progressed over time. This could give a clue as to what is causing performance bottlenecks. I highly recommend retaining both Karpenter controller logs and metrics in production environments, since they are invaluable when it comes to root causing these types of issues.

Since the information we have to work with is limited, I would recommend increasing the CPU requests for the Karpenter controller pods as a starting point since that's the most obvious bottleneck. If you're still running into issues, the following information would help root cause:

• Karpenter controller logs

• CPU and memory utilization metrics

• All NodePools (if there's more than the one you already provided)

• All deployment manifests

  • Focus on scheduling constraints (node selectors, pod affinity / anti-affinity terms, topology spread constraints, etc)

• Other Karpenter metrics (e.g. karpenter_scheduler_queue_depth)

Karpenter does not host itself (not using the Nodepool to provison karpenter itself),
I understand what you are saying but even though Karpenter is limited to 1 core of cpu, why it got stuck in the first place?
It worked for 2 minutes and then got stuck for ever, did we reached some kind of deadlock or something like that?

I will increase the cpu provisoned just to be sure and I will improve our monitoring / metrics scraping, but still it sound like a bug because I believe Karpenter had more than enough resources to handle this kind of situation.