Provisioner creates excess NodeClaims for simultaneously pending pods even without topology constraints #3121

Observed Behavior

When multiple pods become Pending simultaneously (e.g., batch Job creation), Karpenter's provisioner creates significantly more NodeClaims than needed. The first provisioning batch is correct (N pods = N NodeClaims), but subsequent provisioner loops create additional NodeClaims even though the initial ones should satisfy all pending pods.

This happens regardless of whether topology constraints (podAntiAffinity, topologySpreadConstraints) are used or not.

Test Results

We tested multiple configurations with 3 simultaneously created Job pods:

In all cases, the first batch at T+0s correctly creates 3 NodeClaims. But subsequent loops at T+30s, T+60s, T+90s create extras even though pods should already be "claimed" by the initial NodeClaims.

Expected Behavior

Karpenter should create exactly N NodeClaims for N pending pods when:

• The initial batch correctly identifies N pods need N nodes
• The pods have been "claimed" against in-flight NodeClaims
• No new pending pods have arrived

Subsequent provisioner loops should recognize that pending pods are already accounted for by in-flight NodeClaims and skip provisioning.

Reproduction Steps

# 3 Jobs applied simultaneously - no topology constraints, pure resource-based
apiVersion: batch/v1
kind: Job
metadata:
  name: karp-test-1
  namespace: test
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 300
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"
      labels:
        app: test-job
    spec:
      nodeSelector:
        my-nodepool-label: "true"
        karpenter.k8s.aws/instance-size: "xlarge"
      containers:
        - name: test
          image: busybox:1.36
          command: ["sh", "-c", "sleep 300"]
          resources:
            requests:
              cpu: 3300m
              memory: 500Mi
      restartPolicy: Never

Apply 3 copies simultaneously: kubectl apply -f test-jobs.yaml

NodePool Configuration

apiVersion: karpenter.sh/v1
kind: NodePool
spec:
  disruption:
    consolidateAfter: 2m
    consolidationPolicy: WhenEmptyOrUnderutilized
  limits:
    cpu: "100"
    memory: 500Gi
  template:
    spec:
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["m"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.k8s.aws/instance-cpu
          operator: Gt
          values: ["2"]

NodeClaim Creation Timestamps (showing the pattern)

my-nodepool-4rqs5   2026-06-30T09:08:46Z  ← Correct first batch (3)
my-nodepool-r46hm   2026-06-30T09:08:46Z
my-nodepool-zf798   2026-06-30T09:08:46Z
my-nodepool-zsj5p   2026-06-30T09:09:16Z  ← Extra (30s later)
my-nodepool-m6rqt   2026-06-30T09:09:48Z  ← Extra (62s later)
my-nodepool-xdtbt   2026-06-30T09:09:48Z
my-nodepool-xvnzl   2026-06-30T09:09:48Z

Root Cause Analysis

The provisioner correctly batches pods and creates NodeClaims in the first pass. However, during subsequent reconciliation loops:

1. NodeClaims are in Unknown state (node launching but not Ready)
2. Daemonsets get scheduled on the new nodes, consuming capacity
3. The provisioner re-evaluates and determines pods may not fit (daemonset overhead recalculation)
4. Pods appear "unclaimed" and the provisioner creates additional NodeClaims
5. Extra nodes eventually consolidate (empty), but the over-provisioning wastes resources

This does NOT happen when pods arrive sequentially (e.g., Jenkins builds triggering one at a time), only when multiple pods become Pending simultaneously.

Impact

• Unnecessary EC2 instances launched (cost)
• Potential to hit NodePool CPU/memory limits prematurely
• AWS API rate limiting from rapid instance launches
• Confusing behavior for platform teams managing Karpenter

Environment

• Karpenter Version: 1.12.1
• Kubernetes Version: 1.35
• Instance types: m6a.xlarge (4 vCPU, 16Gi)
• BATCH_MAX_DURATION: 10s
• BATCH_IDLE_DURATION: 3s

Related Issues

• Karpenter scaling unproportionately high nodes #2123 - Karpenter scaling unproportionately high nodes
• #4163 - Karpenter simulates node capacity incorrectly causing pod scheduling to fail
• #3725 - When scaling up (by 1 node), Karpenter adds 2 nodes instead