Karpenter Creates 2 Instead of 1 Node

[robertvanloenhout]

Description

Regularly Karpenter is creating 2 nodes for 1 pod. Is this known behaviour?
This doesn't happen always, but it is a waste to create a node that won't be used for anything.
We are using the default batch time settings.
We see in the logs that the "found provionable pod(s)" entry occurs twice (or more) for the same pod and also see multiple "created nodeclaim" entries.
We see this behaviour in Karpenter version 1.0.5, 1,2.0 and 1.2.1 so far.
Any advice is appreciated.

Here is an example of the log when this happens:

{"level":"INFO","time":"2025-02-25T16:29:29.941Z","logger":"controller","message":"found provisionable pod(s)","commit":"652e6aa","controller":"provisioner","namespace":"","name":"","reconcileID":"2e7aa4fd-e611-40a4-8010-adae318d1938","Pods":"myns/mypod-88ec2e12-666e-4035-98c9-7528a8033b6c-6khwj","duration":"66.788305ms"}
{"level":"INFO","time":"2025-02-25T16:29:29.941Z","logger":"controller","message":"computed new nodeclaim(s) to fit pod(s)","commit":"652e6aa","controller":"provisioner","namespace":"","name":"","reconcileID":"2e7aa4fd-e611-40a4-8010-adae318d1938","nodeclaims":1,"pods":1}
{"level":"INFO","time":"2025-02-25T16:29:29.953Z","logger":"controller","message":"created nodeclaim","commit":"652e6aa","controller":"provisioner","namespace":"","name":"","reconcileID":"2e7aa4fd-e611-40a4-8010-adae318d1938","NodePool":{"name":"karpenter-mypod-nodepool"},"NodeClaim":{"name":"karpenter-mypod-nodepool-9tfpn"},"requests":{"cpu":"1780m","memory":"2832Mi","pods":"7"},"instance-types":"t3.medium"}
{"level":"INFO","time":"2025-02-25T16:29:31.976Z","logger":"controller","message":"launched nodeclaim","commit":"652e6aa","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"karpenter-mypod-nodepool-9tfpn"},"namespace":"","name":"karpenter-mypod-nodepool-9tfpn","reconcileID":"13108887-0d8b-43f8-b9a6-a9730c172e1d","provider-id":"aws:///us-west-2a/i-0ff00702eb67c678e","instance-type":"t3.medium","zone":"us-west-2a","capacity-type":"spot","allocatable":{"cpu":"1930m","ephemeral-storage":"44Gi","memory":"3246Mi","pods":"17"}}
{"level":"INFO","time":"2025-02-25T16:29:33.786Z","logger":"controller","message":"found provisionable pod(s)","commit":"652e6aa","controller":"provisioner","namespace":"","name":"","reconcileID":"9e21a352-9a94-41a7-bed2-9d64650a03e0","Pods":"myns/mypod-88ec2e12-666e-4035-98c9-7528a8033b6c-6khwj","duration":"68.817987ms"}
{"level":"INFO","time":"2025-02-25T16:29:33.786Z","logger":"controller","message":"computed new nodeclaim(s) to fit pod(s)","commit":"652e6aa","controller":"provisioner","namespace":"","name":"","reconcileID":"9e21a352-9a94-41a7-bed2-9d64650a03e0","nodeclaims":1,"pods":1}
{"level":"INFO","time":"2025-02-25T16:29:33.798Z","logger":"controller","message":"created nodeclaim","commit":"652e6aa","controller":"provisioner","namespace":"","name":"","reconcileID":"9e21a352-9a94-41a7-bed2-9d64650a03e0","NodePool":{"name":"karpenter-mypod-nodepool"},"NodeClaim":{"name":"karpenter-mypod-nodepool-jnck5"},"requests":{"cpu":"1780m","memory":"2832Mi","pods":"7"},"instance-types":"t3.medium"}
{"level":"INFO","time":"2025-02-25T16:29:35.885Z","logger":"controller","message":"launched nodeclaim","commit":"652e6aa","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"karpenter-mypod-nodepool-jnck5"},"namespace":"","name":"karpenter-mypod-nodepool-jnck5","reconcileID":"3e25b307-339b-4268-abf7-cbfccc87c14d","provider-id":"aws:///us-west-2b/i-041f7faa982af5d80","instance-type":"t3.medium","zone":"us-west-2b","capacity-type":"spot","allocatable":{"cpu":"1930m","ephemeral-storage":"44Gi","memory":"3246Mi","pods":"17"}}
{"level":"INFO","time":"2025-02-25T16:30:09.305Z","logger":"controller","message":"registered nodeclaim","commit":"652e6aa","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"karpenter-mypod-nodepool-jnck5"},"namespace":"","name":"karpenter-mypod-nodepool-jnck5","reconcileID":"6ea766b3-d456-4bef-8b43-9ca61c385e9d","provider-id":"aws:///us-west-2b/i-041f7faa982af5d80","Node":{"name":"ip-xx-xx-xxx-xxx.us-west-2.compute.internal"}}
{"level":"INFO","time":"2025-02-25T16:30:25.718Z","logger":"controller","message":"initialized nodeclaim","commit":"652e6aa","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"karpenter-mypod-nodepool-jnck5"},"namespace":"","name":"karpenter-mypod-nodepool-jnck5","reconcileID":"d2963818-eb06-4ca9-89ee-8b27c1d9030c","provider-id":"aws:///us-west-2b/i-041f7faa982af5d80","Node":{"name":"ip-xx-xx-xxx-xxx.us-west-2.compute.internal"},"allocatable":{"cpu":"1930m","ephemeral-storage":"47233297124","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"3388292Ki","pods":"17"}}
{"level":"INFO","time":"2025-02-25T16:30:51.952Z","logger":"controller","message":"registered nodeclaim","commit":"652e6aa","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"karpenter-mypod-nodepool-9tfpn"},"namespace":"","name":"karpenter-mypod-nodepool-9tfpn","reconcileID":"68f57d13-ab33-42ca-b882-c9e4e40d1214","provider-id":"aws:///us-west-2a/i-0ff00702eb67c678e","Node":{"name":"ip-yy-yy-yyy-yy.us-west-2.compute.internal"}}
{"level":"INFO","time":"2025-02-25T16:31:13.347Z","logger":"controller","message":"initialized nodeclaim","commit":"652e6aa","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"karpenter-mypod-nodepool-9tfpn"},"namespace":"","name":"karpenter-mypod-nodepool-9tfpn","reconcileID":"a25af346-c53a-42ca-b075-c2fd8fb05c00","provider-id":"aws:///us-west-2a/i-0ff00702eb67c678e","Node":{"name":"ip-yy-yy-yyy-yy.us-west-2.compute.internal"},"allocatable":{"cpu":"1930m","ephemeral-storage":"47233297124","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"3388300Ki","pods":"17"}}

[liafizan]

Might be similar: #6355. Please do check for taints and share any nodepool config is possible.

[robertvanloenhout]

Hi @liafizan . We are not using Cilium or any startup node taints.
Here is my nodepool configuration.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  annotations:
    karpenter.sh/stored-version-migrated: 'true'
  name: karpenter-mypod-nodepool
spec:
  disruption:
    budgets:
      - nodes: 10%
    consolidateAfter: 5m
    consolidationPolicy: WhenEmpty
  template:
    metadata:
      labels:
        project: myproject
        team: myteam
    spec:
      expireAfter: 720h
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: karpenter-mypod-ec2-nodeclass
      requirements:
        - key: kubernetes.io/os
          operator: In
          values:
            - linux
        - key: kubernetes.io/arch
          operator: In
          values:
            - amd64
        - key: k8s.mydomain.com/karpenter-group
          operator: In
          values:
            - mypod
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
            - t3.medium
            - m5.large
            - m5a.large
            - m7a.large
            - m7a.xlarge
            - m7i.large
            - m7i.xlarge
        - key: karpenter.sh/capacity-type
          operator: In
          values:
            - spot
      taints:
        - effect: NoSchedule
          key: k8s.mydomain.com/karpenter-group
          value: mypod

[jonathan-innis]

Can you capture the NodeClaims when Karpenter does this and share it here? Look at the resource requests for the pod and then also observe what the resolved allocatable is for the instance type that Karpenter chooses to launch.

One thing that is odd is that you seem to be seeing relaunching before the first node even registers (unless you are sharing partial logs) which is not normally the behavior we see when things like this happen -- typically this is the behavior like @liafizan when the node has less resources than Karpenter expects

[robertvanloenhout]

I have reproduced the double node spawn problem. In both nodeclaims the resources requests and allocatable have the same values.
I have added the nodeclaims below. In this case the pod ends up running on the node created by karpenter-mypod-nodepool-7vf2n (first one below).

apiVersion: karpenter.sh/v1
kind: NodeClaim
metadata:
  annotations:
    compatibility.karpenter.k8s.aws/cluster-name-tagged: 'true'
    karpenter.k8s.aws/ec2nodeclass-hash: '15889945674291150612'
    karpenter.k8s.aws/ec2nodeclass-hash-version: v4
    karpenter.k8s.aws/tagged: 'true'
    karpenter.sh/nodepool-hash: '16708420244891088869'
    karpenter.sh/nodepool-hash-version: v3
  creationTimestamp: '2025-03-06T10:00:05Z'
  finalizers:
    - karpenter.sh/termination
  generateName: karpenter-mypod-nodepool-
  generation: 1
  labels:
    k8s.mydomain.com/karpenter-group: mypod
    karpenter.k8s.aws/ec2nodeclass: karpenter-mypod-ec2-nodeclass
    karpenter.k8s.aws/instance-category: m
    karpenter.k8s.aws/instance-cpu: '2'
    karpenter.k8s.aws/instance-cpu-manufacturer: intel
    karpenter.k8s.aws/instance-cpu-sustained-clock-speed-mhz: '3100'
    karpenter.k8s.aws/instance-ebs-bandwidth: '4750'
    karpenter.k8s.aws/instance-encryption-in-transit-supported: 'false'
    karpenter.k8s.aws/instance-family: m5
    karpenter.k8s.aws/instance-generation: '5'
    karpenter.k8s.aws/instance-hypervisor: nitro
    karpenter.k8s.aws/instance-memory: '8192'
    karpenter.k8s.aws/instance-network-bandwidth: '750'
    karpenter.k8s.aws/instance-size: large
    karpenter.sh/capacity-type: spot
    karpenter.sh/nodepool: karpenter-mypod-nodepool
    kubernetes.io/arch: amd64
    kubernetes.io/os: linux
    node.kubernetes.io/instance-type: m5.large
    project: myproject
    team: myteam
    topology.k8s.aws/zone-id: usw2-az2
    topology.kubernetes.io/region: us-west-2
    topology.kubernetes.io/zone: us-west-2a
  name: karpenter-mypod-nodepool-7vf2n
  ownerReferences:
    - apiVersion: karpenter.sh/v1
      blockOwnerDeletion: true
      kind: NodePool
      name: karpenter-mypod-nodepool
      uid: 9a8b4849-6899-49d3-9a43-a9c51f48e9c3
  resourceVersion: '318352168'
  uid: 643fd8a0-e848-4e81-8954-ad359e20a289
  selfLink: /apis/karpenter.sh/v1/nodeclaims/karpenter-mypod-nodepool-7vf2n
status:
  allocatable:
    cpu: 1930m
    ephemeral-storage: 44Gi
    memory: 7134084Ki
    pods: '29'
    vpc.amazonaws.com/pod-eni: '9'
  capacity:
    cpu: '2'
    ephemeral-storage: 50Gi
    memory: 7824260Ki
    pods: '29'
    vpc.amazonaws.com/pod-eni: '9'
  conditions:
    - lastTransitionTime: '2025-03-06T10:00:07Z'
      message: ''
      observedGeneration: 1
      reason: Launched
      status: 'True'
      type: Launched
    - lastTransitionTime: '2025-03-06T10:00:33Z'
      message: ''
      observedGeneration: 1
      reason: Registered
      status: 'True'
      type: Registered
    - lastTransitionTime: '2025-03-06T10:00:53Z'
      message: ''
      observedGeneration: 1
      reason: Initialized
      status: 'True'
      type: Initialized
    - lastTransitionTime: '2025-03-06T10:00:53Z'
      message: ''
      observedGeneration: 1
      reason: Ready
      status: 'True'
      type: Ready
  imageID: ami-0d573b1e8a5997a3b
  lastPodEventTime: '2025-03-06T10:00:58Z'
  nodeName: ip-10-16-166-187.us-west-2.compute.internal
  providerID: aws:///us-west-2a/i-035423e44b70e1644
spec:
  expireAfter: 720h
  nodeClassRef:
    group: karpenter.k8s.aws
    kind: EC2NodeClass
    name: karpenter-mypod-ec2-nodeclass
  requirements:
    - key: kubernetes.io/os
      operator: In
      values:
        - linux
    - key: node.kubernetes.io/instance-type
      operator: In
      values:
        - m5.large
        - m5a.large
        - m7a.large
        - m7i.large
    - key: kubernetes.io/arch
      operator: In
      values:
        - amd64
    - key: karpenter.k8s.aws/instance-size
      operator: In
      values:
        - large
    - key: project
      operator: In
      values:
        - myproject
    - key: karpenter.k8s.aws/ec2nodeclass
      operator: In
      values:
        - karpenter-mypod-ec2-nodeclass
    - key: team
      operator: In
      values:
        - myteam
    - key: topology.kubernetes.io/zone
      operator: In
      values:
        - us-west-2a
    - key: k8s.mydomain.com/karpenter-group
      operator: In
      values:
        - mypod
    - key: karpenter.sh/capacity-type
      operator: In
      values:
        - spot
    - key: karpenter.sh/nodepool
      operator: In
      values:
        - karpenter-mypod-nodepool
  resources:
    requests:
      cpu: 1780m
      memory: 6244Mi
      pods: '7'
  taints:
    - effect: NoSchedule
      key: k8s.mydomain.com/karpenter-group
      value: mypod

apiVersion: karpenter.sh/v1
kind: NodeClaim
metadata:
  annotations:
    compatibility.karpenter.k8s.aws/cluster-name-tagged: 'true'
    karpenter.k8s.aws/ec2nodeclass-hash: '15889945674291150612'
    karpenter.k8s.aws/ec2nodeclass-hash-version: v4
    karpenter.k8s.aws/tagged: 'true'
    karpenter.sh/nodepool-hash: '16708420244891088869'
    karpenter.sh/nodepool-hash-version: v3
  creationTimestamp: '2025-03-06T10:00:02Z'
  finalizers:
    - karpenter.sh/termination
  generateName: karpenter-mypod-nodepool-
  generation: 1
  labels:
    k8s.mydomain.com/karpenter-group: mypod
    karpenter.k8s.aws/ec2nodeclass: karpenter-mypod-ec2-nodeclass
    karpenter.k8s.aws/instance-category: m
    karpenter.k8s.aws/instance-cpu: '2'
    karpenter.k8s.aws/instance-cpu-manufacturer: intel
    karpenter.k8s.aws/instance-cpu-sustained-clock-speed-mhz: '3100'
    karpenter.k8s.aws/instance-ebs-bandwidth: '4750'
    karpenter.k8s.aws/instance-encryption-in-transit-supported: 'false'
    karpenter.k8s.aws/instance-family: m5
    karpenter.k8s.aws/instance-generation: '5'
    karpenter.k8s.aws/instance-hypervisor: nitro
    karpenter.k8s.aws/instance-memory: '8192'
    karpenter.k8s.aws/instance-network-bandwidth: '750'
    karpenter.k8s.aws/instance-size: large
    karpenter.sh/capacity-type: spot
    karpenter.sh/nodepool: karpenter-mypod-nodepool
    kubernetes.io/arch: amd64
    kubernetes.io/os: linux
    node.kubernetes.io/instance-type: m5.large
    project: myproject
    team: myteam
    topology.k8s.aws/zone-id: usw2-az3
    topology.kubernetes.io/region: us-west-2
    topology.kubernetes.io/zone: us-west-2c
  name: karpenter-mypod-nodepool-k8v8q
  ownerReferences:
    - apiVersion: karpenter.sh/v1
      blockOwnerDeletion: true
      kind: NodePool
      name: karpenter-mypod-nodepool
      uid: 9a8b4849-6899-49d3-9a43-a9c51f48e9c3
  resourceVersion: '318352063'
  uid: a2c08457-1896-42db-92b6-0d1e07df79c4
  selfLink: /apis/karpenter.sh/v1/nodeclaims/karpenter-mypod-nodepool-k8v8q
status:
  allocatable:
    cpu: 1930m
    ephemeral-storage: 44Gi
    memory: 7134084Ki
    pods: '29'
    vpc.amazonaws.com/pod-eni: '9'
  capacity:
    cpu: '2'
    ephemeral-storage: 50Gi
    memory: 7824260Ki
    pods: '29'
    vpc.amazonaws.com/pod-eni: '9'
  conditions:
    - lastTransitionTime: '2025-03-06T10:00:04Z'
      message: ''
      observedGeneration: 1
      reason: Launched
      status: 'True'
      type: Launched
    - lastTransitionTime: '2025-03-06T10:00:31Z'
      message: ''
      observedGeneration: 1
      reason: Registered
      status: 'True'
      type: Registered
    - lastTransitionTime: '2025-03-06T10:00:51Z'
      message: ''
      observedGeneration: 1
      reason: Initialized
      status: 'True'
      type: Initialized
    - lastTransitionTime: '2025-03-06T10:00:51Z'
      message: ''
      observedGeneration: 1
      reason: Ready
      status: 'True'
      type: Ready
  imageID: ami-0d573b1e8a5997a3b
  nodeName: ip-10-16-179-119.us-west-2.compute.internal
  providerID: aws:///us-west-2c/i-0ce7773a11b8defaf
spec:
  expireAfter: 720h
  nodeClassRef:
    group: karpenter.k8s.aws
    kind: EC2NodeClass
    name: karpenter-mypod-ec2-nodeclass
  requirements:
    - key: kubernetes.io/os
      operator: In
      values:
        - linux
    - key: node.kubernetes.io/instance-type
      operator: In
      values:
        - m5.large
        - m5a.large
        - m7a.large
        - m7i.large
    - key: kubernetes.io/arch
      operator: In
      values:
        - amd64
    - key: karpenter.k8s.aws/instance-size
      operator: In
      values:
        - large
    - key: project
      operator: In
      values:
        - myproject
    - key: karpenter.k8s.aws/ec2nodeclass
      operator: In
      values:
        - karpenter-mypod-ec2-nodeclass
    - key: team
      operator: In
      values:
        - myteam
    - key: k8s.mydomain.com/karpenter-group
      operator: In
      values:
        - mypod
    - key: karpenter.sh/capacity-type
      operator: In
      values:
        - spot
    - key: karpenter.sh/nodepool
      operator: In
      values:
        - karpenter-mypod-nodepool
  resources:
    requests:
      cpu: 1780m
      memory: 6244Mi
      pods: '7'
  taints:
    - effect: NoSchedule
      key: k8s.mydomain.com/karpenter-group
      value: mypod

[robertvanloenhout]

@jonathan-innis I did not delete any entries in the log I shared. It's a complete part of the log where I saw the 2 nodeclaims being created. It's from the development environment so I can control how much is going on there.