Karpenter add unwanted label to node

[omri-shilton]

hello, im having a weird behaviour where my pod has a node selector and taint and my only nodepool has no labels or taint. karpenter then creates a node and adds the label that the pod requires on its own, without me ever indicating that.
and my pod does not schedule on the actual node that has the node selector and taint.

karpenter version: 0.32.1
eks: 1.31

[jmdeal]

Could you provide your NodePool and pod manifests?

[omri-shilton]

nodepool:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h
  template:
    spec:
      nodeClassRef:
        name: default
      requirements:
        - key: type
          operator: Exists
        - key: kubernetes.io/arch
          operator: In
          values:
            - amd64
        - key: kubernetes.io/os
          operator: In
          values:
            - linux
        - key: karpenter.sh/capacity-type
          operator: In
          values:
            - spot
            - on-demand
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values:
            - c
            - m
            - r
            - i
            - d
        - key: karpenter.k8s.aws/instance-cpu
          operator: In
          values:
            - '2'
            - '4'
            - '8'
            - '16'
            - '32'
            - '48'
            - '64'

pod spec:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: "topology.kubernetes.io/zone"
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          actions.github.com/scale-set-name: custom-runner-2
  nodeSelector:
    type: arc
  tolerations:
  - key: "type"
    operator: "Equal"
    value: "arc"
    effect: "NoSchedule"
  volumes:
    - name: tun
      hostPath:
        path: /dev/net/tun
  initContainers:
    - name: kube-init
      image: ghcr.io/actions/actions-runner:latest
      command: ["/bin/sh", "-c"]
      args:
        - |
          sudo chown -R 1001:123 /home/runner/_work
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
  containers:
  - name: runner
    image: ghcr.io/actions/actions-runner:latest
    command: ["/home/runner/run.sh"]
    resources:
      requests:
        cpu: 3000m
    volumeMounts:
      - mountPath: /dev/net/tun
        name: tun
    securityContext:
      capabilities:
        add: ["NET_ADMIN"]
    env:
      - name: ACTIONS_RUNNER_CONTAINER_HOOKS
        value: /home/runner/k8s/index.js
      - name: ACTIONS_RUNNER_POD_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
      - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
        value: "false"

[jmdeal]

I take it the label you're referring to is the test label? You specify a requirement with the exists operator in your NodePool, which is how you instruct Karpenter to create a node with a label matching the requirement from a node selector (docs).

apiVersion: karpenter.sh/v1beta1
kind: NodePool
spec:
  template:
    spec:
      requirements:
        - key: type
          operator: Exists

If you don't want Karpenter to create nodes with this label, you should remove that requirement from your NodePool.

and my pod does not schedule on the actual node that has the node selector and taint

Karpenter only creates nodes for pods which have been marked unschedulable. There's likely some other factor that's preventing the pod from scheduling to the existing node that's unrelated to Karpenter. There should be some indication of why in events.

[github-actions]

This issue has been inactive for 7 days and is marked as "triage/solved". StaleBot will close this stale issue after 7 more days of inactivity.