Karpenter ignores DaemonSet resource requirements, causing DaemonSets to remain unscheduled

[JacobAmar]

Description

Observed Behavior:
Karpenter occasionally provisions nodes without accounting for DaemonSet resource requirements. As a result, some DaemonSets (e.g., node-exporter) do not schedule on the newly-created nodes due to insufficient CPU or memory, despite those DaemonSets specifying explicit resource requests and requiring tolerations.
This leaves nodes without visibility/monitoring agents, and we see FailedScheduling events for those DaemonSet pods.

Related/Previous Issue Reference:
See aws/karpenter-provider-aws#2751 — we observe the same symptoms and scheduling problems described there.

Expected Behavior:
Karpenter should consider resource requests for all DaemonSets expected to run on new nodes during provisioning, so sufficient resources are available for both triggering workload pods and all relevant DaemonSets.

Reproduction Steps (Please include YAML):
This issue is difficult to reproduce reliably, as it occurs randomly depending on scheduling order and cluster load. Below is a representative DaemonSet YAML that triggers the issue in our environment:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
spec:
  template:
    spec:
      containers:
        - name: node-exporter
          resources:
            requests:
              cpu: 50m
              memory: 50Mi
            limits:
              memory: 256Mi
      tolerations:
        - effect: NoSchedule
          operator: Exists
      nodeSelector:
        kubernetes.io/os: linux

1. Apply the DaemonSet above.
2. Schedule a regular workload pod that triggers Karpenter to provision a new node.
3. Observe: Occasionally, the DaemonSet fails to schedule with FailedScheduling due to insufficient resources.

Versions:

• Chart Version: 1.6.2
• Kubernetes Version (kubectl version): v1.33.5-eks-3cfe0ce

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ryan-mist]

Is the DaemonSet failing to schedule on existing nodes (i.e. ones present in the cluster before the DaemonSet is created)? Or is it failing to schedule on newer nodes?

[JacobAmar]

@ryan-mist it failling to schedule on existing nodes. it seems that it dosent calculate its requirements to fit the nodes.

[ryan-mist]

it failling to schedule on existing nodes.

So the daemonset is being created after the node? If the daemonset is created after node creation, then Karpenter won't provision any new capacity for the pending daemonset. Karpenter will only provision enough capacity for pods/daemonsets that exist at the time of scheduling.

If you need this daemonset on the node, you could try to add a priorityClassName to your daemonset. A high enough priority, would evict other workloads to make room for the daemonset, in which Karpenter would provision new nodes for the pending workloads. See https://karpenter.sh/docs/faq/#when-deploying-an-additional-daemonset-to-my-cluster-why-does-karpenter-not-scale-up-my-nodes-to-support-the-extra-daemonset.

[JacobAmar]

@ryan-mist no, the daemonset was exists before, im not going to add priorityClassName as it can affect existing pods that are important, its the same issue as i stated here #2751
karpenter should take daemonsets into account

[github-actions]

This issue has been inactive for 14 days. StaleBot will close this stale issue after 14 more days of inactivity.