DaemonSet pods can remain Pending when workload pods consume Karpenter-calculated DaemonSet overhead on new nodes #9269

Description

Observed Behavior:

We occasionally observe DaemonSet pods remaining Pending with Insufficient cpu or Insufficient memory on newly provisioned Karpenter nodes.

Example scenario:

1. A large number of GitLab Runner pods are pending.
2. Karpenter computes a node for a workload pod while including the expected DaemonSet overhead.
3. Karpenter creates the NodeClaim and the node joins the cluster.
4. The DaemonSet controller creates pods targeting the new node.
5. Before those DaemonSet pods are bound, kube-scheduler schedules other compatible pending GitLab Runner pods onto the node.
6. The node no longer has sufficient allocatable resources for one or more DaemonSet pods.
7. The DaemonSet pods remain Pending with Insufficient cpu or Insufficient memory.

We would like to clarify whether this interaction between Karpenter and kube-scheduler is expected.

Specific questions:

1. Does Karpenter only use DaemonSet overhead for instance-size calculation, without reserving or protecting that capacity until the DaemonSet pods are scheduled?
2. Are pod nomination events expected to influence kube-scheduler, or are they only used internally by Karpenter?
3. Should Karpenter temporarily taint newly provisioned nodes until expected DaemonSets are placed, or is using a custom startup taint the recommended approach?
4. What logs, events, or metrics would best confirm whether this scheduling race is occurring?

There also does not appear to be a clear documentation section or diagram describing the full scheduling flow for DaemonSet overhead, including:

• when DaemonSet overhead is calculated;
• whether that capacity is reserved or only used for instance type selection;
• how this interacts with kube-scheduler;
• how pod nomination events are used;
• what guarantees users should or should not expect.

A documentation clarification and/or diagram would be very helpful.

Expected Behavior:

A node provisioned with DaemonSet overhead included in its resource calculation should have sufficient capacity for all applicable DaemonSet pods and the workload pods for which it was provisioned.

Alternatively, if Karpenter only accounts for DaemonSet overhead during instance type selection and does not reserve that capacity, the documentation should clearly describe this behavior and explain the recommended mitigation.

Reproduction Steps (Please include YAML):

1. Deploy one or more DaemonSets with explicit CPU and memory requests.

2. Create enough workload pods to produce a large pending queue.

3. Ensure the workload pods and DaemonSet pods have equal or similar priorities.

4. Allow Karpenter to provision new nodes for the pending workloads.

5. Compare:

   • Karpenter's computed DaemonSet overhead;
   • the NodeClaim resource requests;
   • pod creation and binding timestamps;
   • the order in which workload pods and DaemonSet pods are bound;
   • FailedScheduling events for the DaemonSet pods.

Versions:

• k8s(eks): 1.34