Stale bufferPodCounts after CapacityBuffer deletion blocks WhenEmpty node disruption

[celalettin1286]

Description

Observed Behavior:

When a CapacityBuffer is deleted on an otherwise idle cluster, the nodes that were backing it fail to scale down via the WhenEmpty disruption path. The nodes sit idle indefinitely until unrelated cluster activity (like a new pending pod elsewhere) wakes the provisioner.

The issue is caused by a missed provisioner trigger during the deletion path of a CapacityBuffer, leading to a stale cluster state cache:

1. The Block: The disruption controller polls every 10s. For WhenEmpty, it checks e.cluster.HasBufferPods() (pkg/controllers/disruption/emptiness.go). This reads the in-memory bufferPodCounts map.

2. The Stale Cache: bufferPodCounts is only updated by the provisioner (pkg/controllers/provisioning/provisioner.go) at the end of a scheduling pass.

3. The Missing Trigger: When a CapacityBuffer is deleted, the buffer controller (pkg/controllers/capacitybuffer/controller.go) catches the IsNotFound error and successfully clears the virtualPodCache. However, it returns immediately without calling c.trigger.Trigger().

4. The Result: On an idle cluster, the provisioner never runs again. The bufferPodCounts map retains the stale entry for the node, and the emptiness check perpetually returns false (refusing to disrupt) because it thinks buffer pods are still present.

Note: This also occurs if a CapacityBuffer remains, but its PodTemplate/scalableRef is deleted (resolved == false), as the trigger is similarly skipped.

Expected Behavior:

When a CapacityBuffer is deleted, its associated empty nodes should immediately become eligible for WhenEmpty disruption and scale down, even if there is no other scheduling activity occurring in the cluster. The CapacityBuffer controller should trigger a provisioner pass upon deletion so the cluster state cache (bufferPodCounts) is accurately refreshed.

Reproduction Steps (Please include YAML):

1. Ensure you have a cluster with no pending pods or scheduling activity.
2. Apply a NodePool configured for WhenEmpty disruption
3. Apply a CapacityBuffer to scale up empty nodes.
4. Wait for the nodes to be successfully provisioned and ready.
5. Delete the CapacityBuffer.
6. Observe: The nodes sit empty indefinitely and are never disrupted.

Verification: Create a dummy pending pod anywhere in the cluster (to wake the provisioner). Observe that the empty nodes immediately get disrupted and scale down once the provisioner pass updates the stale map.

Versions:

• Chart Version: aws-provider 1.14.0
• Kubernetes Version (kubectl version): 1.35.6

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[kubernetes-prow]

This issue is currently awaiting triage.

If Karpenter contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.