Cluster Autoscaler uses SetDesiredCapacity causing AWS to terminate nodes with active workloads

[gfvirga]

Summary

The Kubernetes Cluster Autoscaler scaled down an ASG using SetDesiredCapacity instead of selectively terminating empty nodes via TerminateInstanceInAutoScalingGroup. This caused AWS Auto Scaling to
randomly select and terminate 32 instances, including nodes running active GitLab CI jobs.

Environment

• Cluster Autoscaler Version: v1.34.2
• Kubernetes Version: 1.34
• Cloud Provider: AWS

Timeline (2026-02-04 UTC)

• 04:47:13 - Autoscaler sets desired capacity to 76
• 04:47:23 - Autoscaler sets desired capacity to 94 (scale up)
• 04:47:34 - Autoscaler sets desired capacity to 62 (scale down by 32)
• 04:48:49-04:49:05 - AWS Auto Scaling terminates 32 instances
• 04:48:00 - Active GitLab runner pod evicted (job 668628 failed)

Evidence

CloudTrail Event (04:47:34 UTC):
json
{
"eventName": "SetDesiredCapacity",
"userAgent": "aws-sdk-go/1.48.7 (go1.24.10; linux; amd64) cluster-autoscaler/v1.34.2",
"requestParameters": {
"autoScalingGroupName": "REDACTED",
"desiredCapacity": 62,
"honorCooldown": false
}
}

Kubernetes Events:

• Node ip-10-0-128-21.ec2.internal was marked "unremovable" by autoscaler at 04:40:00
• No "ToBeDeletedByClusterAutoscaler" annotation found on the node
• Node terminated by AWS at 04:48:49 without Kubernetes coordination

Expected Behavior

The cluster autoscaler should:

1. Identify specific empty/underutilized nodes as scale-down candidates
2. Mark them with ToBeDeletedByClusterAutoscaler annotation
3. Use TerminateInstanceInAutoScalingGroup API to remove specific instances
4. Never terminate nodes with active workloads

Actual Behavior

The cluster autoscaler:

1. Made rapid capacity adjustments (76→94→62 in 21 seconds)
2. Used SetDesiredCapacity for bulk scale-down
3. Let AWS Auto Scaling randomly select which instances to terminate
4. Resulted in termination of nodes with running pods

Impact

• GitLab CI job failures
• Workload disruption
• Loss of trust in autoscaler safety

Questions

1. Why did the autoscaler use SetDesiredCapacity instead of TerminateInstanceInAutoScalingGroup?
2. Is there a configuration issue causing aggressive scale-down behavior?
3. Should the autoscaler ever use bulk capacity changes that delegate instance selection to AWS?

Proposed Fix

The autoscaler should always use TerminateInstanceInAutoScalingGroup for scale-down operations to ensure only safe-to-remove nodes are terminated.

[gfvirga]

Might be related to #8196

[gfvirga]

Update:

I've analyzed the cluster autoscaler logs and identified the exact bug that caused this incident.

The Bug

The autoscaler's placeholder instance reconciliation logic incorrectly canceled a valid scale-up operation, causing AWS to terminate 32 random instances.

Timeline with Logs

04:47:13 UTC - Autoscaler initiates scale-up from 62→76 nodes

Final scale-up plan: [{eks-amd64-blue-REDACTED 62->76 (max: 250)}]

04:47:33 UTC - Autoscaler detects AWS hasn't launched all instances yet, creates placeholders

Instance group eks-amd64-blue-REDACTED has only 62 instances created
while requested count is 94. Creating placeholder instances.

04:47:34 UTC - THE BUG: Autoscaler immediately reconciles by setting desired capacity back to 62

asg eks-amd64-blue-REDACTED has placeholders instances with
desired capacity = 94 and active instances = 62. updating ASG to match active instances count

Setting asg eks-amd64-blue-REDACTED size to 62

04:48:49-04:49:05 UTC - AWS Auto Scaling terminates 32 instances to reach desired capacity of 62

Root Cause

The placeholder reconciliation logic in auto_scaling_groups.go:328 has a race condition. When AWS hasn't finished launching requested instances, the autoscaler:

1. Creates virtual placeholder nodes internally
2. Immediately calls SetDesiredCapacity to match the current active instance count
3. This cancels the in-progress scale-up operation
4. AWS then terminates instances to reach the new (lower) desired capacity

Expected Behavior

The autoscaler should wait for AWS to complete the instance launch before reconciling placeholder instances. It should never use SetDesiredCapacity to reduce capacity during an active scale-up operation.

Questions for Maintainers

1. Is the placeholder reconciliation logic intended to cancel in-progress scale-ups?
2. Should there be a grace period before reconciling placeholder instances?
3. Why does the code path use SetDesiredCapacity instead of waiting for AWS to finish launching?

Affected Code

• File: auto_scaling_groups.go
• Line 328: Placeholder reconciliation logic
• Line 267: SetDesiredCapacity call

This appears to be a critical bug in the placeholder instance handling that can cause production disruption.

[gfvirga]

Update: Related to Issue #9043

This is the same bug as #9043, but with a more dangerous failure mode.

Issue #9043: SetDesiredCapacity fails due to ASG min constraints → infinite loop, no terminations

This Issue: SetDesiredCapacity succeeds → AWS terminates 32 nodes with active workloads

Both hit the same code path in auto_scaling_groups.go:328 where placeholder reconciliation incorrectly reduces desired capacity during active scale-up operations.

#9043 fails safely with validation errors. This issue shows the bug can silently terminate production workloads when the capacity reduction is within ASG constraints.

[adrianmoisey]

/area cluster-autoscaler

[maishivamhoo123]

/assign