[ca] fix: undo Scale operation when a scaleUpRequest times out by thatmidwesterncoder · Pull Request #9132 · kubernetes/autoscaler

thatmidwesterncoder · 2026-01-27T18:56:24Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR modifies the cluster-autoscaler to automatically revert (decrease) the target size of a node group when a scale-up request times out.

Currently, when a scale-up fails due to infrastructure provider capacity limits (e.g., CAPI infrastructure provider cannot provision new nodes), the autoscaler removes the scale-up request from tracking and puts the node group in backoff, but it does not decrease the target size back to its original value. This causes the infrastructure provider to indefinitely retry provisioning the failed nodes, even after the workload that triggered the scale-up has disappeared. Manual intervention is required to scale the cluster back down.

With this fix, when a scale-up request times out:

The autoscaler calls DecreaseTargetSize() on the node group to revert the increase
If the decrease fails, a warning is logged and an event is emitted
The scale-up request is then removed from tracking as before

This allows clusters running close to infrastructure capacity to automatically recover from failed scale-ups without manual intervention.

Which issue(s) this PR fixes:

Fixes #9120

Special notes for your reviewer:

The DecreaseTargetSize method is called with a negative value equal to the original Increase from the scale-up request
Error handling is in place - if the decrease fails, we log a warning and emit an event, but still proceed with removing the scale-up request and applying backoff
Existing tests were updated to include WithOnScaleUp handler since the test provider now needs to handle the decrease operation
Three new test cases were added:
- TestExpiredScaleUpRevertsTargetSize - verifies target size is decreased on timeout
- TestExpiredScaleUpRevertsTargetSizeHandlesError - verifies graceful handling when decrease fails
- TestExpiredScaleUpRevertsPartialIncrease - verifies correct decrease amount for partial increases

Does this PR introduce a user-facing change?

Cluster-autoscaler now automatically reverts the target size of a node group when a scale-up request times out. This prevents cloud providers from indefinitely retrying failed node provisioning attempts after infrastructure capacity limits are hit.

linux-foundation-easycla · 2026-01-27T18:56:31Z

The committers listed above are authorized under a signed CLA.

✅ login: thatmidwesterncoder / name: Jacob Lindgren (517b0eb, ae55033)

k8s-ci-robot · 2026-01-27T18:56:33Z

Welcome @thatmidwesterncoder!

It looks like this is your first PR to kubernetes/autoscaler 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/autoscaler has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2026-01-27T18:56:34Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: thatmidwesterncoder
Once this PR has been reviewed and has the lgtm label, please assign x13n for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

cluster-autoscaler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2026-01-27T18:56:34Z

Hi @thatmidwesterncoder. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

snasovich · 2026-02-10T21:24:13Z

@elmiko / @jackfrancis , hello. Seeing how you've reviewed another community PR wondering if you could look into this one as well (or please redirect to someone else). Thank you in advance, much appreciated.

elmiko · 2026-02-12T13:58:46Z

this seems like a nice fix, thank you @thatmidwesterncoder

/ok-to-test

elmiko

this patch makes sense to me, and i appreciate the addition of the tests too. i have a couple questions and we definitely need to get another reviewer to examine this.

elmiko · 2026-02-12T13:59:48Z

cluster-autoscaler/clusterstate/clusterstate.go

+			// Attempt to revert the failed scale-up by decreasing target size.
+			// This prevents cloud providers from indefinitely retrying failed provisioning attempts.
+			if scaleUpRequest.Increase > 0 {
+				klog.V(2).Infof("Reverting timed-out scale-up for node group %v by decreasing target size by %d",


i wonder if this log should also be warning level?

how frequently does this get emitted during a failure scenario?

i wonder if this log should also be warning level?

That is a good point - I missed that the log on line 302 is a warn as well so it makes sense to use the same level.

how frequently does this get emitted during a failure scenario?

It essentially gets emitted every N minutes equivalent to the max-node-provision-time flag. This is per-node group though, so if multiple node so if the autoscaler is attempting to scale M node groups it would appear N*M times. All in all not very often and won't show up unless something bad is going on with the infrastructure provider.

It essentially gets emitted every N minutes equivalent to the max-node-provision-time flag.

this doesn't seem overly chatty to me, i was concerned that if we made this a warning level log then perhaps it would create too much spam in the logs. but i think it's slow enough that even with multiple node groups in failure it would not be too noisy.

elmiko · 2026-02-12T14:06:35Z

cluster-autoscaler/clusterstate/clusterstate_test.go

+	assert.NoError(t, err)
+
+	// Verify DecreaseTargetSize was called even though it returned an error
+	mockedNodeGroup.AssertCalled(t, "DecreaseTargetSize", -4)


do we have any way to detect that the error path was utilized?

yes, I updated the test to monitor for the events that would only be triggered during the error condition (FailedToRevertScaleUp).

elmiko · 2026-02-12T14:11:35Z

cluster-autoscaler/clusterstate/clusterstate_test.go

+	assert.Nil(t, clusterstate.scaleUpRequests["ng1"])
+}
+
+func TestExpiredScaleUpRevertsPartialIncrease(t *testing.T) {


given the title of this test, i would have expected that one of the two nodes requested for scale up has failed. could you highlight the differences from the TestExpiredScaleUpRevertsTargetSize, i'm having trouble seeing them.

Ah - I should have added some nodes to the mocked node group to make it a little more clear. Essentially I wanted to prove that when we have some nodes come online as ready, but not enough to satisfy the scale request that the autoscaler would still timeout waiting for the nodes, resulting in the entire scale-up operation getting undone. I updated it to make it to demonstrate that a little better hopefully.

thanks! i figured that is what you wanted to test, i was having difficulty seeing the difference.

elmiko

thanks for the updates!

/lgtm

given the nature of this change i definitely think we should get another set of eyes on the review. cc @jackfrancis @towca

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jan 27, 2026

k8s-ci-robot added area/cluster-autoscaler do-not-merge/needs-area labels Jan 27, 2026

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. and removed do-not-merge/needs-area labels Jan 27, 2026

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Jan 27, 2026

k8s-ci-robot requested review from feiskyer and vadasambar January 27, 2026 18:56

thatmidwesterncoder force-pushed the undo_scaleup_on_timeout branch 2 times, most recently from 90291a6 to c06cd0a Compare February 10, 2026 16:10

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 12, 2026

elmiko reviewed Feb 12, 2026

View reviewed changes

thatmidwesterncoder added 2 commits February 12, 2026 08:55

fix: undo Scale operation when a scaleUpRequest times out

517b0eb

change undo-scaleup message to warn and improve unit tests

ae55033

thatmidwesterncoder force-pushed the undo_scaleup_on_timeout branch from c06cd0a to ae55033 Compare February 12, 2026 18:18

elmiko reviewed Feb 12, 2026

View reviewed changes

k8s-ci-robot assigned elmiko Feb 12, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 12, 2026

Conversation

thatmidwesterncoder commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

linux-foundation-easycla bot commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jan 27, 2026

Uh oh!

k8s-ci-robot commented Jan 27, 2026

Uh oh!

k8s-ci-robot commented Jan 27, 2026

Uh oh!

snasovich commented Feb 10, 2026

Uh oh!

elmiko commented Feb 12, 2026

Uh oh!

elmiko left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elmiko left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

thatmidwesterncoder commented Jan 27, 2026 •

edited

Loading

linux-foundation-easycla bot commented Jan 27, 2026 •

edited

Loading