[Proposal] Gate-Controlled Scheduling for Cluster Autoscalers Compatibility

[devzizu]

What type of PR is this?

/kind documentation
/kind feature

What this PR does / why we need it:

This design proposal addresses an issue where Volcano incorrectly signals cluster autoscalers (e.g., CA or Karpenter) to scale up nodes even when pods are only waiting for queue capacity, not cluster resources. Currently, Volcano marks all unallocated pods as Unschedulable regardless of the reason, causing autoscalers to interpret queue constraints as insufficient node capacity.

This proposal introduces an opt-in feature using Kubernetes schedulingGates to hide queue-constrained pods from autoscalers, ensuring scale-up operations only trigger for legitimate node-fit failures. This aims to prevent unnecessary infrastructure costs and resource waste.

Which issue(s) this PR fixes:

Fixes #4710

[volcano-sh-bot]

Welcome @devzizu! It looks like this is your first PR to volcano-sh/volcano 🎉

[JesseStutler]

@kingeasternsun @hajnalmt Do you have time help @devzizu to take a look at this proposal?

[hajnalmt]

/cc @hajnalmt

Sure I am starting to review it 👍

[hajnalmt]

Thank you for this great proposal @devzizu.
I am looking forward to the implementation, great idea and solid design change.

I had mostly minor remarks except the capacity plugin change.
What I see is that the pods that are marked scheduling gated by volcano after this change with the annotation they can maybe considered to be inqueue but straight out to be considered allocated seems a too significant change.

Wouldn’t it be sufficient to adjust the DeductSchGatedResources function so that the tasks that are only volcano scheduling gated won't be deducted when hasOnlyVolcanoSchedulingGate returns true? This approach would also maintain compatibility with the overcommit plugin, which I’m slightly concerned about since it heavily depends on scheduling gate behavior.
--- I want to remark that this would mean that these annotated pods would move to inqueue and would never be in the scope of the enqueue action if every other scheduling gate is removed, which is probably what we want.

Thank you for the contribution and the idea once more!

[hajnalmt]

Thank you so much for the updates @devzizu !
The reserved cache is a good idea, but I still had so many questions, please find them below!

[hajnalmt]

We are getting there! Keep up the great work 😊

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign william-wang for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hajnalmt]

Great Work!

Please squash your commits, and sign the DCO! I think this is quite ready.
Do you need any help with the implementation? We will need to add an e2e test for this too.
I think the it shall happen in a different PR though as the design is quite big.

[hajnalmt]

/cc @JesseStutler

I think we can pull this in release-1.15

[Copilot]

Pull request overview

This PR introduces a comprehensive design proposal for addressing a critical compatibility issue between Volcano scheduler and cluster autoscalers (CA/Karpenter). The proposal leverages Kubernetes scheduling gates to prevent unnecessary cluster scale-ups when pods are waiting for queue capacity rather than node capacity.

Changes:

• Adds a detailed design document proposing an opt-in feature using scheduling gates to differentiate between queue capacity constraints and cluster capacity constraints
• Introduces queue reservation mechanism to prevent race conditions when ungated pods remain unscheduled
• Proposes extensions to the admission webhook, scheduler actions, and capacity plugin to implement the gate-controlled scheduling

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The reserved cache accounting could lead to resource starvation in certain scenarios. Consider: if pod-A fails to schedule due to node constraints and gets its gate removed, it reserves queue capacity. If new pods (pod-B, pod-C) are created that could actually schedule, they will be blocked by pod-A's reservation even though pod-A can't make progress until the autoscaler provisions new nodes. The proposal should discuss whether there should be a timeout or mechanism to release reservations for pods that have been ungated but haven't made progress for an extended period, preventing indefinite queue capacity blocking.

[devzizu]

This is actually intended design, but the concern is a valid point: an ungated-but-unschedulable pod can hold queue capacity and block other pods that could schedule. We could add a timeout or similar mechanism to release reservations for pods that stay unschedulable for a long time, but that would need a clear policy and might over-commit the queue when those pods eventually get nodes.

@JesseStutler @hajnalmt – Do you think we should address this in the initial implementation (e.g. document the tradeoff and/or add a reservation timeout), or treat it as follow-up work once we see how it behaves in practice?

[hajnalmt]

I think this is fine and this is the "standard" way of things happening. I am fine with it.