feat: add opt-in scheduling gates for queue capacity control

[devzizu]

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds opt-in scheduling gates for queue capacity control to prevent cluster autoscalers from triggering unnecessary scale-ups when pods are waiting for queue admission rather than cluster capacity.

When a pod opts in via the scheduling.volcano.sh/queue-allocation-gate: "true" annotation, Volcano adds a scheduling gate that delays the Unschedulable condition until the queue has capacity.

The design is further described in #4727.

Which issue(s) this PR fixes:

Fixes #4710

Special notes for your reviewer:

• This feature is opt-in via pod annotation to ensure backward compatibility.
• A simple E2E test was added to validate capacity reservation behavior, but further test cases may be required.

Does this PR introduce a user-facing change?

Add opt-in scheduling gates for queue capacity control. Pods can now use the scheduling.volcano.sh/queue-allocation-gate: "true" annotation to prevent false autoscaler scale-ups when waiting for queue admission.

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign shinytang6 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gemini-code-assist]

Summary of Changes

Hello @devzizu, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical issue where Volcano's scheduler could cause unnecessary cluster autoscaler scale-ups by marking pods as 'Unschedulable' even when they were only waiting for queue capacity. By leveraging Kubernetes scheduling gates, it introduces an opt-in mechanism to delay the 'Unschedulable' condition until a pod is truly blocked by a lack of cluster resources, while also ensuring proper queue capacity accounting for pods that have passed initial admission but are awaiting node allocation.

Highlights

• Opt-in Scheduling Gates for Queue Capacity Control: Introduces an opt-in mechanism using the scheduling.volcano.sh/queue-allocation-gate: "true" annotation. Pods with this annotation will have a Volcano-specific scheduling gate added by the MutatingAdmissionWebhook at creation time.
• Asynchronous Gate Removal: The scheduler now asynchronously removes the Volcano scheduling gate from pods that fail predicate checks or prioritization (i.e., cannot find a suitable node). This signals to cluster autoscalers that a scale-up is needed, preventing false scale-ups when pods are merely waiting for queue admission.
• Queue Capacity Reservation for Ungated Pods: Implements a new mechanism in the capacity plugin to reserve queue capacity for pods that have had their Volcano scheduling gate removed but are still pending. This prevents other pods from consuming the capacity intended for the ungated pod, addressing a potential race condition.
• Gate Removal on Successful Bind: For pods that successfully pass all allocation checks and are bound to a node, the Volcano scheduling gate is removed during the binding operation.
• Enhanced Scheduler Logic: Updates the scheduler's allocate action to manage the lifecycle of these scheduling gates, including skipping externally gated pods, queuing gate removals, and integrating with the new capacity reservation logic.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• docs/design/scheduling-gates-queue-admission.md
  • Added a new design document detailing the motivation, proposal, and high-level implementation of gate-controlled scheduling for cluster autoscaler compatibility.
• pkg/scheduler/actions/allocate/allocate.go
  • Introduced asynchronous workers for scheduling gate removal to avoid blocking the main scheduler loop.
  • Modified allocateResourcesForTasks to enqueue gate removal for pods that fail pre-predicate, predicate, or prioritization steps.
  • Updated organizeJobWorksheet to only skip tasks with external scheduling gates, allowing Volcano-managed gates to be processed.
  • Added Initialize and UnInitialize methods to manage the lifecycle of the async gate removal workers.
• pkg/scheduler/api/helpers.go
  • Added HasOnlyVolcanoSchedulingGate helper function to identify pods with only the Volcano-specific scheduling gate.
  • Added HasQueueAllocationGateAnnotation helper function to check for the opt-in annotation.
• pkg/scheduler/api/job_info.go
  • Added RemoveGateDuringBind boolean field to TaskInfo to signal if a gate should be removed during the bind operation.
  • Modified GetSchGatedPodResources to exclude pods with only the Volcano scheduling gate from resource deduction, ensuring they are counted in in-queue resources.
• pkg/scheduler/api/types.go
  • Defined CleanupReservationsFn type for functions that clean up reserved resources before a statement commit.
• pkg/scheduler/cache/cache.go
  • Updated the Bind method to remove the Volcano scheduling gate from a pod if task.RemoveGateDuringBind is true before performing the standard bind operation.
  • Adjusted RecordJobStatusEvent to skip updating the PodScheduled condition if taskInfo.SchGated is true, respecting the scheduling gate's role.
• pkg/scheduler/cache/util.go
  • Added RemoveVolcanoSchGate function to programmatically remove the Volcano scheduling gate from a pod using a JSON patch.
  • Added AddVolcanoSchGate function to programmatically add the Volcano scheduling gate to a pod using a JSON patch.
• pkg/scheduler/framework/session.go
  • Added cleanupReservationsFns map to store registered cleanup functions.
• pkg/scheduler/framework/session_plugins.go
  • Added AddCleanupReservationsFn to allow plugins to register cleanup logic.
  • Implemented CleanupReservations method to execute all registered cleanup functions before a statement commit.
• pkg/scheduler/framework/statement.go
  • Added Name() and Task() methods to the operation struct for easier access to operation details.
• pkg/scheduler/plugins/capacity/capacity.go
  • Introduced queueGateReservedTasks cache to track pods that have passed capacity checks but are not yet allocated, reserving their queue capacity.
  • Added addTaskToReservedCache, removeTaskFromReservedCache, and buildQueueReservedTasksCache methods to manage this reservation cache.
  • Implemented queueAllocatableWithReserved to include reserved resources in capacity checks.
  • Integrated the reservation cleanup mechanism by registering a CleanupReservationsFn in OnSessionOpen.
• pkg/webhooks/admission/pods/mutate/mutate_pod.go
  • Added patchSchedulingGates function to inject the scheduling.volcano.sh/queue-allocation-gate into a pod's spec.schedulingGates if the opt-in annotation is present.
• staging/src/volcano.sh/apis/pkg/apis/scheduling/v1beta1/labels.go
  • Defined QueueAllocationGateKey constant for the new scheduling gate annotation and name.
• test/e2e/schedulingaction/allocate.go
  • Added a new E2E test case to validate the queue capacity reservation behavior, demonstrating that an unschedulable pod with a removed gate reserves capacity and blocks other pods.
• test/e2e/util/job.go
  • Added Annotations field to TaskSpec to support setting annotations on pods created in E2E tests.
• test/e2e/util/pod.go
  • Added PodHasOnlyVolcanoSchedulingGate and HasSchedulingGatedCondition helper functions for E2E tests to check pod scheduling gate status.

Activity

• The pull request introduces a new feature to address a known issue (Cluster Autoscaler node scale-up for Pods that exceed Queue's capability #4710) regarding cluster autoscaler compatibility.
• A detailed design document (docs/design/scheduling-gates-queue-admission.md) was added to explain the motivation and implementation of this feature.
• A simple E2E test was included to validate the new capacity reservation behavior.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant and well-designed feature to add opt-in scheduling gates for queue capacity control, improving compatibility with cluster autoscalers. While the overall design is comprehensive and includes a valuable E2E test, two significant security-related issues were identified: a data race in the asynchronous gate removal workers that could lead to a scheduler panic (Denial of Service), and a race condition in the JSON Patch logic for removing scheduling gates that could result in the unauthorized removal of security-related gates (Broken Access Control). Both of these critical security issues must be addressed before merging. Additionally, there are minor suggestions for a grammatical correction in the design document and replacing context.TODO() with context.Background() in background operations for improved context management.

[JesseStutler]

I still have some opinions on the proposal and please take a look @devzizu #4727, thanks, we may need to furthur discuss it on the community meeting

[Copilot]

Pull request overview

This PR implements opt-in scheduling gates for queue capacity control to prevent cluster autoscalers from triggering unnecessary scale-ups when pods are waiting for queue admission rather than cluster capacity. The solution addresses issue #4710 by using Kubernetes scheduling gates to delay the Unschedulable condition until pods pass queue capacity checks.

Changes:

• Added opt-in annotation scheduling.volcano.sh/queue-allocation-gate: "true" for pods to participate in gate-based queue admission control
• Implemented asynchronous gate removal workers in the allocate action to remove gates when pods pass capacity checks
• Enhanced capacity plugin with reserved task tracking to prevent race conditions where ungated-but-unscheduled pods could allow queue over-allocation
• Added comprehensive E2E tests and design documentation

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
staging/src/volcano.sh/apis/pkg/apis/scheduling/v1beta1/labels.go	Added QueueAllocationGateKey constant for the opt-in annotation
pkg/scheduler/api/helpers.go	Added helper functions to check for Volcano scheduling gates and annotations
pkg/webhooks/admission/pods/mutate/mutate_pod.go	Added webhook logic to inject scheduling gates for opted-in pods
pkg/scheduler/actions/allocate/allocate.go	Implemented async worker infrastructure and gate removal logic in allocate action
pkg/scheduler/plugins/capacity/capacity.go	Added reserved task cache to account for ungated pods in capacity checks
pkg/scheduler/framework/statement.go	Added Name() and Task() methods to operation struct for cleanup callbacks
pkg/scheduler/framework/session.go	Added cleanupReservationsFns map to session
pkg/scheduler/framework/session_plugins.go	Added AddCleanupReservationsFn and CleanupReservations methods
pkg/scheduler/api/types.go	Added CleanupReservationsFn function type
pkg/scheduler/api/job_info.go	Modified GetSchGatedPodResources to exclude Volcano-only gated pods
pkg/scheduler/cache/util.go	Added RemoveVolcanoSchGate utility function for gate removal
pkg/scheduler/cache/cache.go	Added synchronous gate removal before bind as safety guarantee
test/e2e/schedulingaction/allocate.go	Added comprehensive E2E test for capacity reservation behavior
test/e2e/util/pod.go	Added helper functions for checking gate status in tests
test/e2e/util/job.go	Added Annotations field to TaskSpec for test configuration
pkg/scheduler/util/test_utils.go	Updated test utilities to handle SubGroupPolicy without NetworkTopology
pkg/scheduler/actions/allocate/allocate_test.go	Added unit test for SubGroupPolicy without NetworkTopology
docs/design/scheduling-gates-queue-admission.md	Added comprehensive design document explaining the feature
docs/user-guide/version-compatibility-archive.md	Added version compatibility archive document
README.md	Updated Kubernetes compatibility table

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hajnalmt]

This implementation is more than solid thank you @devzizu ! When Jesse comes back we can schedule a meeting.

Please squash the commits and let's keep the design doc in the separate PR!

[hajnalmt]

I gave an other round what do you think about my comments?

[hajnalmt]

Nor Initialize() nor UnInitialize() is used in our codebase except the unit test helper:

volcano/pkg/scheduler/uthelper/helper.go

Lines 224 to 229 in 49b8f40

	for _, action := range actions {
	action.Initialize()
	action.Execute(test.ssn)
	action.UnInitialize()
	}
	}

What a bummer. Can you create at least a follow up ticket for this please? Currently nothing will close these channels. If this gets merged in we should fix this, so this is not just dead code here.

[hajnalmt]

I also think that this is out of the scope of this feature.

[devzizu]

Oh, this is a surprise for me -- I was counting on it to be called, but indeed I only see it as a test helper 😄

I actually call it here:

https://github.com/devzizu/volcano/blob/1d6ae98c6c3c32c122ad57586a88125f8ee4655a/pkg/scheduler/actions/allocate/allocate.go#L211-L214

P.S. Perhaps we can add this to #5048?

[devzizu]

Created #5052 to followup 👍

[hajnalmt]

Can we introduce a unit test for this function? A test that directly calls it?

I know there is no util_test.go now, but probably we need it to have this verified. I ran into issues where the strategic merge didn't replace and kept the old values, but it would probably come up in the e2e test if this wouldn't work. Nevertheless let's have this to verify that this function works properly.

[devzizu]

Indeed, just added a new test and created the new util_test.go as suggested. Please take a look 🙏

a7286f6