[Feature] Fast fallback on quota exceeded: group-level scaleGate signal on RayCluster

[gangli113]

Search before asking

• I had searched in the issues and found no similar feature requirement.

KubeRay Component

ray-operator

Description

Summary

When KubeRay runs with Kueue, a worker group blocked by quota is indistinguishable
from a slow-starting one, so the Ray Autoscaler waits up to 60 minutes before
falling back to a lower-priority worker group. This issue proposes a group-level
scaleGate signal on the RayCluster CRD so fallback happens within one reconcile
cycle.

Background

The Ray Autoscaler (v2) supports Priority-Aware Worker Group Selection.
Each worker group carries a priority, and when several groups can satisfy the same
resource demand the Autoscaler prefers the higher-priority one. If the preferred
group cannot allocate nodes, the Autoscaler falls back to a lower-priority group.
This fallback loop is the mechanism we want to trigger quickly when a group is
blocked by quota.

Kueue enforces quota through Pod scheduling gates. When a worker group exceeds its
quota, Kueue keeps the newly created Pods gated (spec.schedulingGates contains
kueue.x-k8s.io/admission) and marks the associated Workload with
QuotaReserved=False (Reason: Pending). The Pod is created successfully, so from
the Autoscaler's point of view the node request looks like it is launching slowly,
not failing.

Problem

Current end-to-end behavior when Kueue blocks admission on quota:

1. The KubeRay operator creates the Pod, but it stays gated by kueue.x-k8s.io/admission.
2. Kueue sets the Workload status to QuotaReserved=False (Reason: Pending).
3. The Ray Autoscaler sees a pending Pod, treats the instance as ALLOCATED, and
   waits for RAY_AUTOSCALER_RECONCILE_ALLOCATE_STATUS_TIMEOUT_S (default 3600s)
   before declaring failure and attempting fallback.
4. Result: an hour-long delay before the workload scales up on a fallback group.

Lowering the timeout globally is not a fix — that env var covers the whole
ALLOCATED → RUNNING transition, of which waiting on Kueue is only one part, so a
short value would kill pods that are legitimately still starting.

Goal: signal quota exhaustion for a specific worker group to the Autoscaler
within one reconcile cycle, bypassing the allocate-status timeout.

Proposed design

Add an optional string-list ScaleGate field to ScaleStrategy on each worker group

• KubeRay Operator: Neither reads nor writes this field.
• Kueue (or other schedulers): Add their gate identifier to scaleGate when a worker group is blocked (e.g., due to quota exhaustion), and remove it once admitted. For Kueue, the identifier is kueue.k8s.io/quota-exceeded
• Ray Autoscaler: Reads scaleGate and skips scaling up on that worker group whenever the list is non-empty.

Pros:

• Extensible: Schedulers interacting with RayCluster CRDs can integrate gate signals without modifying Ray or KubeRay codebases.
• Decoupled: KubeRay remains independent and avoids introducing external Go module dependencies.

Cons:

• Scheduler Support: Schedulers that do not natively manage or interact with RayCluster CRDs cannot write to this field.
• Potential Multiple Writers: Currently only Kueue writes to this filed, but potentially more schedulers or other components can write their gate identifier to this field.

API change

ray-operator/apis/ray/v1/raycluster_types.go:

// ScaleStrategy to remove workers
type ScaleStrategy struct {
	// WorkersToDelete workers to be deleted
	WorkersToDelete []string `json:"workersToDelete,omitempty"`
	// ScaleGate contains a list of scaling gates blocking this worker group from
	// scaling up. When Kueue encounters a quota-exceeded error for this worker
	// group, it appends "kueue.k8s.io/quota-exceeded" to ScaleGate.
	// The Autoscaler initiates fallback behavior whenever ScaleGate is non-empty.
	// +optional
	// +listType=set
	ScaleGate []string `json:"scaleGate,omitempty"`
}

Dimension	scaleGate field (this proposal)
Signal scope	Per worker group
Where it lives	RayCluster.spec.workerGroupSpecs[i].scaleStrategy
Who writes it	Kueue
Autoscaler input	Reads one field per group from the cluster spec
Write cardinality	One field per affected group
Discoverability	Visible in the RayCluster spec
CRD schema change	Yes

Open design questions

Raised during review of #5094:

Extensibility to other schedulers. Schedulers other than Kueue (Volcano,
YuniKorn, scheduler-plugins, KAI) are not aware of KubeRay CRDs and would not
write a RayCluster spec.

Alternative A: KubeRay derives the signal and surfaces it in RayCluster.status. The Ray Autoscaler would
consume the gate signal from RayCluster.status instead of spec.

Pros:

• Can support schedulers that do not interact directly with RayCluster custom resources.

Cons:

• Introduces cross-project Go module dependencies into the KubeRay operator.
• every scheduler support needs kuberay side change

Alternative B: Ray Autoscaler reads scheduler signals directly
The Autoscaler bypasses KubeRay entirely to query scheduler APIs (such as Kueue's Workload CRD) directly.

Pros:

• Can support schedulers that do not interact directly with RayCluster custom resources.

Cons:

• Introduces scheduler-specific dependencies directly into the Ray Autoscaler.
• every scheduler support needs ray side change

Add a scaleGate field to the RayCluster CR, plus a second field indicating whether scaleGate is owned by Kueue. If it is owned by Kueue, KubeRay only propagates the information from the CR spec to the RayCluster CR status. If not, KubeRay reads the scheduler's PodGroup CRs and writes the information into the RayCluster CR status.

Scheduler can define there own gate identifier, for Kueue, it's kueue.k8s.io/quota-exceeded. The other logic is not needed for current proposal, as worker group will be gated as long as scaleGate is not empty.

multiple writers to ScaleStrategy field

• Ray autoscaler raplace entire ScaleStrategy in worker_delete_patch
  -- solution: modify worker_delete_patch logic to only modify workersToDelete in ScaleStrategy

• Kueue and potential other scheduler write to ScaleGate filed.
  -- solution: follow the existing example PodSchedulingGate to use +listType=map

Proof of concept

• KubeRay: feat: add ScaleGate field to ScaleStrategy for fast fallback on quota exceeded #5094
• Ray: [Autoscaler][V2] Add ALLOCATION_GATED status for scaleGate fast fallback ray#65478
• Kueue: Add RayCluster fast fallback on quota exceeded via scaling gate kubernetes-sigs/kueue#14656

Use case

No response

Related issues

#4846
ray-project/ray#62996

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[gangli113]

/cc @andrewsykim @kevin85421 @win5923 @machichima @gangli113

[machichima]

Thank you! I'm actually thinking if Kueue can be the one adding label to pod. Then KubeRay Autoscaler can just read label from the pod.
WDYT?

[kevin85421]

Hi @gangli113,

Could you read #5094 (review) and revise the design based on the points I mentioned?

The design should focus on:

• The CRD changes
• Which component (ex: KubeRay, Kueue, other schedulers) writes each field
• Which component (ex: KubeRay, Ray Autoscaler) reads each field
• The contract between the Ray Autoscaler and the RayCluster CRD (basically, it should be covered by the above two questions)

Details such as “Ray Autoscaler — canceling the in-flight request” and “Ray Autoscaler — avoiding reselecting the gated group” are distracting and should be omitted.

[gangli113]

Hi @gangli113,

Could you read #5094 (review) and revise the design based on the points I mentioned?

The design should focus on:

• The CRD changes
• Which component (ex: KubeRay, Kueue, other schedulers) writes each field
• Which component (ex: KubeRay, Ray Autoscaler) reads each field
• The contract between the Ray Autoscaler and the RayCluster CRD (basically, it should be covered by the above two questions)

Details such as “Ray Autoscaler — canceling the in-flight request” and “Ray Autoscaler — avoiding reselecting the gated group” are distracting and should be omitted.

@kevin85421 Thanks for the feedback.

I have revised the issue description based on your review:

• Removed redundant implementation details (such as canceling in-flight requests and avoid-reselecting logic).
• Explicitly documented component read/write responsibilities and the contract between KubeRay, schedulers, and the Ray Autoscaler.
• Added pros and cons for the proposed design.

The core proposal remains the same for now, but I have moved your suggestions and alternative architectural approaches into the Open design questions section—along with their respective pros and cons—to facilitate further discussion.

Thank you! I'm actually thinking if Kueue can be the one adding label to pod. Then KubeRay Autoscaler can just read label from the pod. WDYT?

@machichima By "pod", do you mean standard Kubernetes worker Pods created for the worker group?

If so, having Kueue label gated Pods could let the Ray Autoscaler inspect Pod labels directly without altering the RayCluster CRD. However, a drawback is that it adds dependency from KubeRay/Ray to Kueue. and more dependency if we want the functionality to be extended to other schedulers.

Currently, Kueue surfaces admission status at the job level on its internal Workload CRD (e.g., in the QuotaReserved=False condition message). Having KubeRay or the Ray Autoscaler watch and parse Kueue Workload status directly would avoid CRD changes, but it introduces cross-project Go/Python dependencies on Kueue and doesn't scale cleanly to other schedulers (Volcano, YuniKorn). I've captured this approach under Alternative B in the Open design questions section.

exemaple:

    conditions:
    - lastTransitionTime: "2026-06-22T21:08:28Z"
      message: 'couldn''t assign flavors to pod set **gpu-worker-group**: 
        flavor default-cpu-flavor doesn''t match node affinity, 
        flavor dws-flex-flavor doesn''t match node affinity, 
        flavor spot-flavor doesn''t match node affinity, 
        insufficient quota for cpu in flavor reservation-flavor, 
        previously considered podsets requests (2) + current podset request (5) > maximum capacity (4)'
      observedGeneration: 1
      reason: Pending
      status: "False"
      type: QuotaReserved