[Bug] scheduler-plugins PodGroup is created once and never updated, breaking gang scheduling on scale and suspend

[CheyuWu]

Search before asking

• I searched the issues and found no similar issues.

KubeRay Component

ray-operator

KubeRay Version

v1.7.0

Ray Version

2.52.0

Environment

• Kubernetes version: v1.31.0
• Kubernetes cluster type: kind
• Helm chart version (if applicable): 1.1.0 (helm-chart/kuberay-operator)
• Installation method (Helm / Kustomize / YAML): Helm, with --set batchScheduler.name=scheduler-plugins
• OS: Linux

What happened + What you expected to happen

What happened

With the scheduler-plugins batch scheduler enabled, the PodGroup is created exactly once and never reconciled again. DoBatchSchedulingOnSubmission only creates it on the IsNotFound path — when the Get succeeds it returns without comparing or updating MinMember and MinResources:

kuberay/ray-operator/controllers/ray/batchscheduler/scheduler-plugins/scheduler_plugins.go

Lines 72 to 93 in feeaf72

	func (k *KubeScheduler) DoBatchSchedulingOnSubmission(ctx context.Context, object metav1.Object) error {
	app, ok := object.(*rayv1.RayCluster)
	if !ok {
	return fmt.Errorf("currently only RayCluster is supported, got %T", object)
	}
	if !k.isGangSchedulingEnabled(app) {
	return nil
	}
	podGroup := &v1alpha1.PodGroup{}
	if err := k.cli.Get(ctx, ktypes.NamespacedName{Namespace: app.Namespace, Name: app.Name}, podGroup); err != nil {
	if !errors.IsNotFound(err) {
	return err
	}
	podGroup = createPodGroup(app)
	if err := k.cli.Create(ctx, podGroup); err != nil {
	if errors.IsAlreadyExists(err) {
	return nil
	}
	return fmt.Errorf("failed to create PodGroup: %w", err)
	}
	}
	return nil

There is no other path that could correct the values either: CleanupOnCompletion is a no-op, and ConfigureReconciler does not watch the PodGroup type.

So any spec change that alters the desired pod count leaves the gang gate stale:

1. Scale down deadlocks the cluster. MinMember stays at the old, larger value. Already-Running pods keep running, but as soon as any pod of the group must be scheduled again (eviction, node restart), coscheduling rejects the whole group at PreFilter and every pod stays Pending forever.
2. Scale up silently voids the gang guarantee. MinMember stays at the old, smaller value, so the group is admitted before the full gang can be placed, and MinResources is under-reserved.
3. Suspend leaves a stale gang gate behind. spec.suspend: true deletes all pods, but the PodGroup keeps its pre-suspend values, poisoning later scheduling rounds (e.g. unsuspend after a resize). Same class as [Bug] RayCluster suspend leaks the Volcano PodGroup, leaving queue resources reserved #5183 / [Bug] RayJob suspend leaks the Volcano PodGroup, leaving queue resources reserved #4939 (Volcano, suspend-scoped), but broader: here nothing ever issues an Update at all — and the plugin is not even invoked while suspended, since reconcilePods returns before reaching it.

(1) and (2) need no suspend, autoscaling, or feature gate — a plain kubectl patch on the replica counts is enough.

What you expected to happen

The PodGroup should track the RayCluster spec (the Volcano integration keeps its PodGroup in sync): scaling should be reflected in MinMember and MinResources, and a suspended cluster should not keep a stale gang gate around.

Reproduction script

# 0. Install the scheduler-plugins coscheduling plugin as a second scheduler.
#    NOTE: use the release matching your k8s minor (v0.31.8 for k8s 1.31);
#    the chart on scheduler-plugins master pins v0.34.x, which hangs on k8s 1.31.
helm upgrade --install kuberay-operator kuberay/kuberay-operator \
  --set batchScheduler.name=scheduler-plugins

# 1. Gang-scheduled RayCluster: 1 head + 2 workers.
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.scheduler-plugins.yaml
kubectl wait --for=condition=Ready pod -l ray.io/cluster=test-podgroup-0 --timeout=300s

# 2. Baseline is correct: MinMember=3, MinResources={cpu:3, memory:4Gi}.
kubectl get podgroup test-podgroup-0 \
  -o custom-columns=NAME:.metadata.name,MINMEMBER:.spec.minMember,MINRES:.spec.minResources

# 3. Scale the worker group down from 2 to 1.
#    (minReplicas/maxReplicas must move too — GetWorkerGroupDesiredReplicas clamps
#    replicas into [min, max], and the sample pins both to 2.)
kubectl patch raycluster test-podgroup-0 --type='json' -p='[
  {"op":"replace","path":"/spec/workerGroupSpecs/0/replicas","value":1},
  {"op":"replace","path":"/spec/workerGroupSpecs/0/minReplicas","value":1},
  {"op":"replace","path":"/spec/workerGroupSpecs/0/maxReplicas","value":1}
]'

# 4. BUG: PodGroup still says MinMember=3 / cpu:3 / memory:4Gi,
#    though only 2 pods will ever exist. (Expected: MinMember=2 / cpu:2 / 3Gi.)
kubectl get podgroup test-podgroup-0 \
  -o custom-columns=NAME:.metadata.name,MINMEMBER:.spec.minMember,MINRES:.spec.minResources

# 5. The stale gate bites once pods need rescheduling — simulate an eviction:
kubectl delete pod -l ray.io/cluster=test-podgroup-0

# 6. BUG: recreated pods stay Pending forever.
kubectl get pod -l ray.io/cluster=test-podgroup-0
kubectl get events --field-selector reason=FailedScheduling | tail -5
#   "... cannot find enough sibling pods, current pods number: 2, minMember of group: 3"

Variants of step 3:

• Scale up to 4 workers: all 5 pods are admitted while the PodGroup still says MinMember: 3 (expected: 5 / {cpu:5, memory:6Gi}).
• Suspend ({"spec":{"suspend":true}}): all pods deleted, PodGroup keeps MinMember: 3 / {cpu:3, memory:4Gi}.

Anything else

Are you willing to submit a PR?

• Yes I am willing to submit a PR!