[RayJob] Wait for workers before submitted jobs in sidecar mode by marosset · Pull Request #4429 · ray-project/kuberay

marosset · 2026-01-21T22:52:19Z

Why are these changes needed?

In SideCarMode, the submitted can submit jobs before worker are connected to the Ray cluster causing jobs to run on the head node.

This fix adds a wait loop that polls the Ray Dashboard API to verify that minReplicas workers are joined to the cluster before submitting the job. The minimum expected worker count is retrieved from the spec and passed via RAY_EXPECTED_MIN_WORKERS env var

Related issue number

Fixes #4199

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

ray-operator/controllers/ray/common/job.go

Future-Outlier

Hi, can you help me merge master?

marosset · 2026-01-27T19:07:26Z

Hi, can you help me merge master?

rebased - thanks!

Future-Outlier

cc @fsNick @troychiu @CheyuWu @machichima @400Ping to help review

Future-Outlier

high level looks good

ray-operator/controllers/ray/common/job_test.go

win5923

~~LGTM!~~

Sorry, just find some error need to be fixed.

ray-operator/controllers/ray/common/job.go

Signed-off-by: Mark Rossett <marosset@microsoft.com>

Future-Outlier

if you want the job runs in Ray cluster's worker node, I think you can just set rayStartParams's num-cpus: "0" like this, (you can try this using master branch)

  rayClusterSpec:
    rayVersion: '2.52.0'
    headGroupSpec:
      rayStartParams:
        num-cpus: "0"

I think make sidecar mode's behavior similar to k8s job mode is a more correct behavior, so I'm willing to accept this PR.

correct behavior

win5923

LGTM, thank you!

Also linking #3837 since we’re planning to replace wget with another alternative tool.

andrewsykim · 2026-01-30T20:39:41Z

ray-operator/controllers/ray/rayjob_controller.go

+
+	// In SidecarMode, pass the expected minimum worker count so the submitter can wait for workers to register
+	if submissionMode == rayv1.SidecarMode && rayJobInstance.Spec.RayClusterSpec != nil {
+		minWorkers := common.GetMinReplicasFromSpec(rayJobInstance.Spec.RayClusterSpec)


minReplicas can be 0 or a low number, I wonder if we should only check min replicas when autoscaling is enabled. If autoscaling is not enabled, we should just check replicas.

I can change this and push as a separate commit.
I'll defer to others for what the best number of workers to wait/check for is

we can use this function.

kuberay/ray-operator/controllers/ray/utils/util.go

Lines 410 to 418 in 79b5c30

// CalculateDesiredReplicas calculate desired worker replicas at the cluster level

func CalculateDesiredReplicas(ctx context.Context, cluster *rayv1.RayCluster) int32 {

count := int32(0)

for _, nodeGroup := range cluster.Spec.WorkerGroupSpecs {

count += GetWorkerGroupDesiredReplicas(ctx, nodeGroup)

}

return count

}

Since RayJob in Kubernetes Job mode waits for the RayCluster to reach the Ready state, and the RayCluster’s Ready state requires all pods to be running, including both the head pod and worker pods. The number of worker pods is determined by CalculateDesiredReplicas.

Future-Outlier

In SideCarMode, the submitted can submit jobs before worker are connected to the Ray cluster causing jobs to run on the head node.

if the PR's goal is to wait for worker started, you can use config like this, then ray's scheduler will not schedule jobs on the head node.

  rayClusterSpec:
    rayVersion: '2.52.0'
    headGroupSpec:
      rayStartParams:
        num-cpus: "0"

Future-Outlier · 2026-02-02T05:24:48Z

ray-operator/controllers/ray/common/job.go

+// GetMinReplicasFromSpec calculates the minimum expected worker replicas from the RayClusterSpec.
+// This is used in SidecarMode to determine how many workers should be registered before submitting the job.
+func GetMinReplicasFromSpec(rayClusterSpec *rayv1.RayClusterSpec) int32 {
+	if rayClusterSpec == nil {
+		return 0
+	}
+	count := int32(0)
+	for _, nodeGroup := range rayClusterSpec.WorkerGroupSpecs {
+		if nodeGroup.Suspend != nil && *nodeGroup.Suspend {
+			continue
+		}
+		minReplicas := int32(0)
+		if nodeGroup.MinReplicas != nil && *nodeGroup.MinReplicas > 0 {
+			minReplicas = *nodeGroup.MinReplicas
+		} else if nodeGroup.Replicas != nil && *nodeGroup.Replicas > 0 {
+			// Fall back to Replicas when MinReplicas is not set or is 0.
+			// This handles static clusters where users only set Replicas.
+			minReplicas = *nodeGroup.Replicas
+		}
+		count += minReplicas * nodeGroup.NumOfHosts
+	}
+	return count
+}
+


Can we use the function CalculateDesiredReplicas?

kuberay/ray-operator/controllers/ray/utils/util.go

Lines 410 to 418 in 79b5c30

// CalculateDesiredReplicas calculate desired worker replicas at the cluster level

func CalculateDesiredReplicas(ctx context.Context, cluster *rayv1.RayCluster) int32 {

count := int32(0)

for _, nodeGroup := range cluster.Spec.WorkerGroupSpecs {

count += GetWorkerGroupDesiredReplicas(ctx, nodeGroup)

}

return count

}

Future-Outlier · 2026-02-02T05:38:35Z

ray-operator/controllers/ray/common/job.go

+		// Wait for the expected number of worker nodes to register for the Ray cluster.
+		// RAY_EXPECTED_MIN_WORKERS is set by the controller based on the MinReplicas in the RayClusterSpec.
+		// The loop queries the Ray Dashboard API to get the number of alive nodes and
+		// continues until the number of alive nodes is equal to (expected_workers + 1) for head node.
+		// This ensures that worker pods are connected before the job is submitted otherwise
+		// the jobs may run on the Head node.
+		//
+		// Note: This loop will never timeout and will wait indefinitely if workers never register.
+		// This can be mitigated by setting the RayJob's `activeDeadlineSeconds` field
+		// to enforce a maximum job execution time.
+		//
+		// The wget command includes the x-ray-authorization header if RAY_AUTH_TOKEN is set.
+		// This is required when Ray auth token mode is enabled, otherwise the request will fail with 401.
+		wgetAuthHeader := "${" + utils.RAY_AUTH_TOKEN_ENV_VAR + ":+--header \"x-ray-authorization: Bearer $" + utils.RAY_AUTH_TOKEN_ENV_VAR + "\"}"
+		waitForNodesLoop := []string{
+			"if", "[", "-n", "\"$" + utils.RAY_EXPECTED_MIN_WORKERS + "\"", "]", "&&", "[", "\"$" + utils.RAY_EXPECTED_MIN_WORKERS + "\"", "-gt", "\"0\"", "]", ";", "then",
+			"EXPECTED_NODES=$(($" + utils.RAY_EXPECTED_MIN_WORKERS + " + 1))", ";",
+			"echo", strconv.Quote("Waiting for $EXPECTED_NODES nodes (1 head + $" + utils.RAY_EXPECTED_MIN_WORKERS + " workers) to register..."), ";",
+			"until", "[",
+			"\"$(wget " + wgetAuthHeader + " -q -O- " + address + "/nodes?view=summary 2>/dev/null | python3 -c \"import sys,json; d=json.load(sys.stdin); print(len([n for n in d.get('data',{}).get('summary',[]) if n.get('raylet',{}).get('state','')=='ALIVE']))\" 2>/dev/null || echo 0)\"",
+			"-ge", "\"$EXPECTED_NODES\"", "]", ";",
+			"do", "echo", strconv.Quote("Waiting for Ray nodes to register. Expected: $EXPECTED_NODES ..."), ";", "sleep", "2", ";", "done", ";",
+			"echo", strconv.Quote("All expected nodes are registered."), ";",
+			"fi", ";",
+		}
+		cmd = append(cmd, waitForNodesLoop...)


Can we use something like ray list nodes using ray's CLI?
we are removing wget

@Future-Outlier - I pushed a commit that uses python's urllib instead of wget.
Is this OK instead of ray list nodes?

marosset · 2026-02-02T23:32:29Z

In SideCarMode, the submitted can submit jobs before worker are connected to the Ray cluster causing jobs to run on the head node.

if the PR's goal is to wait for worker started, you can use config like this, then ray's scheduler will not schedule jobs on the head node.
  rayClusterSpec:
    rayVersion: '2.52.0'
    headGroupSpec:
      rayStartParams:
        num-cpus: "0"

This PR is to address #4199 which does look like the primary issue is make sure the jobs do not get scheduled to the head node.

@Future-Outlier - Let me know if we you think it would still be worthwhile to iterate on this PR or if simply specifying the headGroupSpec.rayStateParams.num-cpus: "0" is sufficient.

Future-Outlier · 2026-02-03T07:36:46Z

Hi, @marosset it's great to accept this PR, since we want to make k8s job mode and sidecar mode as similar as possible.

marosset · 2026-02-03T18:04:36Z

Hi, @marosset it's great to accept this PR, since we want to make k8s job mode and sidecar mode as similar as possible.

Great, I'll address the above feedback then!

Signed-off-by: Mark Rossett <marosset@microsoft.com>

…ecar sumitter jobs Signed-off-by: Mark Rossett <marosset@microsoft.com>

Future-Outlier · 2026-02-05T03:04:36Z

Hi, @marosset do you mind fix CI's err?

marosset requested review from MortalHappiness, andrewsykim, kevin85421 and rueian as code owners January 21, 2026 22:52

marosset assigned Future-Outlier Jan 21, 2026

marosset force-pushed the sidecar-submitter-not-waiting-fix branch 2 times, most recently from 2600631 to 4d404a3 Compare January 22, 2026 00:30

win5923 reviewed Jan 24, 2026

View reviewed changes

ray-operator/controllers/ray/common/job.go Outdated Show resolved Hide resolved

marosset force-pushed the sidecar-submitter-not-waiting-fix branch from 4d404a3 to a665203 Compare January 27, 2026 00:01

Future-Outlier reviewed Jan 27, 2026

View reviewed changes

marosset force-pushed the sidecar-submitter-not-waiting-fix branch from a665203 to 30bfb39 Compare January 27, 2026 19:06

Future-Outlier reviewed Jan 28, 2026

View reviewed changes

Future-Outlier changed the title ~~Wait for workers before submitted jobs in sidecar mode~~ [RayJob] Wait for workers before submitted jobs in sidecar mode Jan 28, 2026

Future-Outlier reviewed Jan 28, 2026

View reviewed changes

win5923 reviewed Jan 28, 2026

View reviewed changes

ray-operator/controllers/ray/common/job_test.go Outdated Show resolved Hide resolved

win5923 approved these changes Jan 28, 2026

View reviewed changes

win5923 reviewed Jan 28, 2026

View reviewed changes

ray-operator/controllers/ray/common/job.go Show resolved Hide resolved

win5923 self-requested a review January 28, 2026 17:31

marosset force-pushed the sidecar-submitter-not-waiting-fix branch from 30bfb39 to a995d8b Compare January 28, 2026 23:07

Wait for workers before submitted jobs in sidecar mode

05c0fec

Signed-off-by: Mark Rossett <marosset@microsoft.com>

marosset force-pushed the sidecar-submitter-not-waiting-fix branch from a995d8b to 05c0fec Compare January 28, 2026 23:14

Future-Outlier previously requested changes Jan 29, 2026

View reviewed changes

CheyuWu self-requested a review January 29, 2026 11:47

win5923 approved these changes Jan 29, 2026

View reviewed changes

andrewsykim reviewed Jan 30, 2026

View reviewed changes

Future-Outlier requested changes Feb 2, 2026

View reviewed changes

Future-Outlier moved this to others in @Future-Outlier's kuberay project Feb 2, 2026

Future-Outlier added this to @Future-Outlier's kuberay project Feb 2, 2026

marosset added 2 commits February 3, 2026 15:52

Use python/urllib to query for worker count instead of using wget

7266355

Signed-off-by: Mark Rossett <marosset@microsoft.com>

use utils.CalculateDesiredReplicas when determining when to start sid…

8504c33

…ecar sumitter jobs Signed-off-by: Mark Rossett <marosset@microsoft.com>

Copilot AI mentioned this pull request Feb 5, 2026

Review all open pull requests #4482

Closed

4 tasks

	// CalculateDesiredReplicas calculate desired worker replicas at the cluster level
	func CalculateDesiredReplicas(ctx context.Context, cluster *rayv1.RayCluster) int32 {
	count := int32(0)
	for _, nodeGroup := range cluster.Spec.WorkerGroupSpecs {
	count += GetWorkerGroupDesiredReplicas(ctx, nodeGroup)
	}

	return count
	}

Conversation

marosset commented Jan 21, 2026

Why are these changes needed?

Related issue number

Checks

Uh oh!

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

marosset commented Jan 27, 2026

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

win5923 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Future-Outlier left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

win5923 left a comment

Choose a reason for hiding this comment

Uh oh!

andrewsykim Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

marosset Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

marosset Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

marosset commented Feb 2, 2026

Uh oh!

Future-Outlier commented Feb 3, 2026

Uh oh!

marosset commented Feb 3, 2026

Uh oh!

Future-Outlier commented Feb 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Future-Outlier left a comment •

edited

Loading

win5923 left a comment •

edited

Loading

Future-Outlier left a comment •

edited

Loading