[RayJob] Discuss pre-running deadline semantics for InteractiveMode Waiting

[OneSizeFitsQuorum]

Search before asking

• I searched existing issues and PRs. The closest issue is [Feature] RayJob with Waiting status needs a ttl mechanism #4037, closed by [Feat] Rayjob add ttl before running state #4525, but the merged behavior starts timing at RayJob initialization rather than when InteractiveMode enters Waiting. I found no open issue or PR discussing this semantic gap.

KubeRay Component

ray-operator

Description

We should discuss the deadline semantics for an InteractiveMode RayJob waiting for the user to provide spec.jobId.

The requirement is behavioral, not a predetermined API shape:

After the RayCluster becomes Ready and the RayJob enters Waiting, allow a configured amount of time for spec.jobId to be provided. Resource queueing and RayCluster startup must not consume this Waiting window. If no job ID is provided before the deadline, fail the RayJob with a clear reason and use the normal terminal cleanup path.

This issue intentionally does not assume that adding a field is necessarily the right solution. We should decide whether the existing field should change, whether deadlines should be split, or whether another API shape is preferable.

Current behavior on master

As of feeaf72facab4881ba2b179248bb67315634e229 (2026-08-19):

• ray-operator/apis/ray/v1/rayjob_types.go:283-290 documents preRunningDeadlineSeconds as starting from status.startTime and covering both Initializing and Waiting.
• ray-operator/controllers/ray/rayjob_controller.go:885-917 sets status.startTime during New to Initializing.
• rayjob_controller.go:175-225 checks the same deadline in Initializing before RayCluster readiness, and lines 237-249 check it again in Waiting.
• rayjob_controller.go:1247-1261 always calculates the deadline from status.startTime.

Therefore an InteractiveMode RayJob can reach Waiting with part or all of its configured budget consumed by scheduling and cluster startup. This differs from measuring how long the user has failed to complete the InteractiveMode handoff.

API questions to discuss

Possible directions include, but are not limited to:

1. Change the existing field semantics for InteractiveMode so preRunningDeadlineSeconds starts when it enters Waiting, while retaining initialization-based behavior for other modes. This avoids another field but gives one field mode-dependent meanings.
2. Keep the current broad field and add a Waiting-only field. This is backward compatible but increases API surface and overlapping deadlines.
3. Replace or deprecate the combined field in favor of phase-specific deadlines, such as separate Initializing and Waiting deadlines. This is clearer but needs migration and versioning.
4. Redefine the existing field as Waiting-only and provide a different mechanism for Initializing. This best matches the original [Feature] RayJob with Waiting status needs a ttl mechanism #4037 request but changes the behavior added for [Feature] RayJob with Initializing status needs a ttl mechanism #4178.

Questions for maintainers:

• Are Initializing and Waiting one pre-running phase with a shared budget, or distinct phases needing independent budgets?
• Is a submission-mode-dependent start point acceptable for preRunningDeadlineSeconds?
• If deadlines are split, should the current field remain, be deprecated, or be replaced?
• How should configurations containing both broad and phase-specific deadlines behave?
• What transition timestamp should be persisted so the timer survives reconciliation and operator restarts?

Invariants regardless of API shape

• The requested Waiting budget starts when the ready RayCluster causes the RayJob to enter JobDeploymentStatusWaiting, not at initial RayJob initialization.
• It applies only while spec.jobId remains empty. If the ID is present when reconciling, use the real ID rather than fail or synthesize another one.
• On timeout, set status.jobDeploymentStatus: Failed, use a dedicated reason and clear message, and do not fabricate status.jobId or status.jobStatus.
• Reuse the existing terminal path for status.endTime, deletion rules, TTL, and shutdownAfterJobFinishes; do not add separate RayCluster deletion logic.

Use case

We create an InteractiveMode RayJob, wait for its RayCluster to become usable, submit the real business job externally, and write that submission ID to spec.jobId. If the external handoff never happens, the RayCluster should eventually be released. The user needs the full configured handoff window after readiness; variable scheduling and startup latency should not consume it.

Using an HTTPMode placeholder or monitor is not equivalent. Non-Interactive modes assign the submitted placeholder ID to status.jobId, and the controller copies that Ray job status into status.jobStatus. Those fields would describe the monitor rather than the real business job.

Related issues

• [Feature] RayJob with Waiting status needs a ttl mechanism #4037 requested a TTL for InteractiveMode Waiting and was closed by [Feat] Rayjob add ttl before running state #4525.
• [Feat] Rayjob add ttl before running state #4525 introduced preRunningDeadlineSeconds as a combined Initializing/Waiting deadline.
• [Feature] RayJob with Initializing status needs a ttl mechanism #4178 requested a deadline for RayJob/RayCluster initialization.
• [RayJob] UserMode -> InteractiveMode and check rayjob.spec.jobId instead of annotation #2446 established that InteractiveMode uses the real spec.jobId supplied by the user.

Are you willing to submit a PR?

• Yes, after the API semantics are agreed upon.

[machichima]

Hi,
The interactive mode is not a user facing feature, this is only used by kuberay CLI

[OneSizeFitsQuorum]

Thanks for the clarification. We understand that InteractiveMode was primarily introduced for the KubeRay kubectl plugin.

Our use case is similar to the plugin workflow: we need to provision the RayCluster before the actual Ray job is known, then submit the real job later and update spec.jobId.

Before InteractiveMode, we had to use a placeholder job in HTTPMode. As a result, status.jobId and status.jobStatus represented the placeholder rather than the actual business job.

Since InteractiveMode is now exposed in the RayJob API and documentation, we were wondering whether there is room to further improve its lifecycle behavior. A Waiting timeout could also help when the CLI or another submitter exits before updating spec.jobId.

We are open to discussing whether this should reuse preRunningDeadlineSeconds, adjust its semantics, or use another design.

[machichima]

The initializing step usually doesn't consume much of the budget, and slow cases (e.g. head pod failing to start) are exactly what the pre-running timeout is meant to catch.

You mentioned that you will "wait for its RayCluster to become usable" then submit the job. Would you be able to handle the timeout on your end instead?

Basically we are trying to discourage users from using InteractiveMode directly, so we tend not to add InteractiveMode-specific changes unless truly necessary.

[OneSizeFitsQuorum]

Thanks for the explanation. I realize that my previous comments focused too much on how we use InteractiveMode and did not clearly explain the timing problem we are trying to solve.

In our environment, a RayJob may legitimately wait several hours for resource admission from a batch scheduler. We want to allow up to 24 hours for that phase. However, once the RayCluster is ready, we only want to allow 1 hour for the actual Ray job submission to complete.

There seems to be a similar handoff window in the kubectl plugin flow. The plugin creates the RayJob and reserves a submission ID before waiting for the RayCluster, then performs the actual ray job submit after the cluster becomes ready. If the user interrupts the CLI or the client disconnects while waiting, the cluster may still become ready later, but the actual Ray job will never be submitted. The exact resulting status may differ from our integration, but the underlying interrupted handoff is similar.

This makes a single timeout difficult to configure. If we set it to 1 hour, any RayJob that remains queued for more than 1 hour will fail and be cleaned up before it is admitted, even though that queueing delay is expected. We therefore have to configure a much larger value, such as 24 hours. However, if another RayJob is admitted and becomes ready within a few minutes, the same value may allow the ready cluster to remain unused for almost 24 hours when the actual submission never happens. In other words, the cleanup delay depends on how long scheduling happened to take, rather than on an intentional post-readiness limit.

Conceptually, what we need is an independent post-readiness submission window: scheduling may take up to 24 hours, but after readiness the actual Ray job should become observable within 1 hour. We are open to discussing whether this should reuse the existing field, use a separate phase deadline, or be handled in another way.