Skip to content

[Dashboard] Add flush() after job_id is populated #52780

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

LeoLiao123
Copy link
Contributor

@LeoLiao123 LeoLiao123 commented May 5, 2025

Why are these changes needed?

See the issue description and debugging details in ray-project/kuberay#3508.

Manual Test

command :

kubectl ray job submit\         
  --working-dir . \
  --name my-rayjob \
  --runtime-env-json='{"excludes":[
      "ray-operator/bin",
      "ray-operator/bin/k8s",
      ".git",
      "apiserver/pkg/swagger/datafile.go"
    ]}' \
  -- python task.py | ts '[%Y-%m-%d %H:%M:%S]'

task.py :

import time
import ray

ray.init(address="auto")

@ray.remote
def f():
    for i in range(20):
        print(i)
        time.sleep(1)
    return 1

print(ray.get([f.remote()]))

Result before adding flush()
Log output is delayed until after task.py completes :
before

my-rayjob remains in the Waiting state until the script finishes :
image

Result after adding flush()
Log output appears immediately after submission :
after
my-rayjob transitions to Running state right after submission.
image

Related issue number

Closes ray-project/kuberay#3508

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@LeoLiao123 LeoLiao123 marked this pull request as ready for review May 5, 2025 07:32
@LeoLiao123
Copy link
Contributor Author

@MortalHappiness PTAL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug][Kubectl-Plugin] RayJob stucks at Waiting state for long-running jobs
1 participant