[Dashboard] Add flush() after job_id is populated

[LeoLiao123]

Why are these changes needed?

See the issue description and debugging details in ray-project/kuberay#3508.

Manual Test

command :

kubectl ray job submit\         
  --working-dir . \
  --name my-rayjob \
  --runtime-env-json='{"excludes":[
      "ray-operator/bin",
      "ray-operator/bin/k8s",
      ".git",
      "apiserver/pkg/swagger/datafile.go"
    ]}' \
  -- python task.py | ts '[%Y-%m-%d %H:%M:%S]'

task.py :

import time
import ray

ray.init(address="auto")

@ray.remote
def f():
    for i in range(20):
        print(i)
        time.sleep(1)
    return 1

print(ray.get([f.remote()]))

Result before adding flush()
Log output is delayed until after task.py completes :

my-rayjob remains in the Waiting state until the script finishes :

Result after adding flush()
Log output appears immediately after submission :

my-rayjob transitions to Running state right after submission.

Related issue number

Closes ray-project/kuberay#3508

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[LeoLiao123]

@MortalHappiness PTAL

[MortalHappiness]

LGTM. Thanks!

[MortalHappiness]

@edoakes @jjyao Could you help to merge this PR? Thanks. The context is that kubectl ray job submit (one subcommand of kubectl Ray plugin in KubeRay repo) relies on the output of the ray job submit command. We need to flush stdout to ensure the job submission ID can be properly read.

[edoakes]

relies on the output of the ray job submit command

This is not a good thing to rely on, it's not a public API... do we have plans to fix it? You should be able to use the REST API directly instead?

[edoakes]

please leave a comment here so that it isn't inadvertently broken again in the future

and see my other feedback, we really shouldn't rely on these types of behavior

[LeoLiao123]

@edoakes I’ve added the comment. Thanks for the review!

[MortalHappiness]

relies on the output of the ray job submit command

This is not a good thing to rely on, it's not a public API... do we have plans to fix it? You should be able to use the REST API directly instead?

The kubectl Ray plugin is for interactive use in non-production environments, so I thought it was acceptable. But if you think we should not depend on it, I can open an issue in KubeRay and find some contributors to work on replacing it with the dashboard API instead.

Currently kubectl ray job submit is a wrapper for ray job submit.

[MortalHappiness]

Created issue ray-project/kuberay#3556. I'll add more description on that issue.

[edoakes]

Wrapping ray job submit is fine, but relying on stdout output as a public API is not. If this is required for functionality, would suggest removing that dependency

[MortalHappiness]

Okay I got your point. We'll still call the ray job submit CLI but not rely on its stdout. We'll use the dashboard API to get the job submission ID.