[k8s] Better error message for stale jobs controller

[kyuds]

Fixes #4268

Noticed that task.py is the only place that actually validates resources AND we can know for sure that the task in question is indeed a jobs controller.

Got feedback and changed to updating and checking cluster status on jobs.launch

Tested (run the relevant ones):

• Code formatting: install pre-commit (auto-check on commit) or bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
• Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
• Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)

[cg505]

We could also hit this in the case where the resources are just invalid, right? Is there some way we can check that the cluster already exists?
Also would love to figure out how to move this out of task.yaml, e.g. into jobs.launch. May require a new special exception type for invalid resources, or global_user_state checks. Maybe too hard.

[kyuds]

so existing behavior will also throw an error when the resources are just invalid anyways (I run into this all the time, saying that kind doesn't have any resources that are 4+ CPU). The only difference with this particular error message is that it will ask the user if a certain kubernetes cluster exists or not.

Also would love to figure out how to move this out of task.yaml, e.g. into jobs.launch. May require a new special exception type for invalid resources, or global_user_state checks. Maybe too hard.

Edit: found a way to move things into jobs.launch, specifically using backend_utils.refresh_cluster_status_handle. Basically, I try to refresh to see if the cluster itself is alive, and if not, I will alert users appropriately.