Temporary solution for llm-d#299 (llm-d#300)

While not particularly elegant, the ability to capture the return
boolean from `wait_for_job` and ensuring `standup.sh` does not progress
until it is `True` will at least prevent the starting of `profile` and
`decode` `pods` with models still being downloaded.

Signed-off-by: maugustosilva <maugusto.silva@gmail.com>

Author: maugustosilva

setup/functions.py

@@ -461,6 +461,7 @@ async def wait_for_job(job_name, namespace, timeout=7200, dry_run: bool = False)
         return False
     except Exception as e:
         announce(f"Error occured while waiting for job {job_name} : {e}")
+        return False
     finally:
         await api_client.close()
 

setup/steps/04_ensure_model_namespace_prepared.py

@@ -127,12 +127,15 @@ def main():
                 verbose=ev["control_verbose"]
             )
 
-            asyncio.run(wait_for_job(
+            job_successful = False
+            while not job_successful :
+                job_successful= asyncio.run(wait_for_job(
                 job_name="download-model",
                 namespace=ev["vllm_common_namespace"],
                 timeout=ev["vllm_common_pvc_download_timeout"],
                 dry_run=ev["control_dry_run"]
             ))
+                time.sleep(10)
 
     if is_openshift(api) and ev["user_is_admin"] :
         # vllm workloads may need to run as a specific non-root UID , the  default SA needs anyuid