Status of tasks in hyperqueue

[pmu2022]

I have a general question. In our Python framework that manages and executes dynamic task graphs, I would like to utilize hyperqueue for submission of tasks. Is it possible to receive some callback whether a job I submitted has been completed, so that potential new jobs can be started?

[spirali]

There is wait_for_jobs in Client that blocks until the job is not finished.

E.g.:

job_id = client.submit(job)
client.wait_for_jobs([job_id])

[Kobzol]

It is currently not possible to wait for the completion of individual/specific tasks within a job though. We could add it if you have a use-case :)

[lahwaacz]

If waiting is only for the purpose of starting new jobs/tasks, it can be already achieved with dependencies.

[Kobzol]

Right, of course, you can specify dependencies with the usual semantics. It's not currently possible in the Python API to have an actual callback to react to a task being completed in the "driver" Python script.