What's wrong
If Artifex runs into an error while executing a job, there isn't currently any way to recover the interrupted job. The user must then start from scratch by creating a new job.
What needs to be done
Users must have the possibility to resume a job that errored out. To do this, implement the following:
- Upon job creation, create a
running_job.txt file.
- Keep track of the job's current stage (generating training data, creating training dataset, training model...) on the
running_job.txt.
- Implement a
resume_job(job_id: str, stage: str) method which resumes the job with job id = job_id from the provided stage.
What's wrong
If Artifex runs into an error while executing a job, there isn't currently any way to recover the interrupted job. The user must then start from scratch by creating a new job.
What needs to be done
Users must have the possibility to resume a job that errored out. To do this, implement the following:
running_job.txtfile.running_job.txt.resume_job(job_id: str, stage: str)method which resumes the job with job id =job_idfrom the providedstage.