Tracking of fourfit job and other task success #26
Description
Current tracking of failed fourfit jobs, and pipeline tasks in general is poor. Consolidated log files will record logged errors, but job completion and exit status are not explicitly tracked. This can result in silent failures which are not noticed until much later.
- fourfit is non-standard in return code, where non-zero return codes carry specific information
- job failures (primarily fourfit) due to OOM may not leave a logged trail, and some other completeness check should be done
- the pipeline structure is not amenable to recovery mid-cycle due to the mixing of inputs and outputs