During a recent test of the Prefect workflows, a file was not available on host running the workflows due to Lustre mounting of data drives not being complete. This did not result in an Exception and a failed state in the Prefect workflow - it was shown as complete (succeeded without any errors).
This is due to incorrect handling of exceptions in linker.py - in particular, in line 43, while an error is shown to have happened, the failed state is noted but never displayed to the users as part of a failed workflow.
This lack of feedback to Prefect can be remedied by at least two different mechanisms:
- allowing exceptions to propagate upwards, so that the stack trace can be seen by Prefect
- continue using the existing "failed" mechanism and return a failed state with all of the failures in this task
The latter is preferred because allowing the exceptions to propagate will result in feedback about only the first exception seen, while the failed list enables a view of all problem files.
During a recent test of the Prefect workflows, a file was not available on host running the workflows due to Lustre mounting of data drives not being complete. This did not result in an Exception and a failed state in the Prefect workflow - it was shown as complete (succeeded without any errors).
This is due to incorrect handling of exceptions in linker.py - in particular, in line 43, while an error is shown to have happened, the failed state is noted but never displayed to the users as part of a failed workflow.
This lack of feedback to Prefect can be remedied by at least two different mechanisms:
The latter is preferred because allowing the exceptions to propagate will result in feedback about only the first exception seen, while the failed list enables a view of all problem files.