You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Summary: When using DaskTaskRunner, when a dask worker unexpectedly dies for any reason, all previous successfully completed tasks are re-run on other workers, even though Prefect treats them as complete. This is a problem for expensive or non-idempotent tasks.
The reason is likely that dask scheduler thinks that we need the futures' data still on the worker and if that worker dies, it schedules a recompute. To avoid this behavior, the futures either need to be dropped, or a .release() needs to be called.
Here's a flow that shows this behavior on both prefect 2 and 3:
Bug summary
Summary: When using
DaskTaskRunner
, when a dask worker unexpectedly dies for any reason, all previous successfully completed tasks are re-run on other workers, even though Prefect treats them as complete. This is a problem for expensive or non-idempotent tasks.The reason is likely that dask scheduler thinks that we need the futures' data still on the worker and if that worker dies, it schedules a recompute. To avoid this behavior, the futures either need to be dropped, or a
.release()
needs to be called.Here's a flow that shows this behavior on both prefect 2 and 3:
When running this script, note that after the worker is crashed, the tasks that were completed before are re-computed on the new worker.
For Prefect 2, this PR should help: #13536. Not sure why, but the same thing happens in Prefect 3 in my tests.
Version info (
prefect version
output)Additional context
No response
The text was updated successfully, but these errors were encountered: