Unclean exits of the worker leave tasks in limbo

[RealOrangeOne]

Currently, if a worker exits in an unclean manner, the task remains marked as "PROCESSING", and thus is never picked up for re-processing nor marked as completed / failed.

[davegaeddert]

Curious to see what solution you come up with, but in a similar project mine is to have user-configurable limit (ex. 2 hrs), and then periodically find anything that has been processing for longer than that and mark it as "LOST". Excited to see some of this make its way into Django!

[RealOrangeOne]

Having a limit, after which tasks can be picked up again seems to be the standard way forward - it's what many other queue libraries do. I'm currently looking to see if there's a way to explicitly check if the task is still running, although assuming we have "at least once" semantics, that's less important.

[mgax]

On a non-SQLite database, it would be possible to lock DBTaskResult rows, and the lock would be automatically released if the worker died. But I'm not sure if there's a clean way to get the same effect with SQLite (where a lock would, in fact, lock the entire database, thereby breaking concurrent workers).

[andrewgy8]

if a worker exits in an unclean manner

Curious what you mean by this? Are you anticipating other ways of shutting down the worker that are not covered by the shutdown method?

[RealOrangeOne]

Yes, there are plenty. If the process shuts down cleanly, it's handled. If the server suddenly drops off (eg power or network issue), or a less-clean exit (eg segfault).

[jonatron]

django-background-tasks has a max run time and checks locked_at: django-background-tasks models.py
django-tasks could do the same with started_at in .ready, or there could be a management command to change the status of tasks where started_at exceeds the max run time. If performance is important ( #15 ) , the management command would be better, because if workers are running a query to find jobs frequently, then that query should be fast.