How to reenqueue orphaned jobs

[victoronascimento]

Thanks once again for this library!

I wanted to know what is the best way to re-enqueue orphaned jobs. If I understand the logic properly, we have the field last_heartbeat_at. If we have jobs that are in_progress and last_heartbeat_at is bigger than heartbeat we could, theoretically, re-enqueue them, right?

Something like:

update underway.task 
   set state = 'pending' 
 where state = 'in_progress'
   -- heartbeat is 30 seconds by default so this is just a safe timeframe 
   and last_heartbeat_at < now() - '1 hour'::interval 

Are my assumptions correct? Of course I am not considering here the number of attempts and so on...

[maxcountryman]

I'm not sure it's safe to re-enqueue this way. You'd need to consider failed attempts relative to configured attempts and so on.

[victoronascimento]

Agreed. I'd add something like:

set retry_policy.max_attempts = (retry_policy).max_attempts + 1,
      state = 'pending' 

But in anyway, the logic for this detection is correct? Last heartbeat is the way to find stale jobs?

[maxcountryman]

A delayed heartbeat isn't a perfect mechanism, but yes, you can decide a threshold you think is reasonable to indicate that a job is now stale and won't ever complete.

Just bear in mind that there's no implicit guarantee that a job won't complete between this query and whatever you do next. Put differently, it's up to you to ensure that property.

This is where distributed systems start to become hard, "here be dragons" etc.