Allow re-queueing of "exception" jobs

[mnbbrown]

One of the very useful features of https://github.com/gocardless/que is holding "exception" jobs so they can be re-queued manually, or manually deleted. How it does this depends on the retry behaviour:

• without backoff: the status is set to "failed" and the execute_after is set to NULL (i.e. failed, never attempt to re-queue)
• with backoff: the status is not changed, but the execute_after is set to the next backoff timestamp (i.e. re-attempt)

I see a couple of options here:

1. Store the job payload in the job statistics table. This means re-queueing jobs would be a matter of moving jobs back from pgqueuer_statistics to pgqueuer to trigger a retry.
2. Adding first class support to the pgqueuer table (a new status such as "failed"). This would also mean allowing executor's to customise what happens to a job after they are executed.

Thoughts? I'll experiment in my branch..

P.S. sorry I dropped off the radar before Christmas - things got very busy.

[janbjorge]

I hope things have settled down for you.

I see the value in allowing manual intervention for failed jobs. Instead of storing job payloads in pgqueuer_statistics, I think a cleaner approach would be to re-add the job to the queue with a manual execution flag or similar mechanism. This way, failed jobs remain actionable without bloating the statistics table.

A couple of thoughts on implementation:

1. Manual Re-queuing with an Explicit Flag

   • Failed jobs could be re-added to pgqueuer table with a special flag (manual_run = true, for example) that bypasses standard retry logic.
   • This would allow users to manually trigger job execution when needed, while keeping failure handling explicit.

2. A New Executor for "Re-Queue with Delay"

   • Another option could be introducing an executor type that supports "re-queue with delay," which defaults to immediate execution (delay = 0).
   • This would offer flexibility while keeping job management intuitive.

Some open questions:

• Would manual re-queued jobs still count toward normal retry limits, or should they be treated as fresh attempts?
• Should jobs marked for manual execution override standard backoff logic?
• Would it make sense to expose re-queueing through an API/CLI command for easier interaction?

Curious to hear your thoughts! Looking forward to seeing what you experiment with in your branch. 🚀

[mnbbrown]

Just my notes:

retry behaviour

The current retry/backoff implementation hogs a worker's resources. Instead the Executor should signal to the QueueManager that it want's this job to be retried, or (my current preferred solution) any Exceptions are handled by the QueueManager - but the Executor can override it's behaviour (max_attempts, max_delay, jitter, etc) either by throwing a special type of exception, returning some JobExecutionResult dataclass, or exposing some config on a well-known attribute (Executor.retry_configuration)

maintaining backwards compatibility

The default behaviour (at least until next semver bump) can be exactly the same as now. That siad, I think a failed job should only be discarded/moved to the statistics table explicitly by the user (probably by cancelling the job?). This is to avoid any data loss/make recovery easier (payload is lost when the job is moved to stats table).

I think I'm going to start by introducing a new Exception type that can be used by Executors to trigger the retry behaviour.

[mnbbrown]

I feel like #317 is ready for a bit of a review from you @janbjorge.

I'm not super happy with using the RetryableExeception, maybe just optionally returning a data structure like JobFailed(retryable=True,reschedule_for=xyz, exception=WhateverCausedTheJobToFail) would be better? But maybe not...

[Qolum]

Are there any updates on this feature?

[slnc]

This would come in really handy for me too.

[janbjorge]

@Qolum and @slnc

Have you looked at the existing retrying mechanisms? If so, could you explain your use case a bit, and why the existing retrying mechanisms do not fit?

Br
JB

[slnc]

@janbjorge I've looked at pgqueuer's code and I've updated my code to reflect it. I didn't look into pgqueuer's retry logic and assumed incorrectly on certain behaviors and that, combined with scheduled jobs not getting auto-deleted resulted in opening my inbox and seeing 6k email errors overnight because I've email notifications logic. My +1 to this was a vote to add more visibility / flexibility to configure retry logic.

The github issues I filed earlier today are suggestions related to this.

Thanks again for publishing the library. This is for a small project where I'm trying to stick to minimal infra with scale requirements for which pgqueuer is a perfect fit.

[janbjorge]

@mnbbrown @slnc I beleave #474 closes this issue? or am i not understanding / seeing something?

[slnc]

@janbjorge Thank you, I've checked the PR and it looks like it does. I won't be able to confirm it for at least a few days, so closing the issue looks sensible if @mnbbrown can't confirm it either.

[janbjorge]

@slnc thanks for the feedback, ill keep it open until you are able to test.