backlogCount cannot tell a retry backoff from a scheduled delay

[georgekaran]

Context

BullMqAuditQueue.backlogCount is the number POST /api/audits refuses submissions on — the aggregate backstop the per-IP buckets cannot provide, since those bound one source each while the queue is shared by all of them.

It counts waiting only. It counted waiting and delayed until #13, on a premise its own comment stated: nothing enqueues an audit with a delay of its own, so a delayed job can only be one inside its retry backoff. The nightly re-audit fan-out breaks that deliberately — it hands the queue a night's work with delays of up to six hours. Counting that would have made the submission endpoint answer 503 for the length of the window, every night, against idle workers.

The fix traded one imprecision for another, and this issue is the other one. Raised by review on #49.

The residual

BullMQ puts a job that threw and a job deliberately scheduled into the same delayed set, and nothing in the count distinguishes them. So while a job is inside its retry backoff the depth under-reads, and the admission check sees room it did not have a moment earlier.

The worst case is a handler that fails fast. With the worker churning through jobs that throw immediately, a large fraction of the queue can be in backoff at once, the depth reads near zero, and another full AUDIT_QUEUE_MAX_DEPTH of submissions is admitted on top.

Bounded, but real: the overshoot is roughly attempts × cap rather than unbounded, because a job can only re-enter backoff a fixed number of times before it fails for good, and every retried job re-enters waiting where it is counted again. The queue still spikes and drains rather than growing without limit — which is the property the cap exists for — it just drains from a higher peak than intended, during an outage.

Pinned by a spec (bullmq-job-queue.spec.ts, "does not count a job inside its retry backoff") so the limitation is understood rather than assumed away, in the same style as DbRequestAudit.queueIsSaturated's check-then-act spec.

What would actually fix it

The count cannot separate the two while both kinds of delay live on the same queue. Two directions, and the second is the one I would take:

Track scheduled work separately. Keep the delay where it is and maintain a count of scheduled-but-not-yet-runnable jobs to subtract. Cheap to describe, unpleasant in practice: the counter has to survive restarts, expire correctly when a job is promoted, and stay right when a scheduled job is retried — a second source of truth about the same set.

Keep the audit queue holding only runnable work. Move the jitter off it entirely: the fan-out enqueues one placement job per page on the reaudit queue, delayed by that page's offset; when it fires, it creates the audit row and enqueues with no delay. Then delayed on the audit queue means retry backoff and nothing else, the original premise is true again, and backlogCount can go back to counting both with no ambiguity at all.

The second also improves something unrelated: today the audit row is created at fan-out time and can sit queued for six hours before its job is even eligible, which the dashboard renders as in progress. Creating it at placement time makes "queued" mean "about to run".

It is not free. It adds a second job type and a worker that branches on job name, and it moves the day-scoped dedupe insert to placement time, so the placement job needs its own idempotency key (reaudit-<pageId>-<day>) on top of the unique index that already backstops it. That is why it is an issue rather than a patch on #49 — it wants measurements and its own tests, not a hurried change to a shipping branch.

Worth doing first

Establish whether it matters at the current scale before rebuilding anything. The overshoot needs a handler that fails fast and a full queue; neither is observed today. Cheapest evidence: log the depth alongside each refusal, and count how often a refusal or an admission happens while jobs are in backoff.

Scope

In

• Decide between the two directions above, with a measurement rather than a guess.
• Implement it so that backlogCount counts everything owed a worker slot, and nothing else.
• Keep the spec that currently pins the residual, inverted to assert the new behaviour.

Out

• Reverting to counting delayed on the audit queue while the scheduler still delays there. That is the 503-every-night bug Daily re-audit scheduler with per-domain jitter #13 fixed.
• Any change to AUDIT_QUEUE_MAX_DEPTH itself. The number is not what is wrong.

Acceptance criteria

• A job in retry backoff is counted as backlog.
• A job scheduled hours into the future is not.
• Both are covered by specs against real BullMQ, not a fake that agrees with whichever method is called.
• DECISIONS.md records which direction was taken and what the measurement showed.