Workers should continually extend their redelivery timeout leases for in-flight tasks

[chrisguidry]

Not entirely clear if this is supported with Redis directly, but ideally a task that is still healthy and alive should be able to periodically extend its claim, which would allow folks to use the a lower redelivery timeout for more timely processing, but still have good reliability.

[chrisguidry]

Research notes on lease extension

Did some exploration on how Redis streams can support extending message leases. Here's what I found:

The key insight: XCLAIM with min-idle-time=0

Redis's XCLAIM command can be used to reset the idle timer on a message, effectively extending its lease:

If IDLE is not specified, an IDLE of 0 is assumed, that is, the time count is reset because the message has now a new owner trying to process it.

So calling XCLAIM stream group consumer 0 message_id (with min-idle-time=0) allows a worker to "re-claim" its own message, resetting the idle timer and preventing redelivery.

Proposed approach

1. Add a _lease_extender_loop() background task in the Worker (similar to how _scheduler_loop works)
2. This loop periodically iterates through active_tasks (which already maps task -> message_id) and calls XCLAIM
3. Extension interval would be redelivery_timeout / 2 for a 50% safety margin
4. With automatic lease extension, we can remove the hard cap that limits user Timeout to redelivery_timeout

Partial implementation exists

Started implementing this on branch extend-lease-211. The basic structure works:

• Added _lease_extender_loop() method
• Spawns alongside scheduler task in _worker_loop()
• New tests in tests/test_lease_extension.py (6 tests passing)

However, there's a behavioral change that breaks some existing tests: removing the automatic redelivery_timeout cap on task execution means tasks without an explicit Timeout dependency can now run indefinitely. Some tests relied on the old behavior where tasks would be forcefully timed out.

Next steps

• Decide if tasks without Timeout should truly run indefinitely (current direction) or have some fallback
• Fix the failing timeout-related tests in test_fundamentals.py
• Run full test suite to verify no regressions

References

• XCLAIM docs
• XAUTOCLAIM docs (what we currently use for redelivery)
• XPENDING docs (can query idle time if needed)