Dead Letter Queue for Failed Asynchronous Message Processing

[JamesEjembi]

Problem Statement

Messages that fail processing in the event bus are silently dropped with no retry or inspection path. Implement a dead letter queue (DLQ) that captures failed messages with full error context, supports configurable retry with exponential backoff, and provides management API for inspection, replay, and purging.

Technical Bounds

• Max retries: 3 (exponential backoff: 30s, 2min, 10min).
• DLQ storage: PostgreSQL table with TTL of 14 days; partitioned by day.
• Error context: original message body, error type, stack trace, retry count, failed_at, consumer_id.
• Management API: GET /dlq (list, filter), POST /dlq/{id}/retry, POST /dlq/purge.
• Alert: PagerDuty when DLQ depth > 500 for > 5 minutes.

Steps

1. Add retry logic to message consumer: on processing error, increment retry_count, re-enqueue with backoff delay.
2. On 3rd failure, move message from queue to dead_letter_queue table with full error context.
3. Implement management API endpoints for inspection and retry.
4. Add scheduled job: daily purge of DLQ entries older than 14 days.
5. Add monitoring: dlq_depth gauge, dlq_enqueue_total counter.
6. Integration test: submit 10 messages to a consumer that always fails, verify all land in DLQ after 3 retries.

[guptakumarranjeet150]

Hello @maintainers,

I would love to take this on. My planned approach:

• Trace the regression in VeriNode-Labs/VeriNode-Backend causing Dead Letter Queue for Failed Asynchronous Message Processing and implement a targeted, minimal fix.
• Add a reproduction test case to prevent any future regression.

Excited to work on this—please assign to me and I'll submit a clean PR shortly!