[Backend] — Notification Delivery Retry with Dead-Letter Queue

[Olowodarey]

Problem
notifications.service.ts sends notifications once; transient failures (email/push provider hiccups) are lost with no retry or visibility.

Requirements

• Retry failed deliveries with capped exponential backoff.
• Route permanently-failed notifications to a dead-letter store with the failure reason.
• Metric/log counters for sent/retried/dead-lettered.
• Tests: transient failure retries then succeeds; permanent failure is dead-lettered.

Acceptance Criteria

• Failed notifications are retried and, if unrecoverable, captured for inspection. Covered by tests.

Files

• backend/src/notifications/notifications.service.ts
• backend/src/notifications/email.service.ts
• backend/src/notifications/notifications.module.ts

[bilhokista]

I have opened #1792 for this issue.

A note on scope, since part of the requirement was already met: retry with capped exponential backoff already exists in EmailService.deliverEmailWithRetry (network errors, 5xx and 429 retried with base * 4^attempt plus jitter; 4xx thrown immediately). I left that policy untouched.

What was missing is what happens after it gives up. processQueue shifts the message off the in-memory queue, and on failure it logs one line and returns, so the message is gone with nothing left to inspect or replay. There were also no counters.

The PR adds:

• a dead_lettered_emails table and entity, storing the recipient, subject, both rendered bodies, the failure message, the attempt count and the original queue time, with a migration
• a reason enum distinguishing permanent (the provider gave a verdict, resending unchanged will fail again) from retries_exhausted (no verdict, worth replaying once the provider is healthy), since the operational response differs
• getDeliveryCounters() returning { sent, retried, deadLettered }, and counters logged inline with every dead-letter
• a guarantee that a failing dead-letter write cannot take down queue processing for the messages behind it

Tests cover the two cases named in the issue plus the exhausted-budget path, the failing dead-letter write, and the counters.

One open question I did not want to guess at: the table has no retention policy, so a long outage will grow it. I am glad to add a pruning job if you would like one in scope.