Redis Broker - Handle stream deletion of unacknowledged messages

[epalchan]

When the Redis stream reaches max length, the oldest messages will be deleted even if they're unacknowledged.
If possible, it would be beneficial to implement a mechanism to handle the deletion of unacknowledged messages by one of the following potential methods:

• Delete the oldest acknowledged messages, trimming the stream length and allowing for the triggering and acknowledgment of new messages.
• Stop receiving new messages when the stream only contains unacknowledged messages (should be controlled by a flag to enable the functionality).
• Trigger the forwarding of the oldest message to the DLQ before deletion when the stream only contains unacknowledged messages (should be controlled by a flag to enable the functionality).

Additionally, it would be beneficial if the Prometheus metrics could include stream metrics (such as XPENDING for example) that would allow setting up alerting based directly on the broker rather than failure handling only once messages are already being dropped.

[odacremolbap]

Hi @epalchan .

This is something that we could try to do here to show some insights on how data is stored/delivered at the stream.

Ingress: We could provide metrics of the time difference between the last item being ingested from a source and the most recent event being evicted.

Egress (triggers): We could provide metrics of the time difference between the event being sent (broker time) and the time that the event was ingested (Redis time)

Then users could graph both values x time and make sure that the ingestion/deletion span is higher (maybe 2x) than the ingestion/subscription-delivery span, and setup alerts on it.

There are 2 caveats to this solution:

• performance, we might need to add 2 (maybe more) calls to ingest/egress operations.
• metrics for egress will compare datetime set at redis with datetime from the broker instance.

[epalchan]

Hi @odacremolbap ,

The ingress metrics suggested could be beneficial but could also generate false positives as the time difference may be non existent in a HA and high flow efficient environment with low response times- acknowledged event is immediately triggered and actioned, low limit is set on max stream length that is in line with expected ingress message numbers so the most recent evicted items may have a low/no time difference to ingested items but they are still sent downstream.

The egress metrics may be beneficial for showing potential Redis backend/connectivity issues but would miss the mark in alerting when stream max length is reached and ingested events are not being triggered at all.