[Persistence] If persistence write fails contents of batch to be persisted will be lost

[MendelMonteiro]

In the following code when an exception is thrown the batch will still be cleared and the acks/msgs that were present will never be persisted.

I think we have two options, either leave the messages in the batch so that they are processed after the next delay or re-enqueue the entries in the batch so that they will be re-processed.

Any thoughts @ocoanet , @ltrzesniewski ?

        private void PersistAndClearBatch(List<MatcherEntry> batch)
        {
            try
            {
                PersistBatch(batch);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Unexpected error happened");
                _bus.Publish(new CustomProcessingFailed(GetType().FullName!, ex.ToString(), SystemDateTime.UtcNow));
            }
            finally
            {
                batch.Clear();
            }
        }

[ocoanet]

Well, I suspect that most of the persistence errors are transient problems related to temporary server issues (high load, GC, etc.). Also the persistence CQL statements are all idempotent because they explicitly set the timestamp to a fixed value.

So retrying failed entries seems to be both safe and rational. However, we need to make sure that invalid entries will not be retried forever.

I do not have a strong opinion on the retry strategy.

• Retrying right away instead of requeueing might be simpler. It would also make it easier to add a limit on the retry count.
• Requeueing could be a better solution if we wanted to support long duration server unavailability (retrying regularly with a small delay between retries).

I think both strategies would be an improvement over the current behavior. I would probably go for the first one as a first step.