Subs randomly failing when broker restarted (ArgumentNullException: Value cannot be null. (Parameter 'source'))

[bvandewe]

I tried this:

When restarting a broker with multiple subs (usually bc a sub is stuck or failed), then it happens consistently that one or more subs are failing while the broker restarts.

While the failing sub(s) change apparently randomly between subsequent restarts, they always throw logs as follows:

fail: CloudStreams.Broker.Application.Services.SubscriptionHandler[0]
      An error occurred while streaming cloud events for subscription 'mosaic-adapter_content-streams_mosaic.formset.copied.v1': System.ArgumentNullException: Value cannot be null. (Parameter 'source')
         at System.Linq.AsyncEnumerable.ToAsyncEnumerable[TSource](IObservable`1 source) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToAsyncEnumerable.Observable.cs:line 24
         at CloudStreams.Broker.Application.Services.SubscriptionHandler.InitializeCloudEventStreamAsync() in /src/src/broker/CloudStreams.Broker.Application/Services/SubscriptionHandler.cs:line 257

This is extremely impacting as restart a broker almost always results in one or more subs failing. Not sure how to recover, except trying to restart many times, hoping for one without a failing sub...

The same logs were reported in #105 but i doubt this is the same problem.

This happened:

Subscriptions are marked failed and can only be recovered by restarting the broker, eventually.

I expected this:

No response

Is there a workaround?

No response

Anything else?

No response

Component(s)

No response

Platform(s)

No response

Community Notes

• Please vote by adding a 👍 reaction to the issue to help us prioritize.
• If you are interested to work on this issue, please leave a comment.name: Bug Report 🐞