Subscriber Outages

[mmmmmrob]

Hi All, loving mubsub, so thanks :)

I couldn't find any discussion on what to do in the event of a subscriber outage.

A typical scenario might be a website taking signups, where a mubsub subscriber listens for new signups and sends a confirmation email. If this confirmation email subscriber suffers some downtime, there will be signups that published a new signup message that the email subscriber won't have received.

How do others handle this scenario?

[scttnlsn]

In the email example it sounds like you may want to use a job queue instead (so that messages are only processed by a single consumer). Perhaps https://github.com/scttnlsn/monq could work for your use case.

If you really do want to broadcast to multiple subscribers then Mubsub may be the right choice. Internally, when starting up a subscriber, we fetch the most recently inserted doc and start listening for messages inserted afterward. I don't see why this doc ID could not be manually passed in by the subscriber. The subscriber would just need to additionally keep track of every document it sees and persist it somewhere (so if the subscriber crashes the ID can be read on startup). This additional persistence would be outside the scope of Mubsub.

[mmmmmrob]

It would be hugely useful to have a combo of the two (some colleagues and I wrote one for Meducation called Propono https://github.com/meducation/propono — in Ruby using AWS services)

Having the facility to send one message and have it delivered to several subscribers, each of whom process the message at least once, would be awesome. A combo of monq and mubsub maybe?

[scttnlsn]

Yeah, there could always be a single Mubsub subscriber that just puts messages onto a Monq queue. Would you still need Mubsub catchup/replay in that case?

[mmmmmrob]

I definitely would want/like that — I'm looking for a module that hides the guaranteed delivery and fan-out of pubsub in one place.

[scttnlsn]

Even if Mubsub supported a catchup/replay mechanism it'd be up to the subscriber to keep track of which messages it has seen (and so wouldn't really hide this functionality). Each time a subscriber receives a message it would need to record the message ID somewhere persistent. Then, when the subscriber restarts it looks up the last received ID and subscribes to messages sent thereafter. This could be in application code or some sort of Mubsub add-on module since it would hold opinions on how/where to persist the last seen ID. If subscribing to messages after a given ID suits your use case I would be in favor of adding that feature to Mubsub.

[mmmmmrob]

The functionality I'm looking for is maybe more appropriate to monq, here's a description of the same fanout functionality in rabbitmq. Maybe this is out of scope for both mubsub and monq. I'll take a look at rabbitmq. https://www.rabbitmq.com/tutorials/tutorial-three-python.html