Nats: guarantee order of message reception

[slawo]

Currently there is no guarantee messages are received in order when using nats streaming.

This can be easily fixed by checking the msg.Sequence.

2 cases can be implemented:

• SequenceGuarantee we receive the right sequence of events. 4,5,6,7,8...
• OrderGuarantee: we receive the messages in order 4,5,8,9,30,31,34... which will work when group subscriptions are expected to be used.

[slawo]

One proposed solution is to create a flag Ordering which will be either Sequential, Ordered or None

[mjgarton]

Why would we want this to be configurable? substrate, at the api level, guarantees order, and should do whatever it needs to make sure the underlying broker does that, and to error of not. We should not be exposing this choice to the consumer. If they don't want ordering, don't use substrate.

[slawo]

Because there are valid cases when using groups with multiple consumers where the sequential test will fail.

[slawo]

So maybe we can get rid of None, but we can keep Ordered and Sequential (default)

[mjgarton]

Why can't we expose no configuration, but always unconditionally check that the sequence is increasing and error if it goes down?

That works when you have only one consumer.
But when you have many consumers each consumer will get messages only ordered ( 1,3,6,10,...) but never sequential ( 1.2.3.4,....).

[mjgarton]

@gedw99 It's not clear to me which comment you were replying to.

@slawo @gedw99
If you request "sequential" and have multiple consumer instances running and using the same queue-group, what would you expect to happen in each consumer?

[mwain]

I didn't think Nats Streaming itself guaranteed message order?
Due to the way the server will attempt redelivery of messages it hasn't received an ack for?

[mjgarton]

Yes, it's unfortunate, but nats-streaming "tries" to deliver in order, but there are cases where it does not.

We can't really fix that in substrate, but we might possibly detect it and protect the consumer from it by erroring or something. It's not clear whether it's worth the complexity.