JetStream consumers not processing messages

[irinafratila]

Observed behavior

Hi all,
We are observing an inconsistent issue while migrating from core NATS pub/sub to JetStream pub/sub.
In a 3-node NATS cluster, some JetStream consumers are created successfully but never process messages. These consumers appear active, no JS client errors are reported, but in their state Last Delivery: never and Unprocessed messages accumulate.

Expected behavior

All consumers pull messages from the stream 100% of the time they are brought up.

Server and client version

v0.3.1

Host environment

No response

Steps to reproduce

This behavior is reproducible when scaling NATS dynamically, but not 100% of the time. Reproducibility increases with higher numbers of publishers and consumers. In our testing, the same workload on a statically configured 3-node cluster does not show this behavior.
Please see this repository for full reproduction steps, logs, and sample code: https://github.com/irinafratila/nats-jetstream

[piotrpio]

Hello @irinafratila, thanks for reporting the issue and for the repository with reproduction, that's very helpful! I'll be lloking at this and will get back to you soon.

[piotrpio]

Ok, here with an update:

Firstly, I was able to easily reproduce the issue, thanks again for the linked repo, insanely useful.

So it turns out this is not a go client issue but rather a nats-server issue surrounding building clusters with DNS records resolving to multiple IPs and using those as routes. Sometimes (and this is random, that's why you're not able to reproduce it 100% of the time), the formed cluster does not have the whole connection pool created for each route. Connection pooling for clusters was introduced as part of v2 routes back in v2.10. By default, each route forms 3 connections (this can be configured with cluster -> pool_size property in config). When this bug occurs, the pool-filling chain re-resolves DNS on each step, which can cause it to connect to a different server than the one it was filling slots for, leaving permanent gaps in the connection pool. The server is not aware the server pool is incomplete - it still can assign specific subscriptions to specific connections in the pool, even when they're not created. This in turn causes consumers to be permanently stuck.

To confirm this, you can try setting cluster: { pool_size: -1 } in config, which disables connection pooling. This is a workaround, not a fix - connection pooling with dynamic DNS setup such as yours should work. I'll create an issue in nats-server so that it can be fixed in a future release.

As a side note - when using Consume(), I would suggest to always use jetstream.ConsumeErrHandler() option to make Consume() report errors such as missing heartbeats, server unavailable etc:

		_, err = cons.Consume(func(msg jetstream.Msg) {
			// handle msg
		},
			jetstream.ConsumeErrHandler(func(consumeCtx jetstream.ConsumeContext, err error) {
				fmt.Println("Consume error:", err)
			}),
		)

[irinafratila]

Thanks @piotrpio for the detailed explanation! The workaround seems to have done the trick. I'll keep an eye on nats-server for the next release