ChanQueueSubscribe slow consumer help needed

[susana-garcia]

Hello,

We are having an issue with our nats consumers and I’m wondering if someone could help me. We monitor our NATS JetStream with prometheus/grafana and there I see:

• 12.8K pending messages and there are increasing by a 1000 per hour since a few hours.
• Messages added and processed per second per consumer vary between 0.3 and 1.

We use ChanQueueSubscribe and I saw sometimes in the logs a nats: slow consumer, messages dropped error message. But weird enough, I see new messages coming into both running apps and doing what they’re supposed to do. Plus it takes a few milliseconds to handle a new message received by the consumer.

Any idea what might be off?

We use for the consumers and this is our setup:

Thank you in advance.

[wallyqs]

What is the size of the channel? It needs to have enough capacity to handle the traffic, something like 512k cap is the default in other APIs like Subscribe.

[susana-garcia]

Uhi, that might be the problem then, the size is 1 😳
I'll update it and let you know if it helped.
Thank you very much for the support.

[wallyqs]

no problem! :) will move to discussion instead of issue

[susana-garcia]

Also, shouldn't be the pending messages seen here in Unprocessed Messages (nats consumer info) or that's something else?

[susana-garcia]

So, I've realized I didn't understand how the stream works with different retention policies and consumers.
I made a POC and here is what I found out: I tried Interest and Limits Retention Policies with push consumers and Interest Retention Policy with pull consumer.
The only one that deletes the messages after ack them is Interest Retention Policy with push consumer.

[ripienaar]

Work queue policy deletes after any 1 ack from any 1 consumer (and so overlaps in interest is not allowed)

interest policy deletes after all consumers with matching interest acked a message.

[susana-garcia]

Exactly, also I forgot to mention, but with limits, the messages get deleted when one of the limits defined is reached. Sorry, it's my bad I assumed that that happened automatically after ack a message (in the case of Limits Retention Policy).
Thanks for all.

[ripienaar]

yes, ilmits apply to all cases regardless of retention method, they work together.