How to make queue evenly distribute the work to workers ?

[geminigeek]

hi,

i have few workers/consumers for a queue , all have qos of 10 or 15, i can see the consumer which starts first always gets the jobs , if jobs are less, can anyone suggest how to evenly use the workers in this scenario.

one idea is to make one queue per worker/consumer and a master queue with just one router consumer,

the router consumer starts with 30 qos, checks the count of pending jobs in all queues of workers and wherever the count is less add the job to that consumer queue, but it seems complex design, anyone got a solution for this issue

[spuun]

This is a feature/bug in LavinMQ. All messages may end up in one consumer if it is fast enough to process all messages as they are delivered. It's designed this way for performance reasons. One could argue that you have unnecessary many workers and could scale down the number of consumers.

However, this is not an expected behavior for most people so this is something we'll do something about.

[geminigeek]

As mentioned here from docs If all of the consumers have equal priorities (see below for a discussion on consumer priority), they will be picked to receive messages on a round-robin basis.

It's seems logic is already available to implement this, Why can't we have round robin thing for all workers behind a flag?

It's just an idea, I have a use case for this, many users might not , still just an idea

[carlhoerberg]

You probably want to lower the prefetch a lot for your workers, set it to 1 or 2.

[geminigeek]

hi i want to process lots of request, if i set to prefetch 1/2 its just 1/2 jobs will be processed concurrently while the hardware the worker runs on can do 15 easily. i want to run one worker per physical hardware/location

use case

• high concurrency / parallel execution as much as hardware can take
• round robin using all deployed infrastructure over multiple Datacenter simultaneously , even if jobs in queue are less than prefetch count

i am writing my own single router consumer to handle this , it seems its not possible currently, i also need retry management , if job is retrying (retry >1) send to the worker/consumer which has not processed last failed tries

requirements: something like nginx load balancer kind of setup for distribution of jobs, as nginx can be pinned to send requests based on user cookie etc, we can use retry for that but in reverse