Cooperative scaling for Kafka #7092
Description
Proposal
Use the throughput of the messages i.e. the rate of consumption as a meta feedback loop that can reverse the scaling out of the consumers e.g. if the lag increases and as a result we scale up the consumers, but in fact it leads into a reduce throughput, we pause or reverse the scale up.
The idea is inspired by PID controllers in classical engineering and I've called it cooperative scaling, because it's main purpose is to manage back pressure on downstream services.
Use-Case
The problem I've seen in practice is that it's often simple to scale up the consumers, but it often actually reduces the throughput because it causes more pressure on downstream services e.g. databases which do not scale as quickly or as much as the consumers. True story, once we brought down the entire IBM Event Streams region (long story why IBM lol) because we maxed out our consumers.
Is this a feature you are interested in implementing yourself?
No
Anything else?
No response