Kafka 4.2.0 Queues

[aleksandar78]

Describe the feature

Hi,
vertx-kafka-client has been upgraded to kafka-client 4.2.
The underlying library already ships with KafkaShareConsumer, the Share Groups consumer introduced in KIP-932 and flaged as production-ready in Kafka 4.2.

I was wondering if there are any plans to expose this in the Vert.x API, similarly to how KafkaConsumer is currently wrapped.

Share Groups are a big deal for use cases like job queues and multi-tenant/multi-app workloads because they finally give Kafka proper per-record acknowledge semantics (ACCEPT / RELEASE / REJECT) and let multiple consumers share the same partition cooperatively, without the partition-per-consumer constraint of classic consumer groups.

I don't know how difficult it is to wrap a new type of consumer in vert.x event loop with back-pressure etc, but I'm really interested to avoid using worker/virtual thread verticles with official apache kafka blocking client in vert.x application.

A few questions to understand where things stands:

• Is this already on the roadmap for 5.1 or a later milestone?
• Any known blockers or design concerns with integrating Share Groups into the Vert.x threading model or current vertx-kafka-library?

Thanks for the great work on the Kafka 4 upgrade.

Contribution

No response

[ppatierno]

While this could be a very useful addition, we don't have a plan for it (so no ETA). Are you willing for a contribution?

[aleksandar78]

Hi @ppatierno,

I'm not deeply familiar with the vertx-kafka-client internals or how the current KafkaConsumer wrapper is structured, so I'd need some guidance before jumping in.

I'm actively using Vert.x 5 / Kafka 4.2 in a multi-tenant project, and Share Groups are a real need for us, so I'm motivated to make this work.

A few things I'd like to figure out:

• What is the right channel where we could discuss the architectural pattern to follow?
• Does the actual implementation satisfy all standards in a way that I can consider it as a reference?
• One more thing worth flagging is a naming convention, as the term Shared in the library could have a double interpretation (SharedProducer - instance shared across vertices, SharedConsumer - new Kafka Share Group protocol)

Thanks

[ppatierno]

What is the right channel where we could discuss the architectural pattern to follow?

Does the actual implementation satisfy all standards in a way that I can consider it as a reference?

To answer both questions, I think you could start implementing it by following the patterns we have right now and then we can discuss on the PR.

One more thing worth flagging is a naming convention, as the term Shared in the library could have a double interpretation (SharedProducer - instance shared across vertices, SharedConsumer - new Kafka Share Group protocol)

The current consumer class is KafkaConsumer so I would assume a KafkaSharedConsumer should be good.