Consumer batch processing

[shaiyahleba]

Hi,

Is it possible to process a batch of records in a single consumer instead of just a single record at a time?

To achieve greater parallelism beyond the number of the topic's partition, it would be great if we could process in parallel (or even concurrently) a batch of records on the consumer side.

Thanks

[natansil]

Hi @shaiyahleba ,
I'm glad to report we are working on a batch consumer feature, and we will open-source it soon.
I will update this issue once this happens.

Natan

[natansil]

btw, Are you currently using Greyhound.
Can you share which API? ZIO/Scala Future/Java?

[shaiyahleba]

Great thanks! Any ETA?

We’re using the Java api.

[natansil]

Do you think you can contribute to the Java api of batch consumer?
It could make the feature available faster.

[shaiyahleba]

Unfortunately not at the moment.

[natansil]

I am trying to make this task a priority in upcoming quarter. I hope to update here soon on ETA

[shaiyahleba]

Great thanks

[natansil]

@shaiyahleba
The plan is to have batch consumer available by end of May.
FYI

[natansil]

@shaiyahleba batch consumer is ready! (it's found in version 0.2.0)