[FEATURE REQ] Allow specifying maximum batch size for EventHubBufferedProducerClient

[otto-gebb]

Library name

Azure.Messaging.EventHubs

Please describe the feature.

Motivation: I want to limit the number of messages lost in case of a crash.

If I understand correctly the inner workings of EventHubBufferedProducerClient, it stores pending messages in two places (imagine for simplicity we have only one partition and one concurrent "sender process"):

• a limited-size buffer (bounded channel)
• the current accumulated batch, which is constantly being filled by messages from the buffer.

The client flushes the current accumulated batch either

• on timeout or
• when the accumulated batch size reaches a limit.

There is a way to limit the size of the first place, i.e. the buffer via MaximumEventBufferLengthPerPartition, there is no such possibility for the second storage, i.e. the current batch.
As a result, the batch may grow undesirably large, leading to a loss of too many messages if the process crashes.

One can mitigate this by setting MaximumWaitTime to a low value to flush the batches frequently, but this method feels unreliable. (What if the producer starts pushing messages twice as fast? The in-memory batch will grow twice as big, compared to what was estimated when the setting MaximumWaitTime was chosen.)

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jsquire]

H@otto-gebb. Thank you for your suggestion. We'll take this under consideration and provide an update when a decision is made.