Skip to content

fix: reserve min buffer for high mem load control - #116

Merged
zmstone merged 1 commit into
kafka4beam:mainfrom
zmstone:260519-min-buffer-memory
May 19, 2026
Merged

fix: reserve min buffer for high mem load control#116
zmstone merged 1 commit into
kafka4beam:mainfrom
zmstone:260519-min-buffer-memory

Conversation

@zmstone

@zmstone zmstone commented May 19, 2026

Copy link
Copy Markdown
Contributor

Reserve a minimum producer buffer under high memory pressure.
When drop_if_highmem is enabled and the system reports high memory usage,
the producer used to drop every newly-arrived call's bytes regardless of how
little was already buffered, which could leave the sender with nothing to
push to Kafka. The producer now keeps at least
(max_send_ahead + 1) * max_batch_bytes bytes buffered (enough to fill
every in-flight slot), and only drops the bytes that exceed this reserve.

@zmstone
zmstone merged commit 4bab2e0 into kafka4beam:main May 19, 2026
4 checks passed
@zmstone
zmstone deleted the 260519-min-buffer-memory branch May 19, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants