Skip to content

fix: apply max_linger_ms to memory-mode queues (linger before pop) - #119

Merged
zmstone merged 2 commits into
kafka4beam:main-4.1from
zmstone:260723-linger-for-memory-mode
Jul 23, 2026
Merged

fix: apply max_linger_ms to memory-mode queues (linger before pop)#119
zmstone merged 2 commits into
kafka4beam:main-4.1from
zmstone:260723-linger-for-memory-mode

Conversation

@zmstone

@zmstone zmstone commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

What

Since 4.1.1, the linger delay (max_linger_ms) was applied only before enqueue, and only when replayq is writing to disk (an IOPS optimization to batch disk writes). Memory-mode (and not-yet-offloaded) producers therefore sent each arrival immediately, resulting in many small produce requests under steady low-to-moderate load.

This PR adds the complementary linger on the pop side of the queue: when it is time to form a new produce request and the queue holds less than min(max_linger_bytes, max_batch_bytes) bytes, the producer waits up to max_linger_ms for more calls to accumulate before popping. This pairs with max_linger_bytes the same way Kafka's own linger.ms pairs with batch.size — no new config key.

Behavior

  • The wait ends as soon as a full batch's worth of bytes is queued, so under sustained load full batches stream with zero added latency; only under-sized (tail) batches are delayed, bounded by max_linger_ms.
  • The before-enqueue linger (disk-writing state) is unchanged; the pop-linger only applies when not writing to disk, so there is no double-linger.
  • Retry/resend paths are not affected (already-popped requests are never lingered).
  • After a reconnect the queue is flushed without lingering (it had the whole disconnected period to accumulate; this also covers the linger expiring while disconnected).
  • Default max_linger_ms = 0: behavior identical to 4.1.10.
  • Hot beam-reload safe: the new state key is accessed with maps:get/3 defaults, so the new code can be hot-loaded into a running producer with the old state.

Since 4.1.1, the linger delay was applied only before enqueue, and only
when replayq is writing to disk (to batch disk writes).  Memory-mode
producers therefore sent each arrival immediately, resulting in many
small produce requests under steady low-to-moderate load.

Now, when it is time to form a new produce request and the queue holds
less than min(max_linger_bytes, max_batch_bytes) bytes, the producer
waits up to max_linger_ms for more calls to accumulate before popping
the queue.  The wait ends as soon as a full batch worth of bytes is
queued, so under sustained load full batches are sent with no added
latency; only under-sized (tail) batches are delayed, bounded by
max_linger_ms.

The default max_linger_ms = 0 keeps the previous behavior.
Comment thread test/wolff_tests.erl Outdated
@zmstone
zmstone changed the base branch from main to main-4.1 July 23, 2026 16:48
@zmstone zmstone closed this Jul 23, 2026
@zmstone zmstone reopened this Jul 23, 2026
@zmstone

zmstone commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

close + reopen to trigger ci

@zmstone
zmstone merged commit 4379cca into kafka4beam:main-4.1 Jul 23, 2026
4 checks passed
@zmstone
zmstone deleted the 260723-linger-for-memory-mode branch July 23, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants