fix: do not linger unless writing to disk - #106
Conversation
| case Bytes < MaxLingerBytes of | ||
| is_linger_continue(#{config := #{max_linger_ms := 0}}) -> | ||
| false; | ||
| is_linger_continue(#{calls := Calls, config := Config, replayq := Q}) -> |
There was a problem hiding this comment.
@zmstone thanks for implementing this, I think it will help us to get faster PUBACKs when using the EMQX Cloud Kafka integration with hybrid buffer mode and max_linger_ms set to a non-0 value.
I see that is_writing_to_disk will return true when the tail segments are written to disk in offload mode. I am afraid that this will cause PUBACK latency to increase due to linger before append when Kafka is unavailable. Would it be possible to also skip linger before append when replayq offloads to disk? I think should not be necessary to linger when offload mode writes to disk: append should still write to memory, but tail elements are written to disk, which should allow batching disk writes.
An additional question: we need our messages to Kafka to be batched to avoid increasing load on our Kafka cluster. Will all messages that are queued during a timeframe of max_linger_ms still be batched into a single message at the popping end of the queue, or does this change cause the producer to send messages to Kafka more frequently? The desired behavior for us is that in hybrid mode the producer still lingers, but after enqueueing the messages, decoupling PUBACK latency from kafka message batching.
There was a problem hiding this comment.
For EMQX, PUBACK latency increase to cause back-pressure towards MQTT client, otherwise the memory part will eventually overflow.
For Kafka load:
There are other ways to throttle requests towards Kafka.
The most effective is to lower the send-ahead counter.
EMQX has this config named "Max Inflight" from the UI and max_inflight in config file.
The default value is 10, if you lower to 1, there is only going to be 1 request sent to Kafka before Kafka acknowledges. Hence the pending ones will effectively linger (and form a larger batch).
The unconditional linger will add unnecessary latency even when Kafka and Kafka clients are both idling.
It's like the 40m delay of TCP stack without NO_DELAY flag.
3 participants
No description provided.