Fix the queue populator LogConsumer drain deadlock on rebalance - #2787
Open
delthas wants to merge 1 commit into
Open
Fix the queue populator LogConsumer drain deadlock on rebalance#2787delthas wants to merge 1 commit into
delthas wants to merge 1 commit into
Conversation
Contributor
Hello delthas,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 6 files with indirect coverage changes
@@ Coverage Diff @@
## development/9.4 #2787 +/- ##
===================================================
- Coverage 74.88% 74.62% -0.27%
===================================================
Files 201 201
Lines 13754 13753 -1
===================================================
- Hits 10300 10263 -37
- Misses 3444 3480 +36
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Contributor
Request integration branchesWaiting for integration branch creation to be requested by the user. To request integration branches, please comment on this pull request with the following command: Alternatively, the |
On revoke, _drainAndUnassign() refused to unassign while a commit was pending, waiting for a DRAINED event that only a successful commit callback could emit. librdkafka parks its timed auto-commit during a rebalance and only commits stored offsets when serving the unassign, so the commit the drain waited for could not happen until the drain released: the consumer held its partitions for maxPollIntervalMs-1000 (299s by default), freezing the whole group, then disconnected with no reconnect path, leaving a zombie pod until the liveness probe restarted it. close() funnels through the same drain, making graceful shutdowns overrun the k8s grace period. Rework the drain to mirror BackbeatConsumer's revoke path: - gate only on the in-flight batch; storeOffsets() now emits DRAINED on every path and the _pendingCommit flag is removed; - commit stored offsets explicitly (best-effort, never gating) before unassigning, with librdkafka's removal-time auto-commit as fallback; - cap the drain timeout at 30s and keep the deliberate disconnect so the healthcheck restarts a populator with a genuinely stuck batch. Add a functional suite driving a real two-consumer rebalance, asserting prompt release, broker-side committed offsets matching the stored ones, continued consumption after the rebalance, and the timeout path. Issue: BB-825
delthas
force-pushed
the
improvement/BB-825/logconsumer-drain-deadlock
branch
from
July 28, 2026 09:50
c8748c8 to
be5e136
Compare
SylvainSenechal
approved these changes
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
When a consumer group rebalances (typically because a backbeat pod restarts during a rolling update), every consumer must release its partitions before the group can resume. The queue populator's LogConsumer refused to release until its last stored offsets were committed — but librdkafka cannot complete a commit while the rebalance is waiting for the release, and it commits the stored offsets itself as part of releasing anyway. The consumer was waiting on something that can only happen after the wait ends.
So whenever a rebalance arrived within the ~5 seconds following a batch:
maxPollIntervalMs − 1000). The whole group waits for its slowest member, so the entire oplog pipeline — replication entries, bucket notifications, lifecycle — froze for ~5 minutes;Impact on Zenko CI
This was the single largest source of CTST end2end flakiness over the past two months: the two Replication scenarios (object stuck
PENDINGpast the test timeout while the populator was frozen or dead) failed 49 of 257 runs across 20 branches, and the Bucket notifications filter scenario (events delivered ~5 minutes late, after all 4 in-test retries) failed 7 runs — each red job costing a full ~4h end2end attempt. Traced examples: run 29589236290 attempt 1 (populator frozen 15:38:20→15:43:19, then dead until the liveness restart) and run 29082783060 attempt 1 (all 4 retries' notifications delivered in one batch, 44s after the last retry gave up). Merging this should remove both flake clusters.The fix
Same approach as BackbeatConsumer's rebalance handling (#2731 / #2739 / #2740):
storeOffsets()now signals batch completion directly, and the_pendingCommitflag is removed;A new functional suite (
tests/functional/lib/queuePopulator/LogConsumer.js, runs underft_test:lib) exercises a real two-consumer rebalance against kafka: partitions released within seconds of the revoke, committed offsets on the broker matching the stored ones, the consumer still alive and consuming once the group settles, and the timeout/disconnect path.Issue: BB-825