Consume the replication status topic from the earliest offset - #2788
Consume the replication status topic from the earliest offset#2788delthas wants to merge 1 commit into
Conversation
Hello delthas,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 6 files with indirect coverage changes
@@ Coverage Diff @@
## development/9.4 #2788 +/- ##
===================================================
- Coverage 74.88% 74.59% -0.29%
===================================================
Files 201 201
Lines 13754 13755 +1
===================================================
- Hits 10300 10261 -39
- Misses 3444 3484 +40
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Request integration branchesWaiting for integration branch creation to be requested by the user. To request integration branches, please comment on this pull request with the following command: Alternatively, the |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
78e34fd to
068abe5
Compare
4839979 to
5ca65f9
Compare
|
Does the same gap exist in the other backbeat consumers (queue processor, lifecycle, GC, notification)? Whichever ones don't pin fromOffset have the same latent behavior on a fresh group. If out of scope here, can we please have a followup ticket for the audit? |
The replication status consumer did not set fromOffset, so auto.offset.reset fell back to librdkafka's default `latest`. A status message produced onto a partition with no committed offset while the consumer group had no member (e.g. during a rolling update) was then skipped forever once the new consumer resolved its start position: the source object's metadata stayed PENDING permanently and the failed-CRR machinery never learned about the failure. Set fromOffset to 'earliest' so such messages are processed late instead of never. The reset policy is only consulted when a partition has no committed offset, so steady state is unaffected; worst-case replay after a total offset loss is bounded by the topic retention, and UpdateReplicationStatus skips entries whose object already reached a final state. Extract the consumer options into _getConsumerOptions() for unit coverage, and add functional tests pinning the fromOffset semantics on a partition that predates the consumer group. Issue: BB-826
5ca65f9 to
7ffd0cc
Compare
|
Good question — the audit is done, results are in a table on the BB-826 ticket. Summary: no extension consumer pins |
interesting that you mention notification queue processor has the same issue, as this is one of the test that's really often flaky |
SylvainSenechal
left a comment
SylvainSenechal
left a comment
There was a problem hiding this comment.
when all related pr are merged, try to release a backbeat version so we can check if this was really the reason for the flaky tests, it should be visible quickly by doing 2/3 zenko ci run.
Imo this is probably it, as I previously saw the backbeat pods were restarting very often, so i figured some messages were dropped/mishandled during these events
4 participants
The problem
The replication status processor consumes the
backbeat-replication-statustopic and writes each replication's outcome back to the source object's metadata (COMPLETED/FAILED per site); FAILED entries also feed the failed-CRR listing so the object can be retried.Its Kafka consumer never specified where to start reading when a partition has no saved position (no committed offset). In that case the client default applies: start at the end of the log, skipping whatever is already there. That default is never noticed in normal operation, because the group commits as it consumes and always finds its saved position. But it turns one specific race into permanent data loss:
Nobody ever reads that status, and nothing re-sends it: the data processor produced it, got its delivery acknowledgment, and moved on. The object's metadata says
PENDINGforever, and since the failure never reached the failed-CRR listing, it cannot even be retried.Impact on Zenko CI
This is the second half of the top CTST Replication flake, alongside the populator freeze fixed by #2787. In run 29589236290 attempt 1: the status-processor pod was rolled at 15:37:36; the FAILED status for the test's object landed on partition 4, offset 0 — first message ever on that partition — at 15:38:36, during the member gap; the replacement pod joined at 15:38:57 and started at offset 1. The "Re-replicate objects that failed to replicate" scenario (23 of 257 runs failed over two months) then waited its full 500s for a FAILED status that no longer existed anywhere — no test timeout can save this case, which is what makes it the unrecoverable variant of the flake.
The two fixes are complementary: #2787 removes the freezes and zombie pods that cause most of the red runs — and most of the restarts that open these gaps in the first place; this PR ensures that when a restart does happen (deploys never stop), a status produced during the gap is processed late instead of never. Together they should close out the Replication cluster (49 of 257 runs). Outside CI, the same race exists on any production rolling update, with the same silent consequence.
The fix
Set
fromOffset: 'earliest'on the status consumer: with no saved position, read the partition from the beginning. Steady state is unchanged (a saved position always wins over this setting), and the oplog LogConsumer already pinsearliestfor the same reason. In the worst case — a group that lost all its saved positions, e.g. after sitting empty past the offset retention — the consumer replays what the topic still retains (bounded by topic retention, broker default 7 days), andUpdateReplicationStatusre-reads the object's current metadata and skips anything already in a final state, so a replay mostly heals stuck-PENDING objects at the cost of temporary consumer lag.The consumer options move to a
_getConsumerOptions()method so a unit test can pin the option on this component, and two functional tests pin the semantics against a real kafka: an'earliest'consumer picks up a message produced before its group existed, and a default-configured consumer provably skips it — the exact loss mode this PR removes.Issue: BB-826