Improvement - schema reader replay logs#1306
Draft
muralibasani wants to merge 1 commit into
Draft
Conversation
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
badeca7 to
2fdc0ed
Compare
fb7904c to
cefd21c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
About this change - What it does
Bug fix:
set_not_ready()now resets the replay-timing state (start_time,last_check,_replay_start_logged). Without this, after a master rebalance the next "Schema replay completed in X seconds" log reported time since the original process start (potentially hours), and "Starting schema replay" never re-fired. Resets happen inside_ready_lockfor symmetry with the existing_readywrite.Refactor: replaced the dual-purpose
startup_previous_processed_offset(sentinel + rate-calc state) with a dedicated_replay_start_loggedbool. Same intent, clearer code.Log accuracy: reworded "Starting schema replay: N messages to process" → "reading up to offset N", and "Schema replay completed in X seconds (processed N messages)" → "read up to offset N". On compacted topics (which
_schemasalways is), offsets are not record counts — the old wording mis-suggested 6.9M records were processed when only ~100 lived in the topic after compaction.Removed misleading metric: dropped
recs/sfrom the DEBUG "Replay progress" line. The calculation(offset - prev_offset) / dtmeasures offset-position delta per second, which on compacted topics correlates with neither records nor bytes. This produced absurd numbers like "26M recs/s" because a single consumed record could jump the position by millions of offsets. The progress percentage already conveys the only meaningful signal.Also removes a latent divide-by-zero in the
recs/scalculation that could have triggered if_is_readyran twice within the same monotonic-clock tick (possible right afterset_not_ready).References: #xxxxx
Why this way