fix(kafka): exclude retention-deleted messages from lag for uncommitted offsets - #7937
Open
Nikko-Foti wants to merge 1 commit into
Open
fix(kafka): exclude retention-deleted messages from lag for uncommitted offsets#7937Nikko-Foti wants to merge 1 commit into
Nikko-Foti wants to merge 1 commit into
Conversation
…ed offsets Consumer groups with no committed offset under offsetResetPolicy: earliest were scaled on the partition's full log end offset, counting messages that retention had already deleted and that no consumer could ever read. Idle workloads were held at the partition count as a result. Lag is now measured from the log start offset, so it reflects what a consumer starting from earliest can actually fetch. The earliest offsets are fetched only for partitions that lack a committed offset, and a failed fetch falls back to the previous behavior. Signed-off-by: Nikko Foti <nfoti201@gmail.com>
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
|
Thank you for your contribution! 🙏 Please understand that we will do our best to review your PR and give you feedback as soon as possible, but please bear with us if it takes a little longer as expected. While you are waiting, make sure to:
Once the initial tests are successful, a KEDA member will ensure that the e2e tests are run. Once the e2e tests have been successfully completed, the PR may be merged at a later date. Please be patient. Learn more about our contribution guide. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kafka consumer groups with no committed offset on a partition were scaled on the partition's full log end offset when
offsetResetPolicy: earliest, counting messages retention had already deleted. Idle workloads were held at the topic's partition count as a result.Lag is now
logEndOffset - logStartOffset, i.e. what a consumer starting fromearliestcan actually fetch. Earliest offsets are requested only for partitions lacking a committed offset, so there is no extra broker traffic forlatest, forscaleToZeroOnInvalidOffset: true, or for fully committed groups. A failed request logs and falls back to the previous value rather than failing the poll.Alternative considered: returning 1 for
earliest, matching thelatestbranch. Avoids the extra request but under-reports a genuine retained backlog.Two unit cases were added for the new math and the fallback.
Checklist
make generate-scalers-schemahas been run to update any outdated generated filesFixes #7936
Relates to kedacore/keda-docs#1825