-
Notifications
You must be signed in to change notification settings - Fork 3.6k
[fix][ml] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum #24054
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #24054 +/- ##
============================================
+ Coverage 73.57% 74.22% +0.65%
+ Complexity 32624 32018 -606
============================================
Files 1877 1861 -16
Lines 139502 144196 +4694
Branches 15299 16425 +1126
============================================
+ Hits 102638 107033 +4395
+ Misses 28908 28718 -190
- Partials 7956 8445 +489
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
lhotari
approved these changes
Mar 14, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
nikhil-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Mar 21, 2025
…ded storage by the write quorum (apache#24054) (cherry picked from commit 1309cba) (cherry picked from commit c028eaa)
mukesh-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Mar 21, 2025
…ded storage by the write quorum (apache#24054) (cherry picked from commit 1309cba) (cherry picked from commit c028eaa)
nodece
pushed a commit
to nodece/pulsar
that referenced
this pull request
Mar 27, 2025
…ded storage by the write quorum (apache#24054) (cherry picked from commit 1309cba)
nikhil-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Mar 28, 2025
…ded storage by the write quorum (apache#24054) (cherry picked from commit 1309cba) (cherry picked from commit 729a916)
nikhil-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Mar 29, 2025
…ded storage by the write quorum (apache#24054) (cherry picked from commit 1309cba) (cherry picked from commit 729a916)
srinath-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Mar 31, 2025
…ded storage by the write quorum (apache#24054) (cherry picked from commit 1309cba) (cherry picked from commit 729a916)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
cherry-picked/branch-3.0
cherry-picked/branch-3.3
cherry-picked/branch-4.0
doc-not-needed
Your PR changes do not impact docs
ready-to-test
release/3.0.11
release/3.3.6
release/4.0.4
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
I think pulsar_storage_size is partially double-counted. Currently
pulsar_storage_size = pulsar_storage_logical_size * number of replicas
Number of replicas is the same as the bookie write quorum.
pulsar_logical_size doesn't account for the replication, it is sum of customer bytes no matter where they stored at.
and it looks like pulsar_storage_logical_size (calculated here initially) DOES include pulsar_storage_offloaded_size (calculated here).
But the offloaded ledgers (e.g. S3) do not create multiple replicas.
I think the metric should be calculated as:
pulsar_storage_size = pulsar_storage_offloaded_size + (pulsar_storage_logical_size - pulsar_storage_offloaded_size) * number of replicas
Modifications
adjusted calculations of
pulsar_storage_size
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes
Documentation
doc
doc-required
doc-not-needed
doc-complete
Matching PR in forked repository
PR in forked repository: dlg99#22