Skip to content

[fix][ml] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum #24054

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 14, 2025

Conversation

dlg99
Copy link
Contributor

@dlg99 dlg99 commented Mar 4, 2025

Motivation

I think pulsar_storage_size is partially double-counted. Currently
pulsar_storage_size = pulsar_storage_logical_size * number of replicas

Number of replicas is the same as the bookie write quorum.
pulsar_logical_size doesn't account for the replication, it is sum of customer bytes no matter where they stored at.
and it looks like pulsar_storage_logical_size (calculated here initially) DOES include pulsar_storage_offloaded_size (calculated here).
But the offloaded ledgers (e.g. S3) do not create multiple replicas.

I think the metric should be calculated as:
pulsar_storage_size = pulsar_storage_offloaded_size + (pulsar_storage_logical_size - pulsar_storage_offloaded_size) * number of replicas

Modifications

adjusted calculations of pulsar_storage_size

Verifying this change

  • Make sure that the change passes the CI checks.

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: dlg99#22

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Mar 4, 2025
@dlg99 dlg99 requested review from lhotari and eolivelli March 4, 2025 17:03
@codecov-commenter
Copy link

codecov-commenter commented Mar 4, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.22%. Comparing base (bbc6224) to head (09b589e).
Report is 949 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #24054      +/-   ##
============================================
+ Coverage     73.57%   74.22%   +0.65%     
+ Complexity    32624    32018     -606     
============================================
  Files          1877     1861      -16     
  Lines        139502   144196    +4694     
  Branches      15299    16425    +1126     
============================================
+ Hits         102638   107033    +4395     
+ Misses        28908    28718     -190     
- Partials       7956     8445     +489     
Flag Coverage Δ
inttests 26.72% <100.00%> (+2.13%) ⬆️
systests 23.08% <0.00%> (-1.24%) ⬇️
unittests 73.74% <100.00%> (+0.89%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...ookkeeper/mledger/impl/ManagedLedgerMBeanImpl.java 89.86% <100.00%> (+0.15%) ⬆️

... and 1058 files with indirect coverage changes

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lhotari lhotari added this to the 4.1.0 milestone Mar 14, 2025
@lhotari lhotari changed the title [fix] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum [fix][ml] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum Mar 14, 2025
@lhotari lhotari merged commit 1309cba into apache:master Mar 14, 2025
57 of 58 checks passed
lhotari pushed a commit that referenced this pull request Mar 17, 2025
…ded storage by the write quorum (#24054)

(cherry picked from commit 1309cba)
lhotari pushed a commit that referenced this pull request Mar 17, 2025
…ded storage by the write quorum (#24054)

(cherry picked from commit 1309cba)
lhotari pushed a commit that referenced this pull request Mar 17, 2025
…ded storage by the write quorum (#24054)

(cherry picked from commit 1309cba)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 21, 2025
…ded storage by the write quorum (apache#24054)

(cherry picked from commit 1309cba)
(cherry picked from commit c028eaa)
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 21, 2025
…ded storage by the write quorum (apache#24054)

(cherry picked from commit 1309cba)
(cherry picked from commit c028eaa)
nodece pushed a commit to nodece/pulsar that referenced this pull request Mar 27, 2025
…ded storage by the write quorum (apache#24054)

(cherry picked from commit 1309cba)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 28, 2025
…ded storage by the write quorum (apache#24054)

(cherry picked from commit 1309cba)
(cherry picked from commit 729a916)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 29, 2025
…ded storage by the write quorum (apache#24054)

(cherry picked from commit 1309cba)
(cherry picked from commit 729a916)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 31, 2025
…ded storage by the write quorum (apache#24054)

(cherry picked from commit 1309cba)
(cherry picked from commit 729a916)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants