Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wip(store): Shard sqlite #242

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

wip(store): Shard sqlite #242

wants to merge 3 commits into from

Conversation

john-z-yang
Copy link
Member

@john-z-yang john-z-yang commented Mar 11, 2025 •
edited
Loading

main

Gnuplot not found, using plotters backend
Benchmarking bench_InflightActivationStore/get_pending_activation: Warming up for 3.0000 s
Warning: Unable to complete 256 samples in 5.0s. You may wish to increase target time to 171.0s, or reduce sample count to 10.
bench_InflightActivationStore/get_pending_activation
                        time:   [661.18 ms 662.53 ms 664.10 ms]
                        thrpt:  [6.1678 Kelem/s 6.1823 Kelem/s 6.1950 Kelem/s]
Found 4 outliers among 256 measurements (1.56%)
  2 (0.78%) low mild
  2 (0.78%) high severe
Benchmarking bench_InflightActivationStore/set_status: Warming up for 3.0000 s
Warning: Unable to complete 256 samples in 5.0s. You may wish to increase target time to 116.2s, or reduce sample count to 10.
bench_InflightActivationStore/set_status
                        time:   [445.91 ms 446.69 ms 447.47 ms]
                        thrpt:  [9.1538 Kelem/s 9.1697 Kelem/s 9.1857 Kelem/s]
Found 2 outliers among 256 measurements (0.78%)
  2 (0.78%) high mild

john/shard-store

     Running benches/store_bench.rs (target/release/deps/store_bench-9149732531649d04)
Gnuplot not found, using plotters backend
Benchmarking bench_InflightActivationStore_2_shards/get_pending_activation: Warming up for 3.0000 s
Warning: Unable to complete 256 samples in 5.0s. You may wish to increase target time to 135.1s, or reduce sample count to 10.
bench_InflightActivationStore_2_shards/get_pending_activation
                        time:   [523.55 ms 525.56 ms 528.18 ms]
                        thrpt:  [7.7549 Kelem/s 7.7935 Kelem/s 7.8235 Kelem/s]
Found 12 outliers among 256 measurements (4.69%)
  6 (2.34%) high mild
  6 (2.34%) high severe
Benchmarking bench_InflightActivationStore_2_shards/set_status: Warming up for 3.0000 s
Warning: Unable to complete 256 samples in 5.0s. You may wish to increase target time to 90.1s, or reduce sample count to 10.
bench_InflightActivationStore_2_shards/set_status
                        time:   [355.38 ms 355.98 ms 356.59 ms]
                        thrpt:  [11.487 Kelem/s 11.506 Kelem/s 11.526 Kelem/s]
Found 19 outliers among 256 measurements (7.42%)
  8 (3.12%) low severe
  2 (0.78%) low mild
  1 (0.39%) high mild
  8 (3.12%) high severe

Benchmarking bench_InflightActivationStore_4_shards/get_pending_activation: Warming up for 3.0000 s
Warning: Unable to complete 256 samples in 5.0s. You may wish to increase target time to 114.2s, or reduce sample count to 10.
bench_InflightActivationStore_4_shards/get_pending_activation
                        time:   [444.42 ms 444.90 ms 445.38 ms]
                        thrpt:  [9.1966 Kelem/s 9.2065 Kelem/s 9.2166 Kelem/s]
Found 2 outliers among 256 measurements (0.78%)
  2 (0.78%) high mild
Benchmarking bench_InflightActivationStore_4_shards/set_status: Warming up for 3.0000 s
Warning: Unable to complete 256 samples in 5.0s. You may wish to increase target time to 92.3s, or reduce sample count to 10.
bench_InflightActivationStore_4_shards/set_status
                        time:   [363.92 ms 364.67 ms 365.40 ms]
                        thrpt:  [11.210 Kelem/s 11.232 Kelem/s 11.255 Kelem/s]
Found 23 outliers among 256 measurements (8.98%)
  12 (4.69%) low severe
  1 (0.39%) low mild
  3 (1.17%) high mild
  7 (2.73%) high severe

Benchmarking bench_InflightActivationStore_8_shards/get_pending_activation: Warming up for 3.0000 s
Warning: Unable to complete 256 samples in 5.0s. You may wish to increase target time to 112.9s, or reduce sample count to 10.
bench_InflightActivationStore_8_shards/get_pending_activation
                        time:   [444.70 ms 445.38 ms 446.17 ms]
                        thrpt:  [9.1803 Kelem/s 9.1966 Kelem/s 9.2108 Kelem/s]
Found 5 outliers among 256 measurements (1.95%)
  1 (0.39%) low mild
  3 (1.17%) high mild
  1 (0.39%) high severe
Benchmarking bench_InflightActivationStore_8_shards/set_status: Warming up for 3.0000 s
Warning: Unable to complete 256 samples in 5.0s. You may wish to increase target time to 91.4s, or reduce sample count to 10.
bench_InflightActivationStore_8_shards/set_status
                        time:   [371.25 ms 373.51 ms 376.51 ms]
                        thrpt:  [10.879 Kelem/s 10.966 Kelem/s 11.033 Kelem/s]
Found 23 outliers among 256 measurements (8.98%)
  3 (1.17%) low severe
  7 (2.73%) low mild
  7 (2.73%) high mild
  6 (2.34%) high severe

Benchmarking bench_InflightActivationStore_16_shards/get_pending_activation: Warming up for 3.0000 s
Warning: Unable to complete 256 samples in 5.0s. You may wish to increase target time to 114.1s, or reduce sample count to 10.
bench_InflightActivationStore_16_shards/get_pending_activation
                        time:   [431.45 ms 438.97 ms 446.09 ms]
                        thrpt:  [9.1821 Kelem/s 9.3308 Kelem/s 9.4935 Kelem/s]
Found 45 outliers among 256 measurements (17.58%)
  37 (14.45%) low severe
  2 (0.78%) low mild
  3 (1.17%) high mild
  3 (1.17%) high severe
Benchmarking bench_InflightActivationStore_16_shards/set_status: Warming up for 3.0000 s
Warning: Unable to complete 256 samples in 5.0s. You may wish to increase target time to 93.4s, or reduce sample count to 10.
bench_InflightActivationStore_16_shards/set_status
                        time:   [383.85 ms 386.71 ms 390.08 ms]
                        thrpt:  [10.500 Kelem/s 10.592 Kelem/s 10.671 Kelem/s]
Found 27 outliers among 256 measurements (10.55%)
  2 (0.78%) low severe
  12 (4.69%) low mild
  5 (1.95%) high mild
  8 (3.12%) high severe

@codecov Codecov
Copy link

codecov bot commented Mar 11, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 95.16616% with 16 lines in your changes missing coverage. Please review.

Project coverage is 83.95%. Comparing base (79f6e42) to head (83979f2).

Files with missing lines Patch % Lines
src/store/inflight_activation.rs 94.53% 16 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #242      +/-   ##
==========================================
+ Coverage   83.35%   83.95%   +0.60%     
==========================================
  Files          19       19              
  Lines        3568     3833     +265     
==========================================
+ Hits         2974     3218     +244     
- Misses        594      615      +21

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@john-z-yang john-z-yang force-pushed the john/shard-store branch 8 times, most recently from 630794a to e2a2ca4 Compare March 12, 2025 02:42
markstory
markstory reviewed Mar 12, 2025
View reviewed changes
Copy link
Member

@markstory markstory left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good 👏

src/store/inflight_activation.rs
.into_iter()
.collect::<Result<Vec<_>, _>>()?
.into_iter()
.sum())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These combinator chains 🤩

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we actually might not need these join_all operation since most of them are for upkeep, and we could just run a single upkeep per shard.

@john-z-yang john-z-yang force-pushed the john/shard-store branch 6 times, most recently from 4273e57 to 3ae60ef Compare March 13, 2025 05:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@markstory markstory markstory left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@john-z-yang @markstory