Skip to content
This repository was archived by the owner on Jun 5, 2026. It is now read-only.

Parallelize row groups#843

Draft
simonswine wants to merge 3 commits into
grafana-cold-storage:mainfrom
simonswine:20230711_parallelize-row-groups
Draft

Parallelize row groups#843
simonswine wants to merge 3 commits into
grafana-cold-storage:mainfrom
simonswine:20230711_parallelize-row-groups

Conversation

@simonswine

@simonswine simonswine commented Jul 11, 2023

Copy link
Copy Markdown
Contributor

This retrieves rowGroups in parallel, I don't think this is the way to go.

We rather should look at switchting the querying order to align completely with the block order (seriesId, then timeNanos)

❯ benchstat before.txt after.txt
name                                                                                                             old time/op    new time/op    delta
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-block:contentions:count:contentions:count-12       284ms ± 6%     269ms ± 1%   -5.25%  (p=0.016 n=5+4)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-block:delay:nanoseconds:contentions:count-12       289ms ± 8%     283ms ± 7%     ~     (p=0.151 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-goroutine:goroutine:count:goroutine:count-12       145ms ±15%     129ms ± 2%  -10.77%  (p=0.016 n=5+4)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:alloc_objects:count:space:bytes-12          123ms ± 6%     107ms ± 1%  -13.13%  (p=0.016 n=5+4)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:alloc_space:bytes:space:bytes-12            116ms ± 0%     106ms ± 5%   -8.92%  (p=0.016 n=4+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:inuse_objects:count:space:bytes-12          106ms ± 4%      95ms ± 6%  -10.19%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:inuse_space:bytes:space:bytes-12            106ms ± 5%      99ms ± 7%     ~     (p=0.056 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-mutex:contentions:count:contentions:count-12       128ms ± 2%     121ms ± 3%   -5.52%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-mutex:delay:nanoseconds:contentions:count-12       836ms ± 4%     810ms ± 0%   -3.05%  (p=0.016 n=5+4)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-process_cpu:cpu:nanoseconds:cpu:nanoseconds-12     1.53s ± 1%     1.53s ± 1%     ~     (p=0.690 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-process_cpu:samples:count:cpu:nanoseconds-12       1.47s ± 1%     1.48s ± 3%     ~     (p=0.841 n=5+5)

name                                                                                                             old alloc/op   new alloc/op   delta
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-block:contentions:count:contentions:count-12       492MB ± 0%     522MB ± 1%   +5.98%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-block:delay:nanoseconds:contentions:count-12       466MB ± 1%     495MB ± 1%   +6.14%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-goroutine:goroutine:count:goroutine:count-12       157MB ± 0%     186MB ± 1%  +18.44%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:alloc_objects:count:space:bytes-12          113MB ± 2%     142MB ± 0%  +25.31%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:alloc_space:bytes:space:bytes-12            122MB ± 1%     148MB ± 5%  +21.11%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:inuse_objects:count:space:bytes-12          101MB ± 2%     129MB ± 2%  +27.60%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:inuse_space:bytes:space:bytes-12            108MB ± 1%     138MB ± 3%  +26.92%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-mutex:contentions:count:contentions:count-12       158MB ± 1%     189MB ± 0%  +19.35%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-mutex:delay:nanoseconds:contentions:count-12       785MB ± 0%     813MB ± 1%   +3.56%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-process_cpu:cpu:nanoseconds:cpu:nanoseconds-12    1.56GB ± 0%    1.59GB ± 0%   +1.87%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-process_cpu:samples:count:cpu:nanoseconds-12      1.58GB ± 0%    1.61GB ± 0%   +1.65%  (p=0.008 n=5+5)

name                                                                                                             old allocs/op  new allocs/op  delta
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-block:contentions:count:contentions:count-12       66.6k ± 0%     67.3k ± 0%   +1.07%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-block:delay:nanoseconds:contentions:count-12       80.7k ± 0%     81.3k ± 0%   +0.80%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-goroutine:goroutine:count:goroutine:count-12       90.4k ± 0%     91.2k ± 0%   +0.86%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:alloc_objects:count:space:bytes-12           124k ± 0%      125k ± 0%   +0.56%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:alloc_space:bytes:space:bytes-12             113k ± 0%      114k ± 0%   +0.66%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:inuse_objects:count:space:bytes-12           111k ± 0%      112k ± 0%   +0.70%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-memory:inuse_space:bytes:space:bytes-12             108k ± 0%      109k ± 0%   +0.81%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-mutex:contentions:count:contentions:count-12       69.5k ± 0%     70.4k ± 0%   +1.33%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-mutex:delay:nanoseconds:contentions:count-12       1.30M ± 0%     1.30M ± 0%   +0.06%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-process_cpu:cpu:nanoseconds:cpu:nanoseconds-12     3.12M ± 0%     3.12M ± 0%   +0.03%  (p=0.008 n=5+5)
SelectMatchingProfilesRealBlock/block-01H406NQE5JEQPV6QBCC39BCHK-process_cpu:samples:count:cpu:nanoseconds-12       3.09M ± 0%     3.09M ± 0%   +0.03%  (p=0.008 n=5+5)

@simonswine simonswine changed the title 20230711 parallelize row groups Parallelize row groups Jul 11, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant