Aggregate - #219
Merged
Merged
Conversation
cmhulbert
requested review from
bogovicj and
tpietzsch
and removed request for
tpietzsch
March 26, 2026 20:04
cmhulbert
force-pushed
the
aggregate
branch
4 times, most recently
from
April 3, 2026 15:54
b4aef69 to
bdf7c57
Compare
This is in preparation for implementing prefetch() for cloud storage backends...
* and a test
* Default- and Aggregating- * add tests
…ment feat: use the aggregate/prefetch logic with VolatileReadData. By default, `VolatileReadData.from` now uses AggregatingSliceTrackingLazyRead. This is useful to ensure the default VolatileReadData behaves reasonably even for reading many chunks from the same shard.
It doesn't do any prefetching, just keeps track of what has been materialized (and uses that to avoid materializing slices that are already fully covered).
instead of "...SliceTracking..." which is not their main feature
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This contains code from the previous
batching-aggregatebranch, but adapted to the changes to development since then. Namely VolatileReadData, and how it is now used.This changes the default behavior of
VolatileReadData.fromto return aLazyReadDataover anAggregatingSliceTrackingLazyRead. The motivation for this is that the only way to get the aggregation/prefetch behavior would be to manually wrap it in each KVA implementation. It's desirable to be able to do this, but I think it is better to assume reasonable behavior for the default, and allow specialized implementations to handle the optimizations.Even for FileSystemKVA, we want the aggregating behavior in the default case, since we can not be sure if the files we are reading are actually local (in which case many chunk reads over the same block are maybe fine? I still think probably not ideal), or if they are mounted or remote in some way.
This is especially true since the default behavior for
readBlockcalls out toreadChunksRecursivefor all blocks, resulting in individual partial reads for each chunk, rather than materializing the VolatileReadData once up front. TheAggregatingSliceTrackingLazyReadhandles this.