Support reading multiple regions with single fetch #118496
Conversation
Tim-Brooks
added
>non-issue
:Distributed Indexing/Distributed
elasticsearchmachine
added
the
Team:Distributed Indexing
|
Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing) |
elasticsearchmachine
added
the
serverless-linked
| regionRange, | ||
| regionsListener.acquire() | ||
| ); | ||
| if (regionGaps.isEmpty() == false) { |
There was a problem hiding this comment.
I wonder if we should break it up if we hit a region that needs no data read? We could in principle risk having everything but the first and last region of a file and thus we could download an excess amount of extra data.
We can also build this into the shared stream handler.
Or forego it for now. I'd be ok with that too, but wanted to get views on it.
| @@ -1082,7 +1205,29 @@ void populateAndRead( | |||
| } | |||
| } | |||
|
|
|||
| List<Runnable> gapFillingTasks( | |||
| int gapOffset, | |||
There was a problem hiding this comment.
Is this not the region offset more than the gap offset?
| ) { | ||
| ArrayList<Runnable> gapFillingTasks = new ArrayList<>(); | ||
| for (Tuple<CacheFileRegion<KeyType>, RegionGaps> gapsToFetch : gaps) { | ||
| int offset = Math.toIntExact(gapsToFetch.v2().regionOffset()); |
There was a problem hiding this comment.
This seems unsafe, I think we need this offset to be a long and this has to pass through all the way to relativePos in SourceInputStreamFactory.create(relativePos, ...)?
This commit adds support for filling multiple regions in blob cache with
a single fetch from the object store.