Write full shards, revise DatasetCodec - #190
Merged
Merged
Conversation
tpietzsch
marked this pull request as draft
January 8, 2026 20:51
Closed
This was referenced Jan 9, 2026
Closed
Closed
This gives the maximum pixel position covered by a shard/block at a given nesting level. Used for DatasetAccess.writeRegion().
NestedPosition should be ordered such that positions from a (sub-)shard
are grouped together:
For nested = {X,Y,Z} compare by Z, then Y, then X.
For X = [x,y,z] compare by z, then y, then x. (flattening order)
The List<DataBlock> returned by readBlocks is now in the same order as List<long[]> of positions requested. writeBlocks now doesn't read and decode existing shards that are completely overwritten.
Signed-off-by: Caleb Hulbert <cmhulbert@gmail.com>
Signed-off-by: Caleb Hulbert <cmhulbert@gmail.com>
Signed-off-by: Caleb Hulbert <cmhulbert@gmail.com>
Signed-off-by: Caleb Hulbert <cmhulbert@gmail.com>
tpietzsch
force-pushed
the
write-shard
branch
from
January 12, 2026 10:20
56f29e4 to
41bc251
Compare
Collaborator
Author
|
After our discussion, I changed/added the following:
I also rewrote the git history so that the individual commits make more sense, and put @cmhulbert |
tpietzsch
marked this pull request as ready for review
January 12, 2026 10:40
cmhulbert
self-requested a review
January 12, 2026 15:07
cmhulbert
approved these changes
Jan 12, 2026
cmhulbert
left a comment
Contributor
There was a problem hiding this comment.
I fixed the merge conflict with ShardTest but otherwise, happy to merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds new methods to
DatasetAccess:to read and write a shard as a "larger
DataBlock"This internally reads all the DataBlocks in the shard and re-assembles the data into a larger flattened array (and reverse of that for writing).
(This is in preparation for the "big renaming" when shard becomes DataBlock, and DataBlock becomes chunk.)
This PR also revises the buld
readBlocksandwriteBlocksmethods, fixing TODOs that were still open, namely:readBlocksreturns blocks in the same order as requested. The returnedList<DataBlock>is in the same order as the requestedList<long[]>of block grid positions. It will havenullfor blocks that don't exist. (This probably makes larger parts of Implement shardExists and readBlocksExist #187 #189 unnecessary, becausereadBlocksExistscan just filternulls from the resulting list.)writeBlockschecks whether all blocks in a (sub-)shard are written, and if so avoids reading and decoding the existing data.Finally, this PR also revises generics on
DatasetCodecand mergesDatasetCodecintoBlockCodecon creation, making special treatment inDefaultDatasetAccessunnecessary (i.e.,DatasetAccessdoesn't need to know aboutDatasetCodecat all).