Skip to content

Write full shards, revise DatasetCodec - #190

Merged
cmhulbert merged 18 commits into
developmentfrom
write-shard
Jan 12, 2026
Merged

Write full shards, revise DatasetCodec#190
cmhulbert merged 18 commits into
developmentfrom
write-shard

Conversation

@tpietzsch

Copy link
Copy Markdown
Collaborator

This PR adds new methods to DatasetAccess:

void writeShard(PositionValueAccess pva, DataBlock<T> dataBlock, int level) throws N5IOException;

DataBlock<T> readShard(PositionValueAccess pva, long[] shardGridPosition, int[] shardSizeInPixels, int level) throws N5IOException;

to read and write a shard as a "larger DataBlock"
This internally reads all the DataBlocks in the shard and re-assembles the data into a larger flattened array (and reverse of that for writing).
(This is in preparation for the "big renaming" when shard becomes DataBlock, and DataBlock becomes chunk.)

This PR also revises the buld readBlocks and writeBlocks methods, fixing TODOs that were still open, namely:

  • readBlocks returns blocks in the same order as requested. The returned List<DataBlock> is in the same order as the requested List<long[]> of block grid positions. It will have null for blocks that don't exist. (This probably makes larger parts of Implement shardExists and readBlocksExist #187 #189 unnecessary, because readBlocksExists can just filter nulls from the resulting list.)
  • writeBlocks checks whether all blocks in a (sub-)shard are written, and if so avoids reading and decoding the existing data.

Finally, this PR also revises generics on DatasetCodec and merges DatasetCodec into BlockCodec on creation, making special treatment in DefaultDatasetAccess unnecessary (i.e., DatasetAccess doesn't need to know about DatasetCodec at all).

@tpietzsch
tpietzsch marked this pull request as draft January 8, 2026 20:51
@tpietzsch tpietzsch mentioned this pull request Jan 8, 2026
@cmhulbert cmhulbert linked an issue Jan 9, 2026 that may be closed by this pull request
tpietzsch and others added 15 commits January 10, 2026 22:58
This gives the maximum pixel position covered by a shard/block at a
given nesting level. Used for DatasetAccess.writeRegion().
NestedPosition should be ordered such that positions from a (sub-)shard
are grouped together:
For nested = {X,Y,Z} compare by Z, then Y, then X.
For X = [x,y,z] compare by z, then y, then x. (flattening order)
The List<DataBlock> returned by readBlocks is now in the same order as
List<long[]> of positions requested.

writeBlocks now doesn't read and decode existing shards that are
completely overwritten.
Signed-off-by: Caleb Hulbert <cmhulbert@gmail.com>
Signed-off-by: Caleb Hulbert <cmhulbert@gmail.com>
Signed-off-by: Caleb Hulbert <cmhulbert@gmail.com>
Signed-off-by: Caleb Hulbert <cmhulbert@gmail.com>
@tpietzsch

Copy link
Copy Markdown
Collaborator Author

After our discussion, I changed/added the following:

  • Revise DatasetCodec generics to DatasetCodec<S,T> with DataBlock<T> encode(DataBlock<S>) etc.
  • DatasetAccess.readShard() always truncates at the dataset border.
  • DatasetAccess.read/writeShard() for non-sharded datasets short-cut to just read/write a DataBlock without doing any copying.
  • DatasetAccess.read/writeShard() handle duplicate block positions correctly.

I also rewrote the git history so that the individual commits make more sense, and put @cmhulbert shardExists commits back on top.

@tpietzsch
tpietzsch marked this pull request as ready for review January 12, 2026 10:40
@cmhulbert
cmhulbert self-requested a review January 12, 2026 15:07

@cmhulbert cmhulbert left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fixed the merge conflict with ShardTest but otherwise, happy to merge.

@cmhulbert
cmhulbert merged commit 28a987a into development Jan 12, 2026
2 checks passed
@cmhulbert
cmhulbert deleted the write-shard branch January 12, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement shardExists

2 participants