Shard support

[bogovicj]

Note: we'll squash or rebase the commits below if / when we decide to merge.

motivation

This PR aims to support sharding through readBlocks and writeBlocks methods. One priority was to enable re-use of the code here in n5-zarr, because the n5-format will not support sharding.

It does NOT seek to expose any shard-specific API methods (e.g. readShard, writeShard, deleteShard), nor does it make a decision about whether / how to introduce parallelism into the n5 "core" API
as discussed here.

The API here does expose the information needed for developers to write parallel code.
For example this method in n5-imglib2, parallelizes over shards or blocks as appropriate.

We should consider extending this in the future, but again, this factor was not a priority for this PR in particular.

outline of changes

• add N5KeyValueReader.getAttributesKey enables n5-zarr to better re-use code here
• shard shard support
  • add and implement readBlocks, writeBlocks
    • aggregate blocks by shard when appropriate
  • RawShard stores the raw (compressed) shard data into segments that represent the blocks and index
  • RawShardCodec encodes / decodes RawShards to and from ReadData
    • SegmentedReadData tracks which part of shard data should be "grouped"
    • ShardIndex tracks which byte ranges correspond to particular blocks
  • DatasetAccess abstracts reading / writing blocks from shards / files (the DefaultDatasetAccess)
  • PositionValueAccess is the layer between block coordinates and the KeyValueAccess
  • DeterministicSizeDataCodecs enable compile-time validation of DataCodecs for the shard index.
• many new tests for the new functionality
• assorted code cleanup

behavior

Note: when referring to partial reads below, they are partial only if allowed by the backing KeyValueAccess.
Behavior is unchanged for un-sharded datasets, for sharded datasets:

readBlock

Partially read the shard index, partially read the encoded block, decode and return.

readBlocks

High-level behavior is identical to calling readBlock multiple times and returning the collection but aggregates operations per shard.

At a lower level, it performs one partial read for the shard index and one partial read for each requested block.

writeBlock

If no shard exists, create a shard index containing the one block. Encode the block, write the one block and the index.

Otherwise:
Read the data for the entire shard and decode to a RawShard, and add the encoded block data to the raw

• replaces existing data for the block at that position if it exists
• this updates the shard index

write all the data.
This does not decode / encode blocks that are not touched.

writeBlocks

High-level behavior is identical to calling writeBlock multiple times and returning the collection but aggregates operations per shard, i.e. Calls writeBlock once per unique shard - i.e. has one read and one write operation per unique shard.

extensibility

The use of codecs in this and previous PRs made the implementations in n5-zarr simpler. For example, the padding of "incomplete" blocks needed for zarr is now handled by a codec inserted by the ZarrWriter. As a result, it can directly reuse the writeBlock implementation here.

Example: ZarrKeyValueReader does not have readBlock(s) methods

details: readBlocks

• Creates a PositionValueAccess (PVA)
• gets the reader's DatasetAccess (always a DefaultDatasetAccess )
• Calls DatasetAccess.readBlock with the PVA and the blocks grid position
  • The PVA gets the file / object containing the requested block
    • The ReadData produced should read lazily and partially if possible
    • for example, the FileSystemKVA's implementation
  • readBlockRecursive handles the codec chain: ShardCodecs containing BlockCodecs
  • At the bottom level, the block is decoded from a ReadData
  • At intermediate levels
    • a RawShard is decoded from the ReadData
    • Get the relative position of the lower level block from NestedPosition (e.g. block inside a shard)
    • shard.getElementData slices the ReadData appropriately, which is passed to the lower level

details: writeBlocks

• Creates a PositionValueAccess (PVA)

• gets the writer's DatasetAccess (always a DefaultDatasetAccess )

• Calls DatasetAccess.writeBlocks with the PVA and the grid positions for blocks to write

  • Groups blocks according to the shard to which they belong
  • For each shard, get the existing data if it exists, and call writeShardRecursive
  • Decode the data as a RawShard
  • Add the blocks
  • Aside: handle nested shards too but not so concerned about that right now.