High level topics

[bogovicj]

What is the preferred workflow for writing a region in an image?

When a developer doesn't know if a dataset is sharded or not, how should they go about efficiently writing
data with N5 (or reading, but its easier).

• assume sharded? use readShard / writeShard
• use readBlocks / writeBlocks
• use readBlock / writeBlock and N5 API does some tricks under-the-covers
  • NO in my (JB, CH) opinion.

goals

• behavior must be correct even when using "wrong" access methods 🚫 🐛s
• shard methods will have some, but low overhead when not-sharded, and efficient when sharded
• block methods will be conservative and potentially not efficient when sharded
• blocks should use shard information when possible, fall back to block when not

API methods

Strongly confident that we will add the methods:

• N5Reader.readBlocks
• N5Writer.writeBlocks

Somewhat confident that we will add the methods:

• N5Reader.readShard

• N5Writer.writeShard

• In general, do we include shard specific analogues for things that cover blocks now

  • Do we include isSharded for N5Reader
  • Do we include a way to get shard size from DatasetAttributes?
  • Getting a NestedGrid from DatasetAttributes is another possibility

• DatasetAttributes.groupBlocks

  • takes block positions and returns a map or list-of-lists that groups blocks by shard
  • perhaps belongs in another class / interface (for future-proofing, e.g. nesting, do we care at this level? Likely not, most important is the new separation of parallelizable unit vs compressable unit, only two things)

• DatasetAttributes.blockIterator

• ShardAccess object could be how we handle read/write "policy" in the future (how is this different from LockedFileChannel/ ReadData?)

  • in general, a way to over-ride the default read/write behavior (?)

• use auto-closable property readers and writers to flush things

  • probably not now, but later after we implement policies

shard method details

What type of object does readShard return? and what can one do with it?

• A ReadData
• A Shard object with
• A ShardAccess (if it exists)

Default behaviors

• readBlock for a sharded dataset:
  • read the shard index
  • partial read the block if the backend allows it
• writeBlock / removeBlock for a sharded dataset:
  • reads the whole shard (if it exists)
  • adds / replaces / deletes the block as necessary
  • updates the index as necessary
  • writes / deletes the whole shard as necessary
• writeBlocks should be efficient in the sharded context
  • requires grouping blocks by shard (see above)
• readBlocks cost of ignoring shard is less bad that writeBlocks
  • but it could (maybe should?) be smarter than naiive case when sharded
    • e.g. only read shard index once per shard
    • heuristic for aggregating reads
    • we've already started work on this

[bogovicj]

general agreement

• add readBlocks / writeBlocks
• do not add readShard / writeShard
  • readBlocks / writeBlocks cover necessary use cases
• parallelization is important
• KeyValueAccess should have some responsibility for how multiple partial reads are aggregated
  • e.g. choosing between many partial reads vs one longer contiguous read
• some "nice to have" features are not a priority
  • nesting shards
  • "local staging"

needs clarification

probably informed by prototyping

• What does the signature of writeBlocks look like?
  • writeBlocks(List<int[]> blockCoordinates, Function<int[],DataBlock> blockGenerator, ExecutorService exec)
  • writeBlocks(List<DataBlock> blocks, ExecutorService exec)
  • writeBlocks(List<DataBlock> blocks)
• how is virtualization of DataBlocks handled
  • with a Function (as in the first example)
  • with a LazyDataBlock implementation

disagreement

• what / how much should a developer know about whether a dataset is sharded or not
  • it must be accessible in some way for developers to know what the "unit of parallelization" is
• who is responsible for parallelization of writing
  • N5Writer (Stephan)
  • n5-imglib2 (Tobias)
• sophistication of n5 core methods
  • does n5 core or n5-imglib2 handle parallelization?
• how much do we consider the "nice to have" features now in order to inform the API

[bogovicj]

Old code using the N5 API may use writeBlock in parallel, because it is safe (there were not shards).
We may not want to force maintainers to update this code and could decide that writeBlock
should always be safe to execute in parallel.

What if:

• "Block" refers to the unit that can be written safely in parallel ("shards")
• "Chunk" refers to the smallest unit that can be read

Then:

• we would add methods called readChunks, writeChunks
  • these would
• for unsharded datasets:
  • readBlock behaves as a it does now
  • writeBlock behaves as a it does now
• for sharded datasets:
  • readBlock and writeBlock could have the current behavior but as if the dataset were not sharded
    • with "block size" = "shard size"