Format-specific `DatasetAttributes` challenges

[bogovicj]

One create a "generic" DatasetAttributes instance, and use that with any N5Writer to create a dataset and write blocks with it. For example, this works:

ZarrKeyValueWriter zarr = ...
DatasetAttributes attrs = new DatasetAttributes(...

zarr.createDataset(path, attrs)
zarr.writeBlock(path, attrs, block);

and internally "converts" from the generic to a format-specific DatasetAttributes instance (in this case ZarrDatasetAttributes) using default values when appropriate (in this case, for "fill_value" and "dimension_separator").

One (minor) downside of this is that we pay an extra cost to check and possibly convert the attributes every time a block is
written (see the current ZarrKeyValueWriter.writeBlock.

Another downside is the maintenance burden. So far it has been minimal, but is now more challenging given that we have a public DatasetAttributes constructor with a BlockCodecInfo argument. This means that the BlockCodecInfo needs to be inspected and possibly replaced with an appropriate replacement in order to write data correctly for a specific format.

For example

new DatasetAttributes(
  dimensions,  blockSize, dataType,
  new N5BlockCodecInfo(), 
  new GzipCompression());

would naively produce wrong zarr blocks if the N5BlockCodecInfo is used.

Similarly

new DatasetAttributes(
  dimensions,  blockSize, dataType,
  new RawBlockCodecInfo(), 
  new GzipCompression());

would naively produce wrong n5 blocks if the RawBlockCodecInfo is used.

writeBlock attribute conversion cost

we pay an extra cost to check and possibly convert the attributes every time a block is
written

this is technically avoidable with:

ZarrKeyValueWriter zarr = ...
DatasetAttributes attrs = new DatasetAttributes(...

zarr.createDataset(path, attrs)
DatasetAttributes actualAttrs = zarr.getDatasetAttributes(path);
zarr.writeBlock(path, actualAttrs, block);

but is ugly, and no one should or will do this.

[bogovicj]

Some options

No special codec handling

This would require that developers always know the format for a given writer so that they can create an appropriate DatasetAttributes instance, is antithetical to N5's design philosophy, so is not a good option in my view.

Dataset attributes don't have codecs

This would make it difficult or impossible to customize the codecs zarr v3 datasets, so is not a good option in my view.

Convert Datataset attributes

This is the status quo. We could make this work, but has some downsides:

• high maintenance burden
• the computational cost of conversion remains
  • and is higher than before because conversion is more complicated
• unclear what the behavior should be
  • we would be changing what codecs are used without telling
  • how is conversion done when invalid codecs are specified (e.g. shard codec for n5 format)

N5Writers create DatasetAttributes

Rather than creating DatasetAttributes with a constructor, we could have a method on N5Writer that
creates a format appropriate DatasetAttributes instance with appropriate codecs etc. Something like:

N5Writer n5w;
attributes = n5w.createDatasetAttributes(dimensions, blockSize, dataType, compression);

Besides making the maintenance easier, this would also remove the conversion cost of every writeBlock call.

A significant downside is that this is a breaking API change.

This is my preferred solution right now.

[tpietzsch]

I think, the fact that you have to pass in DatasetAttributes to the writeBlock method is a consequence of the design choice that N5Reader/N5Writer should be stateless.

With a more stateful design, the writer would internally have a Map from dataset paths to DatasetAttributes that is populated when a dataset is created or opened. With the current design, this "caching" of DatasetAttributes is externalized to the user of the API.

The expectation is that "caching" DatasetAttributes is all there is to it. You are not supposed to mess with the attributes. I think you can already break things by just passing different attributes into the writeBlock/readBlock call. You can absolutely corrupt an existing dataset by writing blocks with a different grid size, compression, etc. You can do this already now.

I would definitely avoid doing any conversion etc in writeBlock calls.

Maybe in a toned-down version of your suggestion ("N5Writers create DatasetAttributes") we could get away without API breakage?
We could add the N5Writer.createDatasetAttrinbutes method as you suggest.
Maybe, we @Deprecate the corresponding DatasetAttributes constructor (where only compression is given and the block codec is left open) to encourage using the N5Writer method. We would have to make DatasetAttributes initialization partially lazy. Basically you could pass a partially initialized DatasetAttributes into N5Writer.createDataset and it would be fully initialized with details specific to the N5Writer implementation.

But maybe breaking API is also ok. It wouldn't be too complicated to fix the breakage, I think.

[bogovicj]

Attach an "instantiated" DatasetAttributes to a "generic" instantiated DatasetAttributes instance that is passed to createDataset?

Another use case:
get a DatasetAttributes from one (N5) Writer and pass the same instance to another (Zarr)Writer in order to create an analogous dataset.