StorageFormat enum - usage and naming

[bogovicj]

This issue discusses updates and changes to the StorageFormat enum, which is used by N5Factory to explicitly request a reader or writer of a particular type.

This issue is a fleshing out of @mkitti 's comment here saalfeldlab/n5-ij#116

Proposals

1. We should add ZARR3 to the enum to more clearly refer to Zarr v3.

Questions

1. ZARR will remain, what should its behavior be?

Details

Proposal 1 (add ZARR3)

When we updated this repo to support Zarr v3, we added ZARR2 to the enum to refer to Zarr v2, intending that ZARR refer to the latest version (v3). This probably should be clearer though

Question 1 (Behavior of ZARR)

• Option A: always refers to the latest version \
  • existing code using ZARR to mean zarr v2 may not work as expected
• Option B: uses keys (.zgroup / .zarray / zarr.json) to decide between v2 and v3 if the container already exists, else uses the latest version if a writer is requested
  • in this case using a StorageFormat does not specify a format (the initial purpose of the enum) Proposals
  • this behavior is identical to not providing a StorageFormat

[mkitti]

this behavior is identical to not providing a StorageFormat

If I specify ZARR it should error when opening a N5 dataset. Thus, I think the behavior should deviate from not providing a StorageFormat.

[mkitti]

For example, N5Reader might look like this:

	private N5Reader openReader(@Nullable final StorageFormat storage, @Nullable final KeyValueAccess access, URI location) {


		if (storage == null) {
			for (final StorageFormat format : orderedStorageFormats()) {
				try {
					return openReader(format, access, location);
				} catch (final Throwable ignored) {
				}
			}
			throw new N5Exception("Unable to open " + location + " as N5Reader");

		} else {

			final String containerPath = location.toString();
			switch (storage) {
			case N5:
				return new N5KeyValueReader(access, containerPath, gsonBuilder, cacheAttributes);
			case ZARR3:
				return new ZarrV3KeyValueReader(access,containerPath, gsonBuilder, zarrMapN5DatasetAttributes, zarrMergeAttributes, cacheAttributes);
			case ZARR2:
				return new ZarrKeyValueReader(access, containerPath, gsonBuilder, zarrMapN5DatasetAttributes, zarrMergeAttributes, cacheAttributes);
			case ZARR:
				try {
					return new ZarrV3KeyValueReader(access,containerPath, gsonBuilder, zarrMapN5DatasetAttributes, zarrMergeAttributes, cacheAttributes);
				} catch(N5Exception e) { }
				return new ZarrKeyValueReader(access, containerPath, gsonBuilder, zarrMapN5DatasetAttributes, zarrMergeAttributes, cacheAttributes);
			case HDF5:
				return new N5HDF5Reader(containerPath, hdf5OverrideBlockSize, gsonBuilder, hdf5DefaultBlockSize);
			}
			return null;
		}
	}

[cmhulbert]

This PR (#50) changes the behavior a bit with respect to the ZARR StorageFormat enum. The new behavior is:

StorageFormat.ZARR3 ( prepended schema zarr3: )

• explicitly zarr3. So it will fail for readers on zarr2, and it will create a zarr3 when writing, even if zarr2 metadata is present

StorageFormat.ZARR2 ( prepended schema zarr2: )

• explicitly zarr2. Simillar behavior to the above, but for zarr2

StorageFormat.ZARR (prepended schema zarr: )

• If no existing zarr2 or 3 metadata, then it will either create Zarr3 container if getting an N5Writer, or fail if getting an N5Reader
• Will attempt to create a zarr reader/writer that matches the existing version at that location, if one is present
  • If .zgroup is found for example, result will be Zarr2 Reader/Writer
  • if zarr.json is found, result will be Zarr3 Reader/Writer
  • if metadata for both version are found, prefer Zarr3

So in terms of migration, previous code only had access to StorageFormat.ZARR This means that code will still work over existing zarr2 containers, but will otherwise default to zarr3 when no other information is available.

A technicalynote: the only way to check if an existing container of a given version is present is to try an open it as an N5Reader. This means that when create an N5Writer with StorageFormat.ZARR, we incur an overhead of 1 or 2 additional N5Reader instantiations (or at least attempts). In order we try to get a Zarr3Reader, then a Zarr2Reader, and depending on if one or the other succeeds, or neither, we decide which Zarr N5Writer to create.