Merge pull request #28 from janelia-cellmap/default_chunks_fix

Default chunks fix

Author: d-v-b

.github/workflows/test.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.10', '3.11', '3.12']
+        python-version: ['3.9', '3.10', '3.11', '3.12']
     steps:
     - uses: actions/checkout@v4
     - name: Install dependencies

docs/index.md

@@ -6,18 +6,20 @@ Static typing and runtime validation for Zarr hierachies.
 
 ## Overview
 
-`pydantic-zarr` expresses data stored in the [zarr](https://zarr.readthedocs.io/en/stable/) format with [Pydantic](https://docs.pydantic.dev/1.10/). Specifically, `pydantic-zarr` encodes Zarr groups and arrays as [Pydantic models](https://docs.pydantic.dev/1.10/usage/models/). Programmers can use these models to formalize the structure of Zarr hierarchies, and apply type-checking and runtime validation to Zarr data.
+`pydantic-zarr` expresses data stored in the [Zarr](https://zarr.readthedocs.io/en/stable/) format with [Pydantic](https://docs.pydantic.dev/1.10/). Specifically, `pydantic-zarr` encodes Zarr groups and arrays as [Pydantic models](https://docs.pydantic.dev/1.10/usage/models/). These models are useful for formalizing the structure of Zarr hierarchies, type-checking Zarr hierarchies, and runtime validation for Zarr-based data.
 
 
 ```python
 import zarr
 from pydantic_zarr.v2 import GroupSpec
 
+# create a Zarr group
 group = zarr.group(path='foo')
+# put an array inside the group
 array = zarr.create(store = group.store, path='foo/bar', shape=10, dtype='uint8')
 array.attrs.put({'metadata': 'hello'})
 
-# this is a pydantic model
+# create a pydantic model to model the Zarr group
 spec = GroupSpec.from_zarr(group)
 print(spec.model_dump())
 """
@@ -48,41 +50,54 @@ print(spec.model_dump())
 """
 ```
 
-Important note: this library only provides tools to represent the *layout* of Zarr groups and arrays, and the structure of their attributes. It performs no type checking or runtime validation of the multidimensional array data contained inside Zarr arrays.
+More examples can be found in the [usage guide](usage_zarr_v2.md).
 
 ## Installation
 
 `pip install -U pydantic-zarr`
 
-## Design
 
-A Zarr group can be schematized as two elements:
+### Limitations
 
+#### No array data operations
+This library only provides tools to represent the *layout* of Zarr groups and arrays, and the structure of their attributes. `pydantic-zarr` performs no type checking or runtime validation of the multidimensional array data contained *inside* Zarr arrays, and `pydantic-zarr` does not contain any tools for efficiently reading or writing Zarr arrays.
 
-- `attributes`: A dict-like object with string keys and JSON-serializable values.
-- `members`: A dict-like object with string keys and values that are other Zarr groups, or Zarr arrays.
+#### Supported Zarr versions
 
-A Zarr array can be schematized similarly, but without the `members` property, and with a set of arrays-specific properties like `shape`, `dtype`, etc.
+This library supports [version 2](https://zarr.readthedocs.io/en/stable/spec/v2.html) of the Zarr format, with partial support for [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html). Progress towards complete support for Zarr v3 is tracked by [this issue](https://github.com/d-v-b/pydantic-zarr/issues/3).
 
-Note the use of the term "schematized": Zarr arrays also represent N-dimensional array data, but `pydantic-zarr` does not treat that data as part of the "schema" of a Zarr array.
+## Design
 
-Accordingly, in `pydantic-zarr`, Zarr groups are encoded by the `GroupSpec` class with two fields:
+A Zarr group can be modeled as an object with two properties:
 
+- `attributes`: A dict-like object, with keys that are strings, values that are JSON-serializable.
+- `members`: A dict-like object, with keys that strings and values that are other Zarr groups, or Zarr arrays.
 
-- `GroupSpec.attributes`: either a `Mapping` or a `pydantic.BaseModel`.
-- `GroupSpec.members`: a mapping with string keys and values that must be `GroupSpec` or `ArraySpec` instances.
+A Zarr array can be modeled similarly, but without the `members` property (because Zarr arrays cannot contain Zarr groups or arrays), and with a set of array-specific properties like `shape`, `dtype`, etc.
 
-Zarr arrays are represented by the `ArraySpec` class, which has a similar `attributes` field, as well as fields for all the Zarr array properties (`dtype`, `shape`, `chunks`, etc).
+Note the use of the term "modeled": Zarr arrays are useful because they store N-dimensional array data, but `pydantic-zarr` does not treat that data as part of the "model" of a Zarr array.
+
+In `pydantic-zarr`, Zarr groups are modeled by the `GroupSpec` class, which is a [`Pydantic model`](https://docs.pydantic.dev/latest/concepts/models/) with two fields:
 
+- `attributes`: either a `Mapping` or a `pydantic.BaseModel`.
+- `members`: either a mapping with string keys and values that must be `GroupSpec` or `ArraySpec` instances, or the value `Null`. The use of nullability is explained in its own [section](#nullable-members).
+
+Zarr arrays are represented by the `ArraySpec` class, which has a similar `attributes` field, as well as fields for all the Zarr array properties (`dtype`, `shape`, `chunks`, etc).
 
 `GroupSpec` and `ArraySpec` are both [generic models](https://docs.pydantic.dev/1.10/usage/models/#generic-models). `GroupSpec` takes two type parameters, the first specializing the type of `GroupSpec.attributes`, and the second specializing the type of the *values* of `GroupSpec.members` (the keys of `GroupSpec.members` are always strings). `ArraySpec` only takes one type parameter, which specializes the type of `ArraySpec.attributes`.
 
-Examples using this generic typing functionality can be found in the [usage guide](usage.md#using-generic-types).
+Examples using this generic typing functionality can be found in the [usage guide](usage_zarr_v2.md#using-generic-types).
 
-## Supported Zarr versions
+### Nullable `members`
 
-This library supports [version 2](https://zarr.readthedocs.io/en/stable/spec/v2.html) of the Zarr format, with partial support for [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html). Progress towards complete support for Zarr v3 is tracked by [this issue](https://github.com/d-v-b/pydantic-zarr/issues/3).
+When a Zarr group has no members, a `GroupSpec` model of that Zarr group will have its `members` attribute set to the empty dict `{}`. But there are scenarios where the members of a Zarr group are unknown:
+
+- Some Zarr storage backends do not support directory listing, in which case it is possible to access a Zarr group and inspect its attributes, but impossible to discover its members. So the members of such a Zarr group are unknown.
+- Traversing a deeply nested large Zarr group on high latency storage can be slow. This can be mitigated by only partially traversing the hierarchy, e.g. only inspecting the root group and N subgroups. This defines a sub-hierarchy of the full hierarchy; leaf groups of this subtree by definition did not have their members checked, and so their members are unknown.
+- A Zarr hierarchy can be represented as a mapping `M` from paths to nodes (array or group). In this case, if `M["key"]` is a model of a Zarr group `G`, then `M["key/subkey"]` would encode a member of `G`. Since the key structure of the mapping `M` is doing the work of encoding the members of `G`, there is no value in `G` having a members attribute that claims anything about the members of `G`, and so `G.members` should be modeled as unknown.
+
+To handle these cases, `pydantic-zarr` allows the `members` attribute of a `GroupSpec` to be `Null`.
 
-## Supported Pydantic versions
+## Standardization
 
-This library is based on Pydantic version 2.
+The Zarr specifications do not define a model of the Zarr hierarchy. `pydantic-zarr` is an implementation of a particular model that can be found formalized in this [specification document](https://github.com/d-v-b/zeps/blob/zom/draft/ZEP0006.md), which has been proposed for inclusion in the Zarr specifications. You can find the discussion of that proposal in [this pull request](https://github.com/zarr-developers/zeps/pull/46).

docs/usage_zarr_v2.md

@@ -6,7 +6,7 @@
 
 The `GroupSpec` and `ArraySpec` classes represent Zarr v2 groups and arrays, respectively. To create an instance of a `GroupSpec` or `ArraySpec` from an existing Zarr group or array, pass the Zarr group / array to the `.from_zarr` method defined on the `GroupSpec` / `ArraySpec` classes. This will result in a `pydantic-zarr` model of the Zarr object.
 
-Note that `GroupSpec.from_zarr(zarr_group)` will traverse the entire hierarchy under `zarr_group`. Future versions of this library may introduce a limit on the depth of this traversal: see [#2](https://github.com/d-v-b/pydantic-zarr/issues/2).
+>  By default `GroupSpec.from_zarr(zarr_group)` will traverse the entire hierarchy under `zarr_group`. This can be extremely slow if used on an extensive Zarr group on high latency storage. To limit the depth of traversal to a specific depth, use the `depth` keyword argument, e.g. `GroupSpec.from_zarr(zarr_group, depth=1)`
 
 Note that `from_zarr` will *not* read the data inside an array.
 
@@ -78,7 +78,7 @@ print(dict(group2['bar'].attrs))
 
 ### Creating from an array
 
-The `ArraySpec` class has a `from_array` static method that takes a numpy-array-like object and returns an `ArraySpec` with `shape` and `dtype` fields matching those of the array-like object.
+The `ArraySpec` class has a `from_array` static method that takes an array-like object and returns an `ArraySpec` with `shape` and `dtype` fields matching those of the array-like object.
 
 ```python
 from pydantic_zarr.v2 import ArraySpec
@@ -241,53 +241,64 @@ print(GroupSpec.from_flat(tree).model_dump())
 
 The `like` method works by converting both input models to `dict` via `pydantic.BaseModel.model_dump`, and comparing the `dict` representation of the models. This means that instances of two different subclasses of `GroupSpec`, which would not be considered equal according to the `==` operator, will be considered `like` if and only if they serialize to identical `dict` instances.
 
-The `like` method also takes keyword arguments `include` and `exclude`, which results in attributes being explicitly included or excluded from the model comparison. So it's possible to use `like` to check if two `ArraySpec` instances have the same `shape` and `dtype` by calling `array_a.like(array_b, include={'shape', 'dtype'})`. This is useful if you don't care about the compressor or filters and just want to ensure that you can safely write an in-memory array to a Zarr array.
+The `like` method takes keyword arguments `include` and `exclude`, which determine the attributes included or excluded from the model comparison. So it's possible to use `like` to check if two `ArraySpec` instances have the same `shape`, `dtype` and `chunks` by calling `array_a.like(array_b, include={'shape', 'dtype', 'chunks'})`. This is useful if you don't care about the compressor or filters and just want to ensure that you can safely write an in-memory array to a Zarr array, which depends just on the two arrays having matching `shape`, `dtype`, and `chunks` attributes.
 
 ```python
 from pydantic_zarr.v2 import ArraySpec, GroupSpec
 import zarr
 arr_a = ArraySpec(shape=(1,), dtype='uint8', chunks=(1,))
-arr_b = ArraySpec(shape=(2,), dtype='uint8', chunks=(1,)) # array with different shape
+# make an array with a different shape
+arr_b = ArraySpec(shape=(2,), dtype='uint8', chunks=(1,))
 
-print(arr_a.like(arr_b)) # False, because of mismatched shape
+# Returns False, because of mismatched shape
+print(arr_a.like(arr_b)) 
 #> False
 
-print(arr_a.like(arr_b, exclude={'shape'})) # True, because we exclude shape.
+# Returns True, because we exclude shape.
+print(arr_a.like(arr_b, exclude={'shape'}))
 #> True
 
 # `ArraySpec.like` will convert a zarr.Array to ArraySpec
 store = zarr.MemoryStore()
-arr_a_stored = arr_a.to_zarr(store, path='arr_a') # this is a zarr.Array
+# This is a zarr.Array
+arr_a_stored = arr_a.to_zarr(store, path='arr_a')
 
-print(arr_a.like(arr_a_stored)) # arr_a is like the zarr.Array version of itself
+# arr_a is like the zarr.Array version of itself
+print(arr_a.like(arr_a_stored)) 
 #> True
 
-print(arr_b.like(arr_a_stored)) # False, because of mismatched shape
+# Returns False, because of mismatched shape
+print(arr_b.like(arr_a_stored)) 
 #> False
 
-print(arr_b.like(arr_a_stored, exclude={'shape'})) # True, because we exclude shape.
+# Returns True, because we exclude shape.
+print(arr_b.like(arr_a_stored, exclude={'shape'}))
 #> True
 
-# the same thing thing for groups
+# The same thing, but for groups
 g_a = GroupSpec(attributes={'foo': 10}, members={'a': arr_a, 'b': arr_b})
 g_b = GroupSpec(attributes={'foo': 11}, members={'a': arr_a, 'b': arr_b})
 
-print(g_a.like(g_a)) # g_a is like itself
+# g_a is like itself
+print(g_a.like(g_a))
 #> True
 
-print(g_a.like(g_b)) # False, because of mismatched attributes
+# Returns False, because of mismatched attributes
+print(g_a.like(g_b)) 
 #> False
 
-print(g_a.like(g_b, exclude={'attributes'})) # True, because we ignore attributes
+# Returns True, because we ignore attributes
+print(g_a.like(g_b, exclude={'attributes'}))
 #> True
 
-print(g_a.like(g_a.to_zarr(store, path='g_a'))) # g_a is like its zarr.Group counterpart
+# g_a is like its zarr.Group counterpart
+print(g_a.like(g_a.to_zarr(store, path='g_a')))
 #> True
 ```
 
 ## Using generic types
 
-The following examples demonstrate how to specialize `GroupSpec` and `ArraySpec` with type parameters. By specializing `GroupSpec` or `ArraySpec` in this way, python type checkers and Pydantic can type-check elements of a Zarr hierarchy.
+This example shows how to specialize `GroupSpec` and `ArraySpec` with type parameters. By specializing `GroupSpec` or `ArraySpec` in this way, python type checkers and Pydantic can type-check elements of a Zarr hierarchy.
 
 ```python
 import sys
@@ -324,7 +335,7 @@ print(SpecificAttrsGroup(attributes={'a': 100, 'b': 100}))
 #> zarr_version=2 attributes={'a': 100, 'b': 100} members={}
 
 # a Zarr group that only contains arrays -- no subgroups!
-# we re-use the Tattributes type variable defined in pydantic_zarr.core
+# we re-use the TAttr type variable defined in pydantic_zarr.core
 ArraysOnlyGroup = GroupSpec[TAttr, ArraySpec]
 
 try:

poetry.lock

@@ -11,6 +11,7 @@ python = "^3.9"
 zarr = "^2.14.2"
 pydantic = "^2.0.0"
 typing-extensions = {version = "^4.7.1", python = "<3.12"}
+eval-type-backport = "^0.1.3"
 
 [tool.poetry.group.dev.dependencies]
 pytest = "^7.3.1"

pyproject.toml

@@ -1,11 +1,11 @@
 from __future__ import annotations
+from typing_extensions import TypeAlias
 from typing import (
     Any,
     Dict,
     Literal,
     Mapping,
     Set,
-    TypeAlias,
     Union,
 )
 from pydantic import BaseModel, ConfigDict