Sharding support?

[srivarra]

Hello, is there a way to define sharding parameters with ArraySpec?

[d-v-b]

yes but it's not very convenient:

# /// script
# requires-python = "==3.12"
# dependencies = ["pydantic-zarr"]
# ///

from pydantic_zarr.v3 import ArraySpec

model = ArraySpec(
    shape=(100,), 
    data_type='int16', 
    fill_value=0, 
    chunk_grid={"name": "regular", "configuration": {"chunk_shape": (50,)}},
    chunk_key_encoding={"name": "default", "configuration": {"separator": "/"}},
    codecs=[{"name": "sharding_indexed", "configuration": {"chunk_shape": (10,)}}]
    )

arr = model.to_zarr({}, 'foo')
print(arr.info)
"""
Type               : Array
Zarr format        : 3
Data type          : Int16(endianness='little')
Fill value         : 0
Shape              : (100,)
Shard shape        : (50,)
Chunk shape        : (10,)
Order              : C
Read-only          : False
Store type         : MemoryStore
Filters            : ()
Serializer         : BytesCodec(endian=<Endian.little: 'little'>)
Compressors        : ()
No. bytes          : 200
"""

Unfortunately this is the nature of the way sharding is defined in zarr v3 -- it's designed as a detail (a particular codec) instead of a central feature.

[d-v-b]

at the moment, pydantic zarr is pretty low-level -- the v2.ArraySpec and v3.ArraySpec classes very closely model the array metadata documents. For zarr v2 this was fine, but for zarr v3 IMO creating array metadata is pretty tedious and so I do think we should introduce a higher-level API that exposes a simple shards=... keyword argument when creating an array.