Really great work here! I'd like to add a few comments on chunk layout based on my experience creating global AEF mosaics with Zarr v3 (code here):
- Generally speaking I think it's best to use as few Zarr arrays as possible to describe the data. You can't really get around the one group per UTM zone, as GeoZarr requires this. However you could combine all time steps (years) into a single zarr array under each UTM zone. This would require adding an additional
time coordinate variable and updating the chunking of the array from (H, W, B) to (H, W, B, T). I like this approach because accessing embeddings across multiple years becomes simple numpy array slicing; instead of requiring the user to access each time step in a distinct zarr array. The chunking on the time dimension allows you to append more time steps to the array as embeddings are generated for future years. This approach does make it much harder to pre-pend embeddings (ex. computing earlier time steps).
- It's more standard to have the time variable BEFORE the spatial ones. For example
(T, B, H, W) instead of (H, W, B, T).
x and y is more common over easting and northing.- There is no convention here, but personally I prefer
dimension over band when naming that coordinate array.
- A 4x4 inner chunk is extremely small. Personally I'd recommend increasing that to at least 32x32 or 64x64, while also increasing the shard size to match. You really want the biggest shards possible as it helps reduce the number of files written. For AEF mosaics I use 256x256 inner chunks with 4096x4096 shards (16x16 inner chunks). The combination of small-medium chunk size + large shard size is best for balancing between visualization (low latency) and analytical (high throughput) use cases. This works well because optimized readers like
zarr-python know it doesn't need to download the entire shard just to access a handful of chunks. The current chunking/sharding scheme is great for visualization, I fear it will be inefficient for large scale analysis.
Really great work here! I'd like to add a few comments on chunk layout based on my experience creating global AEF mosaics with Zarr v3 (code here):
timecoordinate variable and updating the chunking of the array from(H, W, B)to(H, W, B, T). I like this approach because accessing embeddings across multiple years becomes simple numpy array slicing; instead of requiring the user to access each time step in a distinct zarr array. The chunking on the time dimension allows you to append more time steps to the array as embeddings are generated for future years. This approach does make it much harder to pre-pend embeddings (ex. computing earlier time steps).(T, B, H, W)instead of(H, W, B, T).xandyis more common overeastingandnorthing.dimensionoverbandwhen naming that coordinate array.zarr-pythonknow it doesn't need to download the entire shard just to access a handful of chunks. The current chunking/sharding scheme is great for visualization, I fear it will be inefficient for large scale analysis.