Option to preserve child metadata during consolidation at parent level

[emmanuelmathot]

Summary

The current consolidate_metadata() function clears child group metadata when consolidating at the parent level. This behavior prevents hierarchical metadata organization where both parent and child groups need consolidated metadata for different access patterns. We propose adding an option to preserve existing child consolidated metadata during parent-level consolidation.

Background

When working with complex hierarchical Zarr stores (like new EOPF Sentinel), there's a need for consolidated metadata at multiple levels:

1. Root level consolidation: For accessing the entire store structure
2. Group level consolidation: For efficient access to specific resolution groups or data subsets

Currently, consolidating metadata at the parent level removes any existing consolidated metadata from child groups, forcing users to choose between parent-level or child-level consolidation but not both.

Use Case: EOPF Sentinel

Data Structure

store.zarr/
├── measurements/
│   └── reflectance/
│       ├── r10m/           # 10m resolution group
│       │   ├── 0/          # Native resolution
│       │   ├── 1/          # Overview level 1
│       │   └── 2/          # Overview level 2
│       ├── r20m/           # 20m resolution group
│       │   ├── 0/
│       │   ├── 1/
│       │   └── 2/
│       └── r60m/           # 60m resolution group
│           ├── 0/
│           ├── 1/
│           └── 2/
└── quality/
    └── l1c_quicklook/
        └── r10m/
            ├── 0/
            ├── 1/
            └── 2/

Required Metadata Consolidation Levels

1. Resolution Group Level (r10m/, r20m/, r60m/):

   • Needed for efficient access to all overview levels within a resolution
   • Enables fast enumeration of available zoom levels
   • Required for GeoZarr multiscales metadata organization

2. Root Level (store.zarr/):

   • Needed for discovering all available resolution groups
   • Enables efficient store-wide operations
   • Required for complete store metadata access

Current Problem

import zarr

# Step 1: Consolidate at resolution group level (works fine)
zarr.consolidate_metadata("store.zarr/measurements/reflectance/r10m")
zarr.consolidate_metadata("store.zarr/measurements/reflectance/r20m") 
zarr.consolidate_metadata("store.zarr/measurements/reflectance/r60m")

# Step 2: Consolidate at root level (removes child consolidated metadata!)
zarr.consolidate_metadata("store.zarr")

# Result: Resolution groups no longer have consolidated metadata
# This breaks efficient access patterns for resolution-specific operations

Current Workaround

We've had to implement a custom consolidation function that manually preserves child metadata by removing the lines

zarr-python/src/zarr/api/asynchronous.py

Lines 220 to 225 in 9dc744d

	# While consolidating, we want to be explicit about when child groups
	# are empty by inserting an empty dict for consolidated_metadata.metadata
	for k, v in members_metadata.items():
	if isinstance(v, GroupMetadata) and v.consolidated_metadata is None:
	v = dataclasses.replace(v, consolidated_metadata=ConsolidatedMetadata(metadata={}))
	members_metadata[k] = v

Proposed Solution

Add an optional parameter to consolidate_metadata() to preserve existing child consolidated metadata:

def consolidate_metadata(
    store: StoreLike,
    path: str | None = None,
    zarr_format: ZarrFormat | None = None,
    preserve_child: bool = False  # New parameter
) -> Group:
    """
    Consolidate the metadata of all nodes in a hierarchy.
    
    Parameters
    ----------
    preserve_child_consolidated : bool, default False
        If True, preserve existing consolidated metadata in child groups
        instead of clearing it during parent consolidation.
    """

Behavior with preserve_child=True:

1. Collect metadata normally from all descendants
2. Check for existing consolidated metadata in child groups
3. Preserve child consolidated metadata in the consolidated structure
4. Allow both parent and child to have consolidated metadata simultaneously

Example Usage:

# Step 1: Consolidate at resolution group level
zarr.consolidate_metadata("store.zarr/measurements/reflectance/r10m")
zarr.consolidate_metadata("store.zarr/measurements/reflectance/r20m")
zarr.consolidate_metadata("store.zarr/measurements/reflectance/r60m")

# Step 2: Consolidate at root level while preserving child metadata
zarr.consolidate_metadata("store.zarr", preserve_child_consolidated=True)

# Result: Both root and resolution groups have consolidated metadata!

Benefits

1. Hierarchical Access Patterns: Enables efficient access at multiple levels
2. Backward Compatibility: Default behavior unchanged (preserve_child=False)
3. Complex Data Structures: Better support for scientific and Earth observation data
4. Performance: Avoids need to re-consolidate child groups after parent consolidation
5. Standards Compliance: Enables better compliance with specifications like GeoZarr that expect metadata at multiple levels

Questions for Maintainers

1. Conceptual Design: Is the current behavior of clearing child metadata an intentional design choice, or could it be enhanced?

2. Implementation Approach: Would you prefer:

   • A new parameter to existing function (as proposed above)
   • A separate function (e.g., consolidate_metadata_hierarchical())
   • A different approach entirely?

3. Metadata Structure: How should the consolidated metadata structure represent preserved child consolidated metadata? Should it:

   • Keep child consolidated metadata as-is in their respective paths?
   • Include references to child consolidated metadata in parent metadata?
   • Use a different organizational approach?

4. Performance Considerations: Are there performance implications we should consider when preserving child metadata during parent consolidation?

related spec issue: zarr-developers/zarr-specs#309

cc @TomAugspurger @maxrjones @d-v-b @vincentsarago

[d-v-b]

it might also be useful to use children's consolidated metadata when building the parent's consolidated metadata. This would save some IO, at the risk of propagating errors about the state of the hierarchy. So maybe we need two independent options at consolidated metadata creation time -- whether lower-level consolidated metadata should be erased, and whether lower-level consolidated metadata should be used as a proxy for listing the contents of that section of the hierarchy

[TomAugspurger]

Can you share an example that shows

1. The on-disk zarr.json for the root and child groups (desired and actual)
2. The in-memory group.metadata after loading (desired and actual)

In particular, I'm wondering whether the issue is that zarr.consolidate_metadata clears existing child metadata on disk, or whether the child consolidated metadata is just not present in the .metadata. I hope that .consolidate_metadata() isn't changing keys other than the one it's supposed to.

[TomAugspurger]

@emmanuelmathot this is the behavior on main. Can you confirm that this looks like what you want?

import zarr

def main():
    store = zarr.storage.MemoryStore()
    root = zarr.create_group(store, path="", overwrite=True)
    measurements = root.create_group("measurements", overwrite=True)
    reflectance = measurements.create_group("reflectance", overwrite=True)
    r10 = reflectance.create_group("r10", overwrite=True)
    r10.create_array("l0", shape=(10, 10), dtype="float32", overwrite=True)
    r10.create_array("l1", shape=(10, 10), dtype="float32", overwrite=True)

    r20 = reflectance.create_group("r20", overwrite=True)
    r20.create_array("l0", shape=(10, 10), dtype="float32", overwrite=True)
    r20.create_array("l1", shape=(10, 10), dtype="float32", overwrite=True)

    r60 = reflectance.create_group("r60", overwrite=True)
    r60.create_array("l0", shape=(10, 10), dtype="float32", overwrite=True)
    r60.create_array("l1", shape=(10, 10), dtype="float32", overwrite=True)

    quality = root.create_group("quality")
    l1c = quality.create_group("l1c", overwrite=True)
    r10m = l1c.create_group("r10")
    r10m.create_array("l0", shape=(10, 10), dtype="float32", overwrite=True)
    r10m.create_array("l1", shape=(10, 10), dtype="float32", overwrite=True)

    for path in [
        "",
        "measurements",
        "measurements/reflectance",
        "measurements/reflectance/r10",
        "measurements/reflectance/r20",
        "measurements/reflectance/r60",
    ]:
        zarr.consolidate_metadata(store, path=path)


    print("root")
    print(zarr.open_consolidated(store, path="").metadata.consolidated_metadata)

    print("\nmeasurements")
    print(zarr.open_consolidated(store, path="measurements").metadata.consolidated_metadata)

    print("\nmeasurements/reflectance/r10")
    print(zarr.open_consolidated(store, path="measurements/reflectance/r10").metadata.consolidated_metadata)

if __name__ == "__main__":
    main()

that outputs

root
ConsolidatedMetadata(metadata={'measurements': GroupMetadata(attributes={}, zarr_format=3, consolidated_metadata=ConsolidatedMetadata(metadata={'reflectance': GroupMetadata(attributes={}, zarr_format=3, consolidated_metadata=ConsolidatedMetadata(metadata={'r60': GroupMetadata(attributes={}, zarr_format=3, consolidated_metadata=ConsolidatedMetadata(metadata={'l0': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=()), 'l1': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=())}, kind='inline', must_understand=False), node_type='group'), 'r20': GroupMetadata(attributes={}, zarr_format=3, consolidated_metadata=ConsolidatedMetadata(metadata={'l0': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=()), 'l1': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=())}, kind='inline', must_understand=False), node_type='group'), 'r10': GroupMetadata(attributes={}, zarr_format=3, consolidated_metadata=ConsolidatedMetadata(metadata={'l0': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=()), 'l1': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=())}, kind='inline', must_understand=False), node_type='group')}, kind='inline', must_understand=False), node_type='group')}, kind='inline', must_understand=False), node_type='group'), 'quality': GroupMetadata(attributes={}, zarr_format=3, consolidated_metadata=ConsolidatedMetadata(metadata={'l1c': GroupMetadata(attributes={}, zarr_format=3, consolidated_metadata=ConsolidatedMetadata(metadata={'r10': GroupMetadata(attributes={}, zarr_format=3, consolidated_metadata=ConsolidatedMetadata(metadata={'l0': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=()), 'l1': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=())}, kind='inline', must_understand=False), node_type='group')}, kind='inline', must_understand=False), node_type='group')}, kind='inline', must_understand=False), node_type='group')}, kind='inline', must_understand=False)

measurements
ConsolidatedMetadata(metadata={'reflectance': GroupMetadata(attributes={}, zarr_format=3, consolidated_metadata=ConsolidatedMetadata(metadata={'r60': GroupMetadata(attributes={}, zarr_format=3, consolidated_metadata=ConsolidatedMetadata(metadata={'l0': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=()), 'l1': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=())}, kind='inline', must_understand=False), node_type='group'), 'r20': GroupMetadata(attributes={}, zarr_format=3, consolidated_metadata=ConsolidatedMetadata(metadata={'l0': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=()), 'l1': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=())}, kind='inline', must_understand=False), node_type='group'), 'r10': GroupMetadata(attributes={}, zarr_format=3, consolidated_metadata=ConsolidatedMetadata(metadata={'l0': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=()), 'l1': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=())}, kind='inline', must_understand=False), node_type='group')}, kind='inline', must_understand=False), node_type='group')}, kind='inline', must_understand=False)

measurements/reflectance/r10
ConsolidatedMetadata(metadata={'l0': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=()), 'l1': ArrayV3Metadata(shape=(10, 10), data_type=Float32(endianness='little'), chunk_grid=RegularChunkGrid(chunk_shape=(10, 10)), chunk_key_encoding=DefaultChunkKeyEncoding(name='default', separator='/'), fill_value=np.float32(0.0), codecs=(BytesCodec(endian=<Endian.little: 'little'>), ZstdCodec(level=0, checksum=False)), attributes={}, dimension_names=None, zarr_format=3, node_type='array', storage_transformers=())}, kind='inline', must_understand=False)

which I think shows consolidated metadata at multiple levels in the hierarchy.

[emmanuelmathot]

Great @TomAugspurger! I now better understand the path parameter. Can you confirm that consolidate_metadata functions as an opt-in for the specified path? Other groups are opt-out by default and thus have their metadata cleared.

[TomAugspurger]

Can you confirm that consolidate_metadata functions as an opt-in for the specified path? Other groups are opt-out by default and thus have their metadata cleared.

I'm not sure I follow sorry. Maybe you could rephrase / show an example of what you're looking for?

When you consolidate_metadata at a path, then zarr-python will scan it and all of its children, and consolidate all their metadata up at path. If you load the data at path then zarr-python will be done doing I/O for any metadata operations: it already knows all the metadata of path/child-group/child-array thanks to consolidated metadata, and so doesn't need to read the zarr.json at path/child-group/child-array.

Consolidating the metadata at path doesn't (or at least shouldn't) affect the metadata written to zarr.json at path/child-group or path/child-group/child-array.

[emmanuelmathot]

Based on comprehensive testing including xarray integration, the zarr consolidation issue described in this issue cannot be reproduced with the current zarr-python development version. The current implementation already preserves child consolidated metadata during parent-level consolidation, supporting the hierarchical access patterns required for complex data structures like EOPF Sentinel.

However, there may be xarray-specific issues in how consolidated metadata is accessed, particularly when xarray looks for consolidated metadata at the root level when accessing child groups. This could be the source of the original issue report.

The original problem description may have been based on a misunderstanding of the consolidation behavior or a misuse of cobining xarray to_zarr() and zarr native read/write.

[TomAugspurger]

Sounds good. Feel free to ping me on any xarray issues that may be filed.