Write CF-compliant Zarr output for interoperability with external analysis tools

[glwagner]

Motivation

ZarrWriter should produce a self-describing dataset that follows applicable CF conventions and can be consumed directly by external analysis and visualization tools.

xarray is one important interoperability target and a useful regression test, but the goal is not to implement an xarray-specific API. The goal is to emit conventional Zarr metadata and coordinate variables that can be understood by CF-aware tools across the Zarr ecosystem.

Current behavior

Ordinary full-dimensional fields written by the current ZarrWriter can be opened with xarray.open_zarr(path, consolidated=false). This works because the arrays carry the Zarr-v2 _ARRAY_DIMENSIONS metadata in the expected raw/C order. However:

• only time is exposed as a coordinate; spatial coordinate variables are not written at the root;
• reduced fields retain singleton axes whose dimension names may be empty strings. Multiple reduced axes produce duplicate dimension names and break labeled operations in xarray;
• stretched, immersed-boundary, and orthogonal spherical-shell grids cannot be represented adequately without writing their coordinates and/or auxiliary grid data;
• multidimensional longitude/latitude coordinates need CF coordinates metadata in addition to, rather than instead of, the array's actual dimension metadata;
• Zarr and NetCDF output currently duplicate substantial dimension, coordinate, metric, and grid-reconstruction logic.

Thus the current output is loadable in a limited case, but it is not consistently self-describing or interoperable.

Proposed requirements

• Every Zarr array has valid, unique dimension metadata.
  • For Zarr v2, preserve _ARRAY_DIMENSIONS with the ordering expected by Zarr readers.
  • Omit/squeeze dimensions reduced by a Nothing location or Flat topology.
• Write one-dimensional spatial coordinate variables at the root, named consistently with their dimensions and carrying appropriate CF attributes such as units and long_name.
• Represent curvilinear grids with logical dimensions plus multidimensional auxiliary coordinates.
  • For example, orthogonal spherical-shell fields should have their actual logical dimensions in _ARRAY_DIMENSIONS.
  • Their longitude/latitude variables should be referenced by a space-delimited CF coordinates attribute.
• Support multiple output grids with consistent, collision-free dimension and coordinate names.
• Define a consistent CF-compatible time representation, including coverage for non-numeric clocks such as DateTime.
• Preserve Oceananigans-native FieldTimeSeries reconstruction and round trips.
• Share format-independent schema construction with NetCDFWriter rather than copying its implementation. Candidate shared functionality includes:
  • dimension and coordinate discovery;
  • reduced-dimension squeezing;
  • default coordinate/output attributes;
  • grid metrics and immersed-boundary data;
  • unique-grid mapping and naming;
  • grid reconstruction metadata.
• Keep only storage mechanics backend-specific: NetCDF defDim/defVar, and Zarr array creation, chunking, compression, resizing/appending, and metadata encoding.

Scientific coordinate arrays needed by external readers should be visible in the root dataset. Private metadata used only to reconstruct Oceananigans grids may remain in grid/ subgroups.

Validation

Add an external interoperability test that writes stores with Julia and verifies that a standards-aware reader such as xarray can open them without manual metadata repair. Representative cases should include:

• RectilinearGrid;
• LatitudeLongitudeGrid;
• reduced fields and Flat topologies;
• OrthogonalSphericalShellGrid;
• immersed-boundary grids where applicable;
• multiple grids in one writer;
• numeric and date/time clocks.

The test should check dimensions, coordinate values, data values, and basic labeled operations. This can remain an external/integration test; ZarrWriter itself should not depend on Python or xarray.

Related work

• Support ImmersedBoundaryGrid for reading zarr output #5639 notes that grid coordinates are not currently written by ZarrWriter.
• Write grid to zarr output #5718 attempts to add grid output, but should be reworked around shared schema logic and must preserve valid Zarr dimension metadata.
• Make Zarr output CF interoperable; expand tests and benchmarks #5610 adds Zarr writer/read benchmarks and tests.
• Add FieldTimeSeries round-trip test harness across writers and grids #5654 adds cross-writer FieldTimeSeries round-trip tests.
• Integrating Oceananigans with Python-xgcm #1334 contains earlier discussion of CF-based interoperability with Python/xgcm for NetCDF output.

[glwagner]

cc @awiteck

[glwagner]

Implementation is now in #5610 at 75f31e479.

The PR writes CF-oriented root coordinates and valid dimension metadata, handles curvilinear auxiliary coordinates and reduced fields, consolidates metadata after appends, preserves Oceananigans round trips, and moves the shared schema/reconstruction logic used by both Zarr and NetCDF into src/OutputWriters.

External validation passes on xarray with both zarr-python 2.18.7 and 3.3.0. #5610 includes Closes #5876, so this issue will close when the implementation merges.

[glwagner]

Final PR head is now b5119eb9c; the follow-up preserves safe decoding of signed infinite numeric metadata (-Inf/+Inf) without reintroducing evaluation.