FixedScaleOffset codec not preserved during zarr v2 to v3 conversion

[emmanuelmathot]

Problem

Input data with scale_factor/add_offset attributes (zarr v2 format) loses encoding during conversion, resulting in uint16 → float64 data type promotion.

Root Cause

• Original data in Zarr sample service has scale_factor: 0.0001, add_offset: -0.1, dtype: "<u2" (uint16)
• Conversion process doesn't detect these v2 attributes to create zarr v3 numcodecs.fixedscaleoffset codec
• Encoding propagation in create_measurements_encoding() overwrites with simple compressor, losing scale/offset configuration

Expected Behavior

Automatically convert zarr v2 scale/offset attributes to zarr v3 FixedScaleOffset codec:

numcodecs.zarr3.FixedScaleOffset(
    offset=-0.1,
    scale=10000,  # 1/0.0001
    dtype='uint16',
    astype='uint16'
)

Files Affected

• src/eopf_geozarr/s2_optimization/s2_multiscale.py (lines 319-329)
• src/eopf_geozarr/conversion/geozarr.py (encoding functions)

Impact

• Data type inflation (uint16 → float64)
• Loss of compression efficiency
• Incorrect data representation in output zarr v3 files

Notes

• for some reasons, xarray decodes the data properly, probably because it reads the attributes and apply the V2 conversion on V3.

cc @vincentsarago

[emmanuelmathot]

https://gitlab.eopf.copernicus.eu/cpm/eopf-cpm/-/issues/985

[emmanuelmathot]

zarr-developers/zarr-extensions#43

[emmanuelmathot]

@d-v-b I lack visibility and a sense of ETA for this one to be ready. I saw passing a couple of PRs and issues about the implementation. Could you update this ticket with the current status?

[d-v-b]

definitely, sorry for going dark. The rust implementation is ready for use (repo at https://github.com/zarr-developers/cast-value.rs/) and the python implementation that uses the rust implementation is also ready for use (https://github.com/zarr-developers/cast-value.py/). Both of these packages are up on their respective package indices (crates.io, pypi)

Today I'm going to wire this up into the data-model repo and see how data export looks with this codec. A note for consumers: they will not be able to read these zarr arrays without registering the cast_value codec. There is only a python implementation for now, so that limits consumption to python users until this codec is implemented in other languages. I am optimistic that we can compile the rust implementation to wasm and use that for a js implementation, but I have no experience there.

[emmanuelmathot]

For JS, the relevant discussion is here I think

[d-v-b]

yes I commented there. I'm sure I could figure out the rust -> wasm thing, but it would be great if someone on the JS side was there to pick this up