Writing kerchunk-generated parquet refs to icechunk repo

[rsignell]

I'm trying to use virtualizarr to create an icechunk repo, using a set of kerchunk-generated references (varr v2) stored in parquet on the Open Storage Network.

After some struggling (thanks @martindurant!), I've got this to work with Xarray and Zarr version 3:

import xarray as xr

endpoint_url='https://usgs.osn.mghpcc.org'
combined_parquet_aws = 's3://esip/rsignell/necofs.parquet'

ds = xr.open_dataset(f'reference://', engine='zarr', chunks={'time':100}, zarr_format=2, consolidated=False,
                     storage_options=dict(fo=combined_parquet_aws, remote_protocol='s3', lazy=True, asyncronous=True,
                                         target_options={'anon': True, 'endpoint_url':endpoint_url},              
                                         remote_options={'anon': True, 'endpoint_url':endpoint_url, 'asynchronous':True}))

When I try to use virtualizarr to write the data to icechunk:

ds.virtualize.to_icechunk(session.store)

I get this error, which seems related to Zarr v2, right?

TypeError: Expected a BytesBytesCodec. Got <class 'numcodecs.zstd.Zstd'> instead.

Is there a workaround for this?

Note: I could not create the references for these NetCDF files using Virtualizarr as it doesn't yet support subchunking, which is critical for this dataset.

[TomNicholas]

TypeError: Expected a BytesBytesCodec. Got <class 'numcodecs.zstd.Zstd'> instead.

This is similar to the error reported in #421, does @mpiannucci's solution from there fix your issue too? Otherwise it should be raised upstream I think.

Note: I could not create the references for these NetCDF files using Virtualizarr as it doesn't yet support subchunking, which is critical for this dataset.

Just to be sure you're referring to #183? Or to #199? Seems like the former given that the original data is compressed?

[rsignell]

Regarding the TypeErorr, I'm using numcodecs=0.15.1 and virtualizarr=1.3.1, which seems like what @mpiannucci's solution was using. So perhaps I'm doing something else wrong. Will compare with his solution more closely.

Regarding the subchunking, the source data is uncompresssed NetCDF 64-bit offset files, so #199 would address the issue. Is that in a state where I could try it out? I see you said local tests were passing last summer, but not sure if I should try it with all the changes that have occurred since then.

[mpiannucci]

You might need to update virtualizarr? Or zarr

[TomNicholas]

I see you said local tests were passing last summer, but not sure if I should try it with all the changes that have occurred since then.

Tthat virtualizarr branch requires an unmerged xarray branch (pydata/xarray#9286), which by now will be out-of-date :( You're welcome to try it but I have no idea if that will work until I get back to it.

[rsignell]

I have the latest virtualizarr (1.3.1) and zarr (3.0.4) from conda-forge, looks like this needs a review: conda-forge/virtualizarr-feedstock#6

[TomNicholas]

It does - I don't think the difference between virtualizarr 1.3.1 vs 1.3.2 is causing your problem though. It's much more likely an upstream issue in either zarr or numcodecs.

[rsignell]

Seems to be raised here also: pydata/xarray#10032