Add icechunk / virtualizarr

[maxrjones]

Using the CI to test the answer to https://github.com/NASA-IMPACT/veda-analytics/issues/173#issuecomment-3155711155

[jsignell]

Nice! Is there a little test we can add to test-notebook.ipynb to make sure that reads work?

[jsignell]

I don't know who well the most recent versions of icechunk and virtulizarr will work with xarray 2025.04.0 either. I just ran into some errors working with the water-insights virtual icechunk store

[batpad]

Just checking in here: looking at @jsignell's comment, I guess we wait for more testing before merging this?

[jsignell]

Just checking in here: looking at @jsignell's comment, I guess we wait for more testing before merging this?

From looking at the logs (https://github.com/NASA-IMPACT/pangeo-notebook-veda-image/actions/runs/16758759036/job/47448196771?pr=42#step:8:226) it does look like the mamba update line bumps xarray and zarr versions so it might be ok. We'll just want to run a few notebooks and add an icechunk read to the existing test notebook.

[maxrjones]

Ideally we'd have the next release of zarr (v3.1.2, coming out this week), xarray (2025.08.0), and Icechunk (v1.1.3). Could we plan to merge on Monday?

[jsignell]

It looks like those releases all happened but didn't include zarr-developers/zarr-python#3369 or earth-mover/icechunk#1161.

[jsignell]

I just merged main to trigger an image build, but it looks like it didn't get the latest xarray and zarr-python. I am a little hesitant to spend too much time trying to get newer versions of xarray and zarr-python into this image that is based off of the pangeo image from 2025.06.02. A better approach would be to open a PR that adds icechunk and virtualizarr to the regular pangeo notebook image and then (after the PR is merged and tagged) create a specialized veda image based off of it.

[maxrjones]

I just merged main to trigger an image build, but it looks like it didn't get the latest xarray and zarr-python. I am a little hesitant to spend too much time trying to get newer versions of xarray and zarr-python into this image that is based off of the pangeo image from 2025.06.02. A better approach would be to open a PR that adds icechunk and virtualizarr to the regular pangeo notebook image and then (after the PR is merged and tagged) create a specialized veda image based off of it.

I just triggered https://github.com/pangeo-data/pangeo-docker-images/actions/runs/17268373278/job/49006047108?pr=602 (based on pangeo-data/pangeo-docker-images#602) to see if virtualizarr is still incompatible with the pangeo-ml notebook, which was the blocker for solving this upstream. But you could hopefully unblock that if it's still an issue and your preference is to solve this at the pangeo-docker-images level.

[jsignell]

As I was writing the test I realized that if we want virtualizarr>=2.0.0 then we need a newer base pangeo-notebook image. See #45 for that approach.

[jsignell]

I suspect that the conda update (which we switched to in #44) might be stricter about not updating underlying dependencies. This is a good thing because updating underlying dependencies in an existing env is error-prone.

[jsignell]

superseded by #45