Add tutorial notebook for `open_virtual_dataset`

[ayushnag]

The newly added open_virtual_dataset and open_virtual_mfdataset functions need a tutorial notebook to show example usage. The current sections I have planned are:

1. Opening a dataset
2. Opening a virtual dataset, saving the reference file, and loading the dataset with xarray
3. Opening a virtual dataset with groups
4. Opening a virtual dataset with preprocessing

cc @betolink @TomNicholas @danielfromearth

[betolink]

Thanks for opening this @ayushnag!

[danielfromearth]

I just tried running this new functionality in an Openscapes Jupyterhub instance. Used !pip install . --upgrade to install earthaccess directly from the main branch (after running git pull).

A ModuleNotFoundError: No module named 'virtualizarr' error was raised when executing the earthaccess.open_virtual_mfdataset() function. Does virtualizarr need to be added to the pyproject.toml?

[danielfromearth]

Oh! I see now that it is declared in the optional-dependencies section of the pyproject.toml. So, I should be able to get past that error now. Do we want to move it into the main dependencies though?

[ayushnag]

I think doing pip install "earthaccess[virtualizarr]" will work once this code is in the release version. For now I believe the method is pip install 'git+https://github.com/nsidc/earthaccess.git@main#egg=earthaccess[virtualizarr]'

[chuckwondo]

You should be able to do pip install .[virtualizarr]

[danielfromearth]

After installing via pip install .[virtualizarr], I get a few couple messages after that. Thoughts on the following?

1) A numpy version warning upon import of xarray (though perhaps this is simply a result of pip installing earthaccess into the base environment for Openscapes' Jupyterhub?):

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.2.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

2. A ValueError during an HDF import statement in virtualizarr:

granules = earthaccess.search_data(
    short_name="TEMPO_NO2_L2",
    version="V03",
    count=3
)

result = earthaccess.open_virtual_mfdataset(
    granules=granules, 
    access="indirect", 
    concat_dim="time",  
    parallel=False, 
    preprocess=None
)

...raises the following (note: I truncated the beginning of the callback, which starts with File [~/earthaccess/earthaccess/dmrpp_zarr.py:91], in open_virtual_mfdataset(granules, group, access, load, preprocess, parallel, **xr_combine_nested_kwargs))...

File [/srv/conda/envs/notebook/lib/python3.10/site-packages/virtualizarr/readers/hdf/hdf.py:20](https://openscapes.2i2c.cloud/srv/conda/envs/notebook/lib/python3.10/site-packages/virtualizarr/readers/hdf/hdf.py#line=19)
     14 from virtualizarr.manifests.manifest import validate_and_normalize_path_to_uri
     15 from virtualizarr.readers.common import (
     16     VirtualBackend,
     17     construct_virtual_dataset,
     18     open_loadable_vars_and_indexes,
     19 )
---> 20 from virtualizarr.readers.hdf.filters import cfcodec_from_dataset, codecs_from_dataset
     21 from virtualizarr.types import ChunkKey
     22 from virtualizarr.utils import _FsspecFSFromFilepath, check_for_collisions, soft_import

File [/srv/conda/envs/notebook/lib/python3.10/site-packages/virtualizarr/readers/hdf/filters.py:16](https://openscapes.2i2c.cloud/srv/conda/envs/notebook/lib/python3.10/site-packages/virtualizarr/readers/hdf/filters.py#line=15)
     13     import h5py  # type: ignore
     14     from h5py import Dataset, Group  # type: ignore
---> 16 h5py = soft_import("h5py", "For reading hdf files", strict=False)
     17 if h5py:
     18     Dataset = h5py.Dataset

File [/srv/conda/envs/notebook/lib/python3.10/site-packages/virtualizarr/utils.py:94](https://openscapes.2i2c.cloud/srv/conda/envs/notebook/lib/python3.10/site-packages/virtualizarr/utils.py#line=93), in soft_import(name, reason, strict)
     92 def soft_import(name: str, reason: str, strict: Optional[bool] = True):
     93     try:
---> 94         return importlib.import_module(name)
     95     except (ImportError, ModuleNotFoundError):
     96         if strict:

File [/srv/conda/envs/notebook/lib/python3.10/importlib/__init__.py:126](https://openscapes.2i2c.cloud/srv/conda/envs/notebook/lib/python3.10/importlib/__init__.py#line=125), in import_module(name, package)
    124             break
    125         level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)

File [/srv/conda/envs/notebook/lib/python3.10/site-packages/h5py/__init__.py:25](https://openscapes.2i2c.cloud/srv/conda/envs/notebook/lib/python3.10/site-packages/h5py/__init__.py#line=24)
     19 # --- Library setup -----------------------------------------------------------
     20 
     21 # When importing from the root of the unpacked tarball or git checkout,
     22 # Python sees the "h5py" source directory and tries to load it, which fails.
     23 # We tried working around this by using "package_dir" but that breaks Cython.
     24 try:
---> 25     from . import _errors
     26 except ImportError:
     27     import os.path as _op

File h5py[/_errors.pyx:1](https://openscapes.2i2c.cloud/_errors.pyx#line=0), in init h5py._errors()

ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

[chuckwondo]

though perhaps this is simply a result of pip installing earthaccess into the base environment for Openscapes' Jupyterhub?

Definitely could be problematic, so try creating a new env instead, and see if you get the same problems.

[TomNicholas]

@danielfromearth (1) and (2) are related - VirtualiZarr has a hard requirement to use numpy 2.0.0 or later because it makes a lot of use of the new variable-length string style internally. I think something in your environment is not compiled with numpy 2.

[battistowx]

@danielfromearth I'm currently using hatch and the pyproject.toml to experiment on my local desktop, I would avoid using the 2i2c Jupyterhub for now since it has dependency locks

[danielfromearth]

@battistowx, I've also tried it locally without issue, but part of my goal with testing was to try it in-region on AWS West 2. I'm still not sure how to modify the environment in 2i2c Jupyterhub, and if there are dependency locks, is there a way around that for testing purposes?

[betolink]

@danielfromearth great timing!! I was testing the latest image for the hub, try restarting your instance in the admin console and use openscapes/python:07980b9 we still need to reinstall earthaccess from source because the virtualizar work has not been released. We should release it today or Monday!

[danielfromearth]

Okay, just tried this in an instance of openscapes/python:07980b9. When running this same code as in my previous comment:

result = earthaccess.open_virtual_mfdataset(
    granules=results, 
    access="indirect", 
    concat_dim="time",  
    parallel=False, 
    preprocess=None
)

The following new error is being raised by xarray:

TypeError: Could not find a Chunk Manager which recognises type <class 'virtualizarr.manifests.array.ManifestArray'>

This is out of my depth. Is it missing some optional dependency, something that provides a "Chunk Manager" that will know how to handle ManifestArrays?