Sunset kerchunk and support all of virtualizarr parsers with a new top level API

[betolink]

Currently earthaccess supports creating virtual datasets via a really not used kerchunk API and via dmrpp with Virtualizarr. Kerchunk has been in many ways superseded by Virtualizarr, but since we only support dmrpp it limits ourselves to datasets with OPeNDAP support. I'll be working on a PR that will enable support for all formats, including the ones Kerchunk supports now.

At the end we'll have this to create a VDS:

import earthaccess as ea

granules = ea.search_data(**params)

def preprocess_metadata(ds):
    return ds

open_root_options = {
    "access": "direct",  # access to cloud data (faster in AWS with "direct")
    "load": False,  # Load coord indexes
    "group": "/" # root group 
    "concat_dim": "time",  # Concatenate files along the time dimension, grid aligned
    "preprocess": preprocess_metadata, # Normalize metadata if needed 
    "data_vars": "minimal",  # Only load data variables that include the concat_dim
    "coords": "minimal",  # Only load coordinate variables that include the concat_dim
    "compat": "override",  # Avoid coordinate conflicts by picking the first
    "combine_attrs": "override",  # Avoid attribute conflicts by picking the first
    "parallel": "dask" # Dask by default, we should support any parallel executor i.e. Lithops
    "parser": "dmrpp" # this will have a hierarchy, by default will be dmrpp, if not found will fallback to hdf5 
}

# here I'm proposing a top levep API change from open_virtual_mfdataset to just virtualize(granules, **kwargs)
vds = ea.virtualize(granules, **open_root_options)

# then we can persist the vds into one of the formats supported by virtualizarr
vds.virtualize.to_kerchunk(*args)
# or 
vds.virtualize.to_icechunk(*args)

Once we have this in place and Virtualizarr supports inline references we eventually should be able to just open a virtual data store and keep appending. This could be done using the vds URI or could be via Julius PR to integrate it with a collection level result.

colections = ea.search_datasets(
  short_name="SPL4SMGP"
)

vds = ea.open_virtual_store(colections[0]) # if the collection has the vds info of course

# then rinse and repeat
new_granules = ea.search_data(**params)
new_vds = ea.virtualize(new_granules, **open_root_options)

import xarray as xr
updated_vds = xr.concat([new_vds, vds])
updated_vds.virtualize.to_icechunk(*args, append_dim="time")

We still need zarr-developers/VirtualiZarr#794 (or zarr-developers/VirtualiZarr#938) to be merged

[betolink]

Update:

PR #1272 got merged and will be released tomorrow, it addresses the first part of this workflow, now earthaccess.virtualize() will enable the creation of virtual datasets with any of the supported virtualizarr parsers, if not provided will try dmrpp first then hdf5.

lithops worked as another way of parallelizing compute.

I think we can partially implement the second part, open_virtual_store() that should return a virtual dataset in an xarray ds, the one thing we don't know how will be eventually implemented is how this information will be listed in the collection record, there could be one or more virtual stores per collection, and this information could be in the related links or a different place. Let's assume we don't worry about this last part, the core of the method is to configure the I/O driver and virtual engine for xarray.

This method can for now take the URL of the virtual store and a collection record so it knows how to configure the auth credentials for each driver.

graph LR
    %% Input Section
    subgraph Parsers ["Input Parsers (Source Formats)"]
        direction TB
        A[GRIB]
        B[COGs]
        C[HDF5 / NetCDF4]
        D[DMRPP]
        E[Kerchunk JSON / Parquet]
        F[Icechunk]
    end

    %% Central Logic
    G{{"VirtualiZarr Engine<br/>(ManifestArray)"}}

    %% Output Section
    subgraph Manifests ["Serialized Formats (Target Manifests)"]
        direction TB
        H[Kerchunk JSON]
        I[Kerchunk Parquet]
        J[Icechunk]
    end

    %% Connections
    A --> G
    B --> G
    C --> G
    D --> G
    E --> G
    F --> G

    G --> H
    G --> I
    G --> J

    %% Styling with standardized, easily readable colors
    %% Replacing Magenta with a standard process Orange (#FF9F43)
    style G fill:#FF9F43,stroke:#333,stroke-width:2px,color:white
    
    %% Keeping the readable light blue storage classification
    classDef storage fill:#e1f5fe,stroke:#01579b,stroke-width:1px,color:#01579b
    class A,B,C,D,E,F,H,I,J storage

Loading

note: not sure if Virtualizarr can open Icechunk as a virtual dataset (chunk manifests)

IMO a good path will be using dmrpp as input and generating kerchunk parquet or icechunk for our virtual stores (for those collections that have dmrpp). If a collection has no dmrpp we can still virtualize() with the native parser.

dmrpp can represent any of the other archival formats and is faster to process than is to extract metadata from the source files.
There are some disadvantages that need to be addressed by OPeNDAP, e.g. standardize dmrpp across all collections, checksum files at dmrpp generation time, maybe a lighter schema or a different serialization (parquet).

So this will be the temporary method signature until we have a canonical way of finding this info in CMR.

# in region icechunk
vds = ea.open_virtual_store("s3://some-bucket/dataset.icechunk", collection_result)
# out of region kerchunk
vds = ea.open_virtual_store("https://some-url/dataset.parquet")

The one blocker was that vrirtualizarr embeds coordinates in the virtual store and the it doesn't know how to read it back as virtual, this is being addressed by zarr-developers/VirtualiZarr#938 in the meantime we can use the skip-coords trick zarr-developers/VirtualiZarr#489 (comment)

For Icechunk, the way to update it is described in here https://github.com/zarr-developers/VirtualiZarr/blob/main/examples/V1/append/noaa-cdr-sst.ipynb (although v1 the method still holds)