Skip to content

About

earthaccess plugin: virtualize granules, create and append icechunk / kerchunk-parquet virtual stores

Resources

Stars

3 stars

Watchers

0 watching

Forks

Repository files navigation

earthaccess-plugin-vds

Plugin preview. This package is part of the v2.0.0 plugin preview and only works with the plugins branch of earthaccess (earthaccess-dev/earthaccess@plugins). It is not on PyPI yet — see Installing and testing the drafts.

Virtual datasets for earthaccess: virtualize NASA granules into icechunk (default) or kerchunk-parquet stores, create new stores, and append to existing ones.

Built on earthaccess-plugin-spec.

Install

This package is not on PyPI yet. It is part of the v2.0.0 plugin preview and must be installed together with the plugins branch of earthaccess. The full steps (credentials, notebook deps, sanity check) are in Installing and testing the drafts.

git clone git@github.com:earthaccess-dev/earthaccess.git
cd earthaccess
git checkout plugins

uv venv
uv pip install -e ".[all]"
# add [kerchunk] and/or [distributed] if you need those extras
uv pip install "earthaccess-plugin-vds[kerchunk] @ git+https://github.com/earthaccess-dev/earthaccess-plugin-vds.git"

Usage

import earthaccess as ea

ea.login()
results = ea.search_data(
    short_name="MUR-JPL-L4-GLOB-v4.1",
    temporal=("2013-01-02", "2013-01-11"),
)

# Create a local icechunk store; the job returns one Asset.
job = ea.plugins.vds.request(results, store="mur_sst.icechunk", concat_dim="time")
asset = job.results()[0]
ds = ea.open(asset.href)

# Append: the same call. The store exists, so append_dim defaults to
# concat_dim and the write goes into the existing repository.
more = ea.search_data(
    short_name="MUR-JPL-L4-GLOB-v4.1",
    temporal=("2013-01-11", "2013-01-21"),
)
asset = ea.plugins.vds.request(
    more, store=asset.href, concat_dim="time"
).results()[0]

# No store → just a virtual dataset (manifest form).
vds = ea.plugins.vds.request(results, concat_dim="time").results()

# Kerchunk-parquet: a references *directory* named *.parquet. Appends write a
# versioned sibling (mur_refs.parquet → mur_refs_v2.parquet) unless
# replace=True, which rewrites in place.
asset = ea.plugins.vds.request(
    more, store="mur_refs.parquet", format="kerchunk", concat_dim="time"
).results()[0]

# Store-to-store merge (both formats): merge a source store into a target.
ea.plugins.vds.request(store="cube_v2", source="cube_v1", append_dim="time")

# Virtualize another plugin's *file* output (e.g. a Harmony subset):
# pass Assets or plain URLs. Derived files have no DMR++ sidecars, so the
# default parser switches to HDFParser. OPeNDAP assets (pydap) are rejected:
# a DAP response is not a byte-range-readable file.
assets = ea.plugins.harmony.request(results, variables=["analysed_sst"]).results()
asset = ea.plugins.vds.request(
    None, assets=assets, store="subset.icechunk", concat_dim="time"
).results()[0]

store= also accepts an obstore S3Store / LocalStore (or an icechunk.Storage together with href=) as a preconfigured client-side escape hatch. Credentials are never serialized: s3:// targets and granule reads resolve them from the environment where the code runs, so the same request also works on a Dask cluster whose workers carry the secrets.

Options

Option Meaning
store Target store: local path, s3:// URI, obstore store, icechunk.Storage, or icechunk.Repository.
href Canonical location for the Asset when store is a Storage/Repository (they carry no readable path).
prefix Repository prefix when store is an obstore S3Store.
region Region of the target S3 bucket (default us-west-2, where NASA S3 lives). A preconfigured Storage/S3Store takes precedence.
format "icechunk" (default) or "kerchunk".
source Existing store to merge into store.
assets File assets/URLs to virtualize instead of search results.
append_dim Append dimension; defaults to concat_dim when the store exists.
replace Kerchunk only: rewrite references in place (stable href).
commit Icechunk only: commit the write (default True).
repository_config Icechunk only: icechunk.RepositoryConfig applied when the repository is created (manifest splitting, inline threshold, ...). Ignored on append.
dedupe Icechunk only: "upsert" (default; skip unchanged, overwrite reprocessed, append new), "skip" (never overwrite) or "off".
key Coordinate identifying a slot; defaults to concat_dim. Only the time coordinate is derivable from CMR metadata.
on_error "raise" (default) or "skip": record a failed granule, ingest the rest, end the job PARTIAL.
verify "metadata" (default) or "etag": also probe each granule's origin ETag (one HTTP HEAD per granule) to catch in-place replacement without a CMR revision bump.
virtual_chunk_containers Icechunk only: extra VirtualChunkContainers keyed by url_prefix, merged over the derived ones (entries win).
access "indirect" (HTTPS, default) or "direct" (S3).
parser VirtualiZarr parser name; "DMRPPParser" default with automatic "HDFParser" fallback.
group, concat_dim, combine, preprocess, drop_variables, loadable_variables VirtualiZarr / xarray combine controls.
load Without store: return a concrete lazy Dataset (parquet round-trip). With store: rejected — use ea.open(asset.href, load=True).
tree Open a single granule as a DataTree (cannot be persisted).
show_progress Progress reporting (reserved for the distributed executor).

Outputs

Stores that reference s3:// chunks also record where to fetch credentials in the icechunk repository metadata, under earthaccess:virtual_chunk_credentials (container url_prefix, provider, collection concept-id and credentials endpoint — never secrets). ea.open_virtual(asset.href) reads that hint, so a reader does not need to know the provider; other icechunk clients ignore it.

Request job.output job.results()
store= given assets [Asset(href, access, auth="none", format="icechunk"|"kerchunk")]
no store, load=False virtual manifest-form xarray.Dataset / DataTree
no store, load=True xarray concrete lazily-loaded xarray.Dataset

Idempotent ingestion

request() is idempotent by default: each granule is keyed by its time step and the ingested keys are recorded in the repository metadata. Re-running a batch skips unchanged granules, overwrites a reprocessed one in place (a CMR revision change is the signal), and appends only genuinely new steps. dedupe="skip" never overwrites; dedupe="off" restores plain appends. With on_error="skip" a failed granule is recorded, the rest are ingested, and the job ends PARTIAL — see job.info()["summary"] and job.info()["errors"].

Regions

NASA Earthdata S3 buckets are only reachable from us-west-2, so access="direct" needs the workers there; HTTPS (access="indirect", the default) works from anywhere. The icechunk write happens on the client and only needs the target bucket's region: region= (default us-west-2) for the s3:// shorthand, or the bucket's region in a preconfigured icechunk.Storage / obstore S3Store. Whether a process actually runs in us-west-2 is not detected (instance metadata is not available everywhere), so co-locate the cluster when using direct access.

Development

Requires Python 3.12+ (icechunk, virtualizarr and earthaccess all do).

pip install -e ".[test,kerchunk]"
pytest
ruff check .
mypy

The test suite is offline: the granule → virtual-dataset step is faked, while create/append/reopen run against real local icechunk repositories and kerchunk-parquet directories.

License

MIT.

About

earthaccess plugin: virtualize granules, create and append icechunk / kerchunk-parquet virtual stores

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages