Repository navigation
feat(icechunk): record credential-endpoint hints in the repository metadata - #2
Merged
Merged
Conversation
Stores that reference s3:// chunks now persist a non-secret hint of where to
fetch credentials, under the canonical key
`earthaccess:virtual_chunk_credentials`:
{"version": 1, "containers": {"<url_prefix>": {
"provider": ..., "collection_concept_id": ...,
"credentials_endpoint": ..., "region": ...}}}
The hint is derived from the source granules' metadata (granule, then
collection) and keyed by the container's exact url_prefix, so readers can
authorize the container without knowing the provider. It is merged on append
and never replaces unrelated repository metadata. No secrets are stored.
Tests: tests/test_credential_hints.py (8 cases: derivation, per-bucket
matching, persistence, append merge, no-hint stores, plugin end to end).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When a store references
s3://virtual chunks, the writer now records whereto fetch credentials in the icechunk repository metadata, under a canonical
key:
Nothing secret is stored — only the endpoint and the collection identity. The
hint is keyed by the container's exact
url_prefix(icechunk matchesauthorization keys exactly, not by prefix) and namespaced so other icechunk
clients ignore it.
Why: a store opened by
ea.open_virtual(asset.href)has no CMR result to derivecredentials from, so S3 chunks currently fall back to environment credentials.
With the hint, the reader can build a refreshable credential provider and
authorize the container — no
provider=/credentials=needed from the caller.(The reader-side change is a separate PR; this one is inert for older readers.)
How
credential_hints(granules, vds): derives one hint pers3://bucket/prefixin the dataset from the source granules' metadata (granule, then collection),
skipping HTTPS chunks and granules without an endpoint.
write_icechunk(..., credential_hints=...): merges the hints intorepo.metadataafter the commit. Merging means appends accumulate prefixesand unrelated metadata survives.
request()computes them automatically; no new option.Tests
tests/test_credential_hints.py(8 cases): derivation, per-bucket matching,missing endpoint, persistence, append merge (keeping unrelated metadata),
no-hint stores, and the plugin end to end.
Full suite: 76 passed, mypy clean, ruff clean, pre-commit 8/8.
Docs
README only (the canonical key and its shape). The walkthrough belongs in the
earthaccess notebooks, as a follow-up.
Follow-ups (not in this PR)
open_virtual(): read the hint and build the credential mapping(also fixes a latent bug where raw
S3Credentialsvalues raiseTypeErrorin icechunk 2.3 — they must be wrapped in
credentials.Credentials).dedupe=,on_error=), stacked on this branch.