Skip to content

feat(icechunk): record credential-endpoint hints in the repository metadata - #2

Merged
betolink merged 1 commit into
mainfrom
feat/credential-hints
Nov 16, 2026
Merged

betolink merged 1 commit into
mainfrom
feat/credential-hints

Conversation

@betolink

Copy link
Copy Markdown
Member

What

When a store references s3:// virtual chunks, the writer now records where
to fetch credentials
in the icechunk repository metadata, under a canonical
key:

"earthaccess:virtual_chunk_credentials": {
  "version": 1,
  "containers": {
    "s3://podaac-ops-cumulus-protected/": {
      "provider": "POCLOUD",
      "collection_concept_id": "C1996881146-POCLOUD",
      "credentials_endpoint": "https://archive.podaac.earthdatacloud.nasa.gov/s3credentials",
      "region": "us-west-2"
    }
  }
}

Nothing secret is stored — only the endpoint and the collection identity. The
hint is keyed by the container's exact url_prefix (icechunk matches
authorization keys exactly, not by prefix) and namespaced so other icechunk
clients ignore it.

Why: a store opened by ea.open_virtual(asset.href) has no CMR result to derive
credentials from, so S3 chunks currently fall back to environment credentials.
With the hint, the reader can build a refreshable credential provider and
authorize the container — no provider=/credentials= needed from the caller.
(The reader-side change is a separate PR; this one is inert for older readers.)

How

  • credential_hints(granules, vds): derives one hint per s3://bucket/ prefix
    in the dataset from the source granules' metadata (granule, then collection),
    skipping HTTPS chunks and granules without an endpoint.
  • write_icechunk(..., credential_hints=...): merges the hints into
    repo.metadata after the commit. Merging means appends accumulate prefixes
    and unrelated metadata survives.
  • request() computes them automatically; no new option.

Tests

tests/test_credential_hints.py (8 cases): derivation, per-bucket matching,
missing endpoint, persistence, append merge (keeping unrelated metadata),
no-hint stores, and the plugin end to end.

Full suite: 76 passed, mypy clean, ruff clean, pre-commit 8/8.

Docs

README only (the canonical key and its shape). The walkthrough belongs in the
earthaccess notebooks, as a follow-up.

Follow-ups (not in this PR)

  • earthaccess open_virtual(): read the hint and build the credential mapping
    (also fixes a latent bug where raw S3Credentials values raise TypeError
    in icechunk 2.3 — they must be wrapped in credentials.Credentials).
  • idempotent ingestion (dedupe=, on_error=), stacked on this branch.

Stores that reference s3:// chunks now persist a non-secret hint of where to
fetch credentials, under the canonical key
`earthaccess:virtual_chunk_credentials`:

    {"version": 1, "containers": {"<url_prefix>": {
        "provider": ..., "collection_concept_id": ...,
        "credentials_endpoint": ..., "region": ...}}}

The hint is derived from the source granules' metadata (granule, then
collection) and keyed by the container's exact url_prefix, so readers can
authorize the container without knowing the provider. It is merged on append
and never replaces unrelated repository metadata. No secrets are stored.

Tests: tests/test_credential_hints.py (8 cases: derivation, per-bucket
matching, persistence, append merge, no-hint stores, plugin end to end).
@betolink
betolink merged commit 5ba78ab into main Nov 16, 2026
2 checks passed
@betolink
betolink deleted the feat/credential-hints branch November 16, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant