Skip to content

feat(icechunk): repository_config, Repository targets, and VCC overrides - #1

Merged
betolink merged 2 commits into
mainfrom
feat/icechunk-config
Nov 16, 2026
Merged

betolink merged 2 commits into
mainfrom
feat/icechunk-config

Conversation

@betolink

Copy link
Copy Markdown
Member

What

Three additive escape hatches for callers that need control over the icechunk
repository, plus a wiring fix for the documented href=/prefix= options.

All of them default to today's behaviour, so existing calls are unchanged.

repository_config=

An icechunk.RepositoryConfig applied when the repository is created —
manifest splitting/preload, inline-chunk threshold, compression, concurrency.

config = icechunk.RepositoryConfig.default()
config.manifest = icechunk.ManifestConfig(
    splitting=icechunk.ManifestSplittingConfig.from_dict(
        {icechunk.ManifestSplitCondition.name_matches("analysed_sst"): {
            icechunk.ManifestSplitDimCondition.DimensionName("time"): 32}}
    )
)
ea.plugins.vds.request(granules, store="mur.icechunk", concat_dim="time",
                       repository_config=config)
  • On append the stored configuration wins; passing it emits a UserWarning
    instead of silently doing nothing.
  • The caller's object is never mutated (RepositoryConfig.merge() returns a new
    config).

store=<icechunk.Repository> (+ href=)

Write into a repository you created yourself; the plugin never creates or
reconfigures it.

  • Emptiness is detected with the first-commit sentinel
    (1CECHNKREP0F1RSTCMT0), so an empty repository takes create semantics
    instead of failing with GroupNotFoundError (which is what happens today
    with a pre-created repo).
  • href= is required because a Repository carries no readable path.
  • repository_config= together with a Repository is rejected (the repository
    already owns its configuration).

virtual_chunk_containers=

Extra VirtualChunkContainer store configs keyed by url_prefix, merged over
the ones derived from the dataset's chunk references (entries win), on create
and append:

virtual_chunk_containers={
    "s3://my-private-bucket/": icechunk.s3_store(region="us-east-1"),
    "https://data.example.com/": icechunk.http_store(headers={"Authorization": "Bearer …"}),
}

Appends persist the merge via repo.reopen(config=…) + save_config() — mutating
repo.config is a no-op copy in icechunk 2.3.

Fix: href= / prefix= were not wired

The Options docstring advertised store=<icechunk.Storage> + href=, but
neither href nor prefix was forwarded to resolve_icechunk_target, so that
escape hatch raised ValueError at write time. They are now forwarded (and
covered by a test).

Tests

  • tests/test_icechunk_config.py: 12 cases (create with config, caller object
    untouched, append warning + stored config wins, Repository empty → create,
    Repository with data → append, missing href, Repository+config rejection,
    Storage+href, VCC merge/override on create and append).
  • Full suite: 68 passed, mypy clean (16 files), ruff clean, pre-commit 8/8.

Docs

README only (options table rows) — the detailed walkthrough goes in the
earthaccess notebooks as a separate follow-up.

Not in this PR

  • No change to ea.open_virtual()/read-side authorization: readers already
    accept authorize_virtual_chunk_access= and the derived VCC mapping.

Three additive escape hatches for callers that need control over the
icechunk repository, all defaulting to today's behaviour:

- repository_config=: an icechunk.RepositoryConfig applied when the repo is
  created (manifest splitting/preload, inline threshold, compression, ...).
  Ignored with a UserWarning on append, where the stored config wins. The
  caller's object is never mutated (merge() returns a new config).
- store=<icechunk.Repository> (+ href=): write into a repository the caller
  already created. Emptiness is detected via the first-commit sentinel, so
  an empty repo takes create semantics instead of failing with
  GroupNotFoundError.
- virtual_chunk_containers=: extra VirtualChunkContainer store configs
  merged over the ones derived from the chunk references (entries win), on
  both create and append. Appends persist the merge via reopen+save_config
  (mutating repo.config is a no-op copy).
- href=/prefix= are now actually forwarded to resolve_icechunk_target, which
  makes the documented store=<Storage>+href= escape hatch work.

Tests: tests/test_icechunk_config.py (12 cases).
@betolink
betolink merged commit 3e9529f into main Nov 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant