Repository navigation
feat(icechunk): repository_config, Repository targets, and VCC overrides - #1
Merged
Merged
Conversation
Three additive escape hatches for callers that need control over the icechunk repository, all defaulting to today's behaviour: - repository_config=: an icechunk.RepositoryConfig applied when the repo is created (manifest splitting/preload, inline threshold, compression, ...). Ignored with a UserWarning on append, where the stored config wins. The caller's object is never mutated (merge() returns a new config). - store=<icechunk.Repository> (+ href=): write into a repository the caller already created. Emptiness is detected via the first-commit sentinel, so an empty repo takes create semantics instead of failing with GroupNotFoundError. - virtual_chunk_containers=: extra VirtualChunkContainer store configs merged over the ones derived from the chunk references (entries win), on both create and append. Appends persist the merge via reopen+save_config (mutating repo.config is a no-op copy). - href=/prefix= are now actually forwarded to resolve_icechunk_target, which makes the documented store=<Storage>+href= escape hatch work. Tests: tests/test_icechunk_config.py (12 cases).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three additive escape hatches for callers that need control over the icechunk
repository, plus a wiring fix for the documented
href=/prefix=options.All of them default to today's behaviour, so existing calls are unchanged.
repository_config=An
icechunk.RepositoryConfigapplied when the repository is created —manifest splitting/preload, inline-chunk threshold, compression, concurrency.
UserWarninginstead of silently doing nothing.
RepositoryConfig.merge()returns a newconfig).
store=<icechunk.Repository>(+href=)Write into a repository you created yourself; the plugin never creates or
reconfigures it.
(
1CECHNKREP0F1RSTCMT0), so an empty repository takes create semanticsinstead of failing with
GroupNotFoundError(which is what happens todaywith a pre-created repo).
href=is required because aRepositorycarries no readable path.repository_config=together with aRepositoryis rejected (the repositoryalready owns its configuration).
virtual_chunk_containers=Extra
VirtualChunkContainerstore configs keyed byurl_prefix, merged overthe ones derived from the dataset's chunk references (entries win), on create
and append:
Appends persist the merge via
repo.reopen(config=…) + save_config()— mutatingrepo.configis a no-op copy in icechunk 2.3.Fix:
href=/prefix=were not wiredThe
Optionsdocstring advertisedstore=<icechunk.Storage>+href=, butneither
hrefnorprefixwas forwarded toresolve_icechunk_target, so thatescape hatch raised
ValueErrorat write time. They are now forwarded (andcovered by a test).
Tests
tests/test_icechunk_config.py: 12 cases (create with config, caller objectuntouched, append warning + stored config wins, Repository empty → create,
Repository with data → append, missing
href, Repository+config rejection,Storage+
href, VCC merge/override on create and append).Docs
README only (options table rows) — the detailed walkthrough goes in the
earthaccess notebooks as a separate follow-up.
Not in this PR
ea.open_virtual()/read-side authorization: readers alreadyaccept
authorize_virtual_chunk_access=and the derived VCC mapping.