Repository navigation
feat(regions): default S3/icechunk targets to us-west-2 - #4
Merged
Merged
Conversation
…ride - new region= option (default us-west-2, where NASA Earthdata S3 lives) for the s3:// shorthand; a preconfigured icechunk.Storage / obstore S3Store keeps its own region (the bridge falls back to us-west-2 when the store has none) - read_icechunk_state / write_ingested_only / write_icechunk thread it through - a region-looking write failure (301/PermanentRedirect) now raises with a hint naming region= - regions.py documents the constraint (direct S3 is us-west-2 only; HTTPS works anywhere; the write is client-side and only needs the target's region). No in-region detection: instance metadata is not reliable everywhere. - README: region row + a Regions section. Tests: target default/override, store bridge default/override, region hint, option default.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
region=(defaultus-west-2) for S3/icechunk targets, plus the documentedconstraint — no in-region detection.
NASA Earthdata S3 buckets live in
us-west-2and direct S3 access is onlyallowed from inside that region. The
s3://shorthand previously builticechunk.S3Options()with no region, so it silently relied on the SDKdefault (us-east-1) and failed with a redirect for NASA buckets.
Options.region(defaultus-west-2) →resolve_icechunk_target→icechunk.S3Options(region=...);read_icechunk_state,write_ingested_onlyandwrite_icechunkthread it through.S3Storekeeps its own region; the bridge falls back to thedefault when the store does not carry one.
301/PermanentRedirect) now raises with ahint naming
region=.regions.pydocuments the rule instead of probing instance metadata(metadata is blocked by policy in many environments and absent on Lambda, so
detection would be unreliable). The README gains a Regions section:
direct S3 → workers in us-west-2; HTTPS → anywhere; the store write is
client-side and only needs the target bucket's region.
Tests
tests/test_regions.py(8):s3://default and override,S3Storebridgedefault/override, region hint, option default, request accepts a custom region.
Companion (already on
plugins, no PR)earthaccess_icechunk_storagenow defaultsregion="us-west-2"fors3://stores unless
storage_optionsoverrides it, so readers work out of the box.