Skip to content

feat(regions): default S3/icechunk targets to us-west-2 - #4

Merged
betolink merged 1 commit into
mainfrom
feat/regions
Nov 16, 2026
Merged

betolink merged 1 commit into
mainfrom
feat/regions

Conversation

@betolink

Copy link
Copy Markdown
Member

What

region= (default us-west-2) for S3/icechunk targets, plus the documented
constraint — no in-region detection.

NASA Earthdata S3 buckets live in us-west-2 and direct S3 access is only
allowed from inside that region. The s3:// shorthand previously built
icechunk.S3Options() with no region, so it silently relied on the SDK
default (us-east-1) and failed with a redirect for NASA buckets.

  • Options.region (default us-west-2) → resolve_icechunk_target →
    icechunk.S3Options(region=...); read_icechunk_state,
    write_ingested_only and write_icechunk thread it through.
  • An obstore S3Store keeps its own region; the bridge falls back to the
    default when the store does not carry one.
  • A region-looking write failure (301/PermanentRedirect) now raises with a
    hint naming region=.
  • regions.py documents the rule instead of probing instance metadata
    (metadata is blocked by policy in many environments and absent on Lambda, so
    detection would be unreliable). The README gains a Regions section:
    direct S3 → workers in us-west-2; HTTPS → anywhere; the store write is
    client-side and only needs the target bucket's region.

Tests

  • tests/test_regions.py (8): s3:// default and override, S3Store bridge
    default/override, region hint, option default, request accepts a custom region.
  • Full suite 115 passed, mypy clean (24 files), ruff clean, pre-commit clean.

Companion (already on plugins, no PR)

earthaccess _icechunk_storage now defaults region="us-west-2" for s3://
stores unless storage_options overrides it, so readers work out of the box.

…ride

- new region= option (default us-west-2, where NASA Earthdata S3 lives) for the
  s3:// shorthand; a preconfigured icechunk.Storage / obstore S3Store keeps its
  own region (the bridge falls back to us-west-2 when the store has none)
- read_icechunk_state / write_ingested_only / write_icechunk thread it through
- a region-looking write failure (301/PermanentRedirect) now raises with a hint
  naming region=
- regions.py documents the constraint (direct S3 is us-west-2 only; HTTPS works
  anywhere; the write is client-side and only needs the target's region). No
  in-region detection: instance metadata is not reliable everywhere.
- README: region row + a Regions section. Tests: target default/override, store
  bridge default/override, region hint, option default.
@betolink
betolink merged commit a88ab9b into main Nov 16, 2026
2 checks passed
@betolink
betolink deleted the feat/regions branch November 16, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant