Add support for opening OME-Zarr datasets in public S3 stores - #84
Merged
Conversation
stefanhahmann
changed the title
stefanhahmann
force-pushed
the
add-s3-support
branch
from
June 11, 2026 14:26
07d8022 to
dd4746d
Compare
|
xulman
previously approved these changes
Jun 15, 2026
xulman
left a comment
xulman
left a comment
Contributor
There was a problem hiding this comment.
Nice work. Thanks.
One of the tests caught my attention, and I was "complaining" that perhaps we could reuse code and maybe even shorten/simplify the (testing) codebase?
Also, I'm thinking if it would be possible to make N5 and ZarrJava throw the same exception. Or is our layer swallowing them and re-sending further to a client code under one common exception?
stefanhahmann
force-pushed
the
add-s3-support
branch
7 times, most recently
from
August 5, 2026 12:07
3435f87 to
3666ab3
Compare
Collaborator
Author
This is now the case. Both backends translate their native store-open failures into a single backend-agnostic exception, StoreAccessException (in the ome-zarr-imglib2 API module):
The Fiji opener (ZarrOpener) then catches just that one type and surfaces it as a user-facing error. |
stefanhahmann
force-pushed
the
add-s3-support
branch
from
August 5, 2026 13:36
f79bbb4 to
a9cfbf9
Compare
Required for S3 support
Previously any URI with a scheme other than http, https, or file was rejected. Accept s3:// as a supported scheme so S3 dataset URIs can be pasted from the clipboard.
Add an s3:// branch that constructs an S3Store using S3Client.create() and bucket/key parsed directly from the URI host and path. An empty key prefix is passed as null to avoid a leading slash in resolved S3 object keys. The N5PyramidBackend already handles S3 through N5Factory and required no changes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Probing would create a short-lived S3Client just for detection. The open path creates the client it needs anyway and reports a clear error if the location is not OME-Zarr, so the probe is skipped for s3:// URIs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
S3Client.create() requires credentials to be configured externally, blocking access to public datasets for users without ~/.aws/credentials or environment variables set. Using a credentials chain that tries the default providers first and falls back to AnonymousCredentialsProvider mirrors in the n5 backend. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both are RuntimeExceptions thrown on store-access failures (S3 auth failures, missing buckets, network errors) that previously escaped every catch block in openImage() and surfaced as unhandled exceptions in Fiji. Now caught and shown as a user-facing error with the store message. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Set Region.US_EAST_1 as the default S3 region in both N5PyramidBackend and ZarrJavaPyramidBackend so that environments without ~/.aws/config (e.g. CI) can open public S3-hosted OME-Zarr datasets without an "Unable to load region" SDK error. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add openImageFromS3 integration test that exercises the full metadata loading pipeline for both backends against a public Janelia OpenOrganelle dataset. Uses openImage() and counting pattern instead of opening in ImageJ/BDV to avoid tile downloads (to not slow the test down even more) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests that both backends report a user-facing error via the error handler when S3 store access fails. The ZARR_JAVA variant mocks ZarrJavaPyramidBackend to throw StoreException; the N5 variant relies on N5Exception.N5IOException being thrown immediately (local failure, no network needed). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ZarrOpener (ome-zarr-fiji) is decoupled from the concrete backends, so it cannot catch zarr-java's StoreException directly. This commit introduces a new class StoreAccessException (ome-zarr-imglib2) Each backend's load() wraps its native store-open failure into it (zarr-java's StoreException, N5's N5Exception), and ZarrOpener catches that single type. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the openImage() methods from ZarrOpener and ZarrOpenActions. Success tests now load via ZarrOpener.getContents() (no UI) Rejection tests use the real openIJWithImage(), which is window-free on failure paths.
stefanhahmann
force-pushed
the
add-s3-support
branch
from
August 6, 2026 14:14
a9cfbf9 to
6dd69dd
Compare
Removes the race condition where OpenResolutionLevelCommand saw no active pyramidal until the async AWT focus event arrived (sometimes failed in the CI under load).
Synchronous active-pyramidal registration makes the EDT flush obsolete.
stefanhahmann
force-pushed
the
add-s3-support
branch
from
August 6, 2026 15:06
bbaec06 to
6a190ed
Compare
xulman
previously approved these changes
Aug 11, 2026
S3Store reported S3 errors as the AWS SDK's unchecked SdkException, which escaped the StoreException-only catch and surfaced unhandled in Fiji. Catch both exception types now and drop the MockedConstruction from storeAccessErrorIsReportedToErrorHandler so ZARR_JAVA exercises the same real path as N5.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables opening
s3://bucket/pathURIs in both the ZarrJava and N5 backends.What's included:
s3://URIs inClipboardUtilsand skip theisZarr()probe for them (which would require a short-lived S3 client just for detection)ZarrJavaPyramidBackend: builds an AWS SDKS3Client(default regionUS_EAST_1; standard AWS credential chain with an anonymous-credentials fallback) and constructs a zarr-javaS3Storefrom the URI's host and path. The AWS SDK ships transitively withzarr-java.N5PyramidBackend: relies onN5Factory's existing S3 support (vian5-aws-s3), only passingRegion.US_EAST_1as the default region so environments without~/.aws/configstill work. Then5-aws-s3dependency was added to theome-zarr-n5module.Error handling:
load(), into a backend-agnosticStoreAccessException(added toome-zarr-imglib2):ZarrJavaPyramidBackendwraps zarr-java'sStoreException,N5PyramidBackendwraps N5'sN5Exceptionthrown while opening the reader / reading multiscale metadata.ZarrOpener) catches the singleStoreAccessExceptionand reports it as a user-facing error. This keeps the Fiji layer decoupled from the concrete backends.Authentication:
~/.aws/credentials,AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEYenv vars, or instance role credentials).Not yet included:
No UI to enter an S3 URI, bucket, region, or credentials. Users must paste an
s3://URI via the existing clipboard/drag-and-drop path.There is currently no API to inject custom credentials programmatically.
Can be tested with this URI from the open organelle project: s3://janelia-cosem-datasets/jrc_mus-choroid-plexus-3/jrc_mus-choroid-plexus-3.zarr/recon-1/em/fibsem-uint8