Skip to content

Store PNGs to google cloud#222

Merged
trautmane merged 19 commits into
newsolverfrom
feature/cloud-pngs
Jun 28, 2025
Merged

Store PNGs to google cloud#222
trautmane merged 19 commits into
newsolverfrom
feature/cloud-pngs

Conversation

@minnerbe

Copy link
Copy Markdown
Collaborator

This PR extends the RenderTilesClient to be able to store rendered tiles¹ to cloud storage².

¹ Only PNGs are tested so far. The actual writing is done using ImageIO.write to write to a ByteArrayOutputStream, which is subsequently uploaded. Every format that is supported by that workflow should work, but the parameters --convertToGray and --quality are ignored right now.

² Only GCS is supported so far. For AWS, the changes to GCS shouldn't be too dramatic (since the actual object access is implemented using N5's KeyValueAccess), but I haven't spent the time figuring out authentication details.

Further caveats:

  • We currently use an old version of N5GoogleCloud (4.1.1), which has bugs in URI handling. The bugs should be fixed in version 5.X.X, but for now all URI handling is done using other tools.
  • The KeyValueAccess interface does not allow for fine-grained control over the upload. This results in all images being tagged with type application/octet-stream. To preview the images directly in GCS, one has to manually set the type to image/png.

I've tested that

  • File system paths are the same as before the refactoring.
  • Writing both images and masks works.
  • GCS paths are correct.
  • The paths are correctly stored in the hack stack and work (see here).

Let me know if that's what you intended, @trautmane!

@minnerbe
minnerbe requested a review from trautmane June 23, 2025 16:11
@trautmane

Copy link
Copy Markdown
Collaborator

Looks great! I'll use this branch to add spark pipeline stuff that wraps what you've done here.

@trautmane
trautmane merged commit 12e909f into newsolver Jun 28, 2025
2 checks passed
@trautmane
trautmane deleted the feature/cloud-pngs branch June 28, 2025 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants