feat(cloud_io): cloud-optimised raster I/O (COG / Zarr / S3 / GCS) (P31)

[MAfarrag]

Context

Modern continental DEMs (Copernicus GLO-30, USGS 3DEP 1 m) are distributed as Cloud-Optimised GeoTIFFs (COG)
on S3 / GCS, optionally as Zarr. Phase 1–3 read these via pyramids.Dataset, which uses GDAL's VSI virtual
filesystem — fine for small files, painful for continental-scale where a single COG can be 50+ GB. Phase 4
makes cloud-streamed I/O a first-class read/write path with range requests, server-side overview selection,
and credential management.

Problem / Current Behaviour

• COG writes worked only via raw GDAL invocations with no helper.
• No Zarr support.
• No first-class S3/GCS/Azure URIs in DEM(...) / DEM.export(...).
• No overview-aware reads — every load reads the full resolution even when the user requests a coarser
  target.

Affected locations

File	Symbol	Notes
src/digitalrivers/cloud_io.py	write_cog(...)	shipped — COG writer with DEFLATE + PREDICTOR=3 + internal overviews.
src/digitalrivers/cloud_io.py	cloud_storage(...)	pending — fsspec-driven S3/GCS/Azure path resolver.
src/digitalrivers/cloud_io.py	(Zarr writer)	pending.

Motivation Example

from digitalrivers import DEM
from digitalrivers.cloud_io import write_cog

dem = DEM.read_file("conditioned.tif")

# Shipped — write a COG with internal overviews + DEFLATE+PREDICTOR=3 compression.
write_cog(dem, "build/conditioned.cog.tif", compress="deflate")

# Pending — S3 / Zarr.
# dem = DEM("s3://copernicus-dem-30m/Copernicus_DSM_COG_10_N00_00_E000_00.tif", target_resolution=90.0)
# dem.export("s3://my-bucket/conditioned.tif", target="cog")
# dem.export("gs://my-bucket/conditioned.zarr", target="zarr")

Proposed Solution

from digitalrivers.cloud_io import write_cog, cloud_storage

# Shipped.
def write_cog(dataset, path: str, compress: str = "deflate") -> str: ...

# Pending API.
def cloud_storage(uri: str, **kwargs) -> tuple["fsspec.AbstractFileSystem", str]: ...

# DEM constructor + export accept s3://, gs://, az://, https:// URIs:
DEM("s3://bucket/key.tif", target_resolution=90.0)            # overview-driven
dem.export("s3://bucket/key.tif", target="cog", overviews="auto")
DEM("s3://bucket/key.zarr"); dem.export("gs://bucket/k.zarr", target="zarr")

Implementation strategy:

• Support s3://, gs://, az://, https:// URIs in DEM(...), DEM.export(...), and conditioning
  functions via rasterio + fsspec (s3fs/gcsfs).
• COG-aware reads: detect COG layout via dataset.IsTiled() and GetMetadataItem("LAYOUT"); prefer reading
  overview level appropriate to the requested cell size via GDAL WarpedVRT.
• Zarr support via rioxarray / xarray.open_zarr.
• Credential discovery: boto3 / google-auth default chains; expose
  digitalrivers.config.set_storage_options(...) for non-default auth.
• GDAL env defaults: VSI_CACHE=YES, GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR (skip sidecar probes wasting
  HEAD requests).

References

• COG specification: https://www.cogeo.org/
• GDAL VSI documentation: https://gdal.org/user/virtual_file_systems.html
• rasterio COG cookbook, rio-cogeo profiles.
• fsspec + s3fs / gcsfs documentation.
• rioxarray Zarr support.
• stackstac AutoParallelRioReader — production pattern for VSI auth + windowed reads in dask graphs.

Out of Scope

• Requester-pays buckets, corporate-proxy edge cases — handle on user demand, not by default.
• Custom credential providers beyond boto3 / google-auth defaults.

Effort Estimate

Size: M (4 days)
Rationale: COG write is straightforward via rasterio. Auth corner cases (corporate proxies, signed
URLs, requester-pays) are the slow part for the cloud paths.

Definition of Done

• cloud_io.write_cog(...) ships with DEFLATE + PREDICTOR + internal overviews.
• Pending: Read a 50 GB Copernicus GLO-30 COG via range requests; total bytes transferred for
  DEM(uri, target_resolution=900.0) <= 100 MB (overview-driven).
• Pending: Write a 4000×4000 conditioned DEM as a COG to S3 with internal overviews; gdalinfo /vsis3/... confirms overviews.
• Pending: Zarr round-trip — write → read → assert bitwise equality.
• Pending: Credentials — code works with both AWS profile-based and env-var-based auth (test via
  localstack / moto).
• Pending: No-credential public reads (Copernicus DEM bucket) work without auth setup.

Status: Partial on branch feat/phase-4. Shipped: write_cog with overviews + compression. Pending:
Zarr writer, S3/GCS range reads, fsspec-backed cloud_storage(...). See tests/test_cloud_io.py.