.zarr.zip on s3

[jeffpeck10x]

I searched and could not find an example of accessing a .zarr.zip from an s3 endpoint without having to first download it entirely. The provided zarr.storage.ZipStore only works on a local path (right?).

I experimented and found that this works:

import s3fs
import zarr
from fsspec import FSMap
from fsspec.implementations.zip import ZipFileSystem

def get_s3_zarr_zip(s3_path):
    s3 = s3fs.S3FileSystem()
    f = s3.open(s3_path)
    fs = ZipFileSystem(f, mode="r")
    store = FSMap("", fs, check=False)

    # optionally use lru cache or otherwise return zarr.group(store=store)
    cache = zarr.storage.LRUStoreCache(store, max_size=2**28)
    return zarr.group(store=cache)

I am wondering if this is alright. Is there anything that could be improved with this approach?

My use-case is read-only. I understand that this approach would not be able to handle updates without updating the entire .zarr.zip.

And more generally, if someone else is searching for a solution like this, I hope this helps!

[rabernat]

This is great @jeffpeck10x! Would you be willing to add this to the tutorial?

I wonder whether the built-in ZipStore has any use for reading from S3. Or is fsspec's ZipFileSystem always required here?

[jeffpeck10x]

I would be very happy to contribute this @rabernat . Please point me to where it would best fit in the tutorial and I will submit a PR.

I think what is novel here is that the ZipFileSystem will handle fetching parts of the zip file without pulling down the whole thing. And s3fs will translate those fetches into ranged requests.

My understanding is that the caveat to this formally being part of ZipStore is that it must be read-only. (Actually, I am not sure what would happen if I tried to pass mode="w" and tried to make an update...)

An alternative might be to update the FSStore to look for the .zip extension? 🤷

Either way, that this works with only a few lines of code perhaps means that it does not require its own storage class, as long as it is documented.

[rabernat]

Please point me to where it would best fit in the tutorial and I will submit a PR.

https://github.com/zarr-developers/zarr-python/blob/main/docs/tutorial.rst

There is already a section addressing Zip stores here - https://zarr.readthedocs.io/en/stable/tutorial.html#storage-alternatives - but only on file storage, not object storage.

Zarr's built-in ZipStore class does support writing mode. However, that would never work on object storage, because object storage doesn't support partial writes to a single object. Instead the workflow for creating a zip store in the cloud seems to be:

• Create the data on a local disk
• Zip it up
• Upload to object storage
• Access using the method you described above

It would be nice to address this full life cycle in the documentation.

[joshmoore]

An additional option would be to use fsspec's handler chaining:

        store = zarr.storage.FSStore(
                url=f"zip::file://{filepath}",  mode="r"
        )

With thanks to @caviere (see https://github.com/zarr-developers/outreachy_2022_testing_zipstore/blob/main/real%20%20world%20data/main.py#L54)

Regardless, 👍 for your version and/or this one being in the tutorial location that would have helped you find them! Thanks.

[jeffpeck10x]

made a small pr to update docs: #1615

[olli4]

With Zarr v3, I used the following code to create a read-only Zarr store from a zipped Zarr in S3:

import zarr
import s3fs

class S3ZipStore(zarr.storage.ZipStore):
    def __init__(self, path: s3fs.S3File) -> None:
        super().__init__(path="", mode="r")
        self.path = path
        
s3 = s3fs.S3FileSystem(anon=True, endpoint_url=S3_ENDPOINT, asynchronous=False)
file = s3.open(f"s3://{S3_BUCKET}/{ZIP_PATH}")
zarr_store = S3ZipStore(file)

It relies on that ZipStore only checks that path is a string or a Path in __init__ and stores it in self.path for use later with zipfile.ZipFile() which also accepts the file-like s3fs.S3File. I'm a bit worried about asynchronous=False perhaps negatively affecting performance? With a non-zipped Zarr in S3 I can use asynchronous=True.

Edit: A load time benchmark shows that ZipStore created from a local zip file suffers a similar performance hit compared to LocalStore:

---
config:
    xyChart:
        width: 900
        height: 300
    themeVariables:
        xyChart:
            backgroundColor: "#000"
            titleColor: "#fff"
            xAxisLabelColor: "#fff"
            xAxisTitleColor: "#fff"
            xAxisTickColor: "#fff"
            xAxisLineColor: "#fff"
            yAxisLabelColor: "#fff"
            yAxisTitleColor: "#fff"
            yAxisTickColor: "#fff"
            yAxisLineColor: "#fff"
            plotColorPalette: "#fff8, #000"
---
xychart-beta
    title "Random Sentinel 2 patch time series load time benchmark (5100 m x 5100 m, 1 year)"
    x-axis ["S3 Zarr", "S3 zipped Zarr", "NVMe Zarr", "NVMe zipped Zarr"]
    y-axis "Mean load time (s)" 0 --> 26
    bar [7.53, 23.9, 1.12, 3.21]
    bar [0, 0, 0, 0, 0, 0, 0, 0, 0]

Loading

[joshmoore]

cc: @jwindhager

[remithiebaut]

Thank you for this solution I had exactly the same issue.
I had some trouble initializing zip store with a dummy path, it worked for me to rewrite the entire init without the assertion :

import zipfile
from zarr.storage._zip import ZipStoreAccessModeLiteral, ZipStore
from zarr.abc.store import Store

class S3ZipStore(ZipStore):
    """
    Custom ZipStore that accepts an s3fs.S3File object for zipfile-based zarr storage.
    """
    def __init__(
        self,
        path: Path | str | S3File,
        *,
        mode: ZipStoreAccessModeLiteral = "r",
        read_only: bool | None = None,
        compression: int = zipfile.ZIP_STORED,
        allowZip64: bool = True,
    ) -> None:
        if read_only is None:
            read_only = mode == "r"

        Store.__init__(self, read_only=read_only)

        if isinstance(path, str):
            path = Path(path)
        self.path = path  # root?

        self._zmode = mode
        self.compression = compression
        self.allowZip64 = allowZip64