A Torrent ZarrStore

[alxmrs]

Coming back from Cloud Native Geo, a major theme was how to build resilient data infrastructure (or ecosystem). A common idea that came up in many discussion groups was could we somehow store scientific data via bit-torrent. I don't think this idea seems too far fetched; we simply need to figure out how the Zarr protocol p2p torrent protocols align.

Projects like https://academictorrents.com/ (https://github.com/academictorrents) seem to validate that a peer-to-peer, public, global store of data is highly desirable – especially in a time when critical medical and environmental data is potentially being lost.

This repo in their project, in particular, seems very xpublish shaped! https://github.com/academictorrents/torrentify. A deeper look into the intersection of these protocols is warranted right now (maybe, even urgently).

Update: @samapriya is definitely one of the people at the conference who first brought up this idea!

[alexgleith]

Is IPFS a plausible alternative?

https://ipfs.tech/

[alxmrs]

"Yet, but" is the short answer (or maybe, "it depends"). IIRC, IPFS is like torrent plus a git-like VCS. I think that IPFS may suit our use case well, but I think that a better fit may be pure torrents + ice-chunk on top of Zarr (to handle version control). More experimentation is definitely need! I know @rabernat has explored IPFS before.

[alxmrs]

OK, I think the byte-block limit rules out IPFS for doing anything besides hosting websites: https://discuss.ipfs.tech/t/what-is-the-maximum-ipfs-block-size/12453/2

Torrenting, on the other hand, was designed for distributing large media files, making it a better fit for Zarr.

[alxmrs]

The file structure of .torrent seems very Zarr-like to me: https://en.wikipedia.org/wiki/Torrent_file#File_Structure

[alxmrs]

I think aira2 may be a useful component to managing IO of torrent-based Zarr stores. It already has BitTorrent support from the docs: https://aria2.github.io/manual/en/html/aria2c.html#bittorrent-download

[samapriya]

We literally had this discussion on academic torrents in our group so exciting to see this form

[rsignell]

FWIW, a bunch of data has been backed up on sciop.net (example) and I've been using aria2 to grab certain datasets and copy them to object storage to make them more accessible, like the NCEI Estuarine Bathymetry: https://nbviewer.org/gist/rsignell/7339b3a4aa8d39eff4cd766e127aa77e

[alxmrs]

Really interesting Rich! The sciop.net catalog is really validating. Further, from reading your notebook, I wonder if what we need is not a new Zarr store by a torrent-based filesystem (say, in fsspec).

[fmigneault]

Would this torrent approach work hand-in-hand with Zarr stores that get extended (i.e.: Modifying existing Zarr stores)? Would that break the integrity of the torrent shared between peers?