Description
Best approach to support reading data from http via a pathlib
-like class, i.e. httppathlib
?
In the pangeo / xarray community we do a lot of reading of remote scientific data (particularly netCDF and Zarr). We generally want to treat 3 cases the same way: local filesystems, cloud storage, and http urls. The latter is important partly because a lot of archival scientific data is still only available from servers via http (e.g. via openDAP urls), and we often want to pull it out and deposit it onto cloud storage (e.g. using pangeo-forge).
We currently use fsspec
to abstract over these different filesystems, but despite much engagement upstream we have unfortunately experienced chronic reliability issues stemming from ill-defined interfaces.
CloudPathlib looks really nice, especially the strict typing and clear interface. (I'm in awe of the AnyPath
virtual superclass trick too - and with #347 would be even cooler!) The Path
abstraction also just seems more like the minimally-useful one, rather than trying to emulate a whole filesystem.
Rather than trying to support every filesystem under the sun as fsspec
does, I'm wondering if we could just use pathlib
, cloudpathlib
, and some new httppathlib
?
Do you have any thoughts on:
- Whether you think this is a good idea?
- The experiments already performed with xarray in Hello from fsspec! #96 (comment)?
- How hard it might be to get a
httppathlib
to conform to thepathlib
interface? - Where such a project might live: in
cloudpathlib
or in a separate repository?