Skip to content

Read from http - httppathlib? #455

Open
@TomNicholas

Description

@TomNicholas

Best approach to support reading data from http via a pathlib-like class, i.e. httppathlib?

In the pangeo / xarray community we do a lot of reading of remote scientific data (particularly netCDF and Zarr). We generally want to treat 3 cases the same way: local filesystems, cloud storage, and http urls. The latter is important partly because a lot of archival scientific data is still only available from servers via http (e.g. via openDAP urls), and we often want to pull it out and deposit it onto cloud storage (e.g. using pangeo-forge).

We currently use fsspec to abstract over these different filesystems, but despite much engagement upstream we have unfortunately experienced chronic reliability issues stemming from ill-defined interfaces.

CloudPathlib looks really nice, especially the strict typing and clear interface. (I'm in awe of the AnyPath virtual superclass trick too - and with #347 would be even cooler!) The Path abstraction also just seems more like the minimally-useful one, rather than trying to emulate a whole filesystem.

Rather than trying to support every filesystem under the sun as fsspec does, I'm wondering if we could just use pathlib, cloudpathlib, and some new httppathlib?

Do you have any thoughts on:

  1. Whether you think this is a good idea?
  2. The experiments already performed with xarray in Hello from fsspec! #96 (comment)?
  3. How hard it might be to get a httppathlib to conform to the pathlib interface?
  4. Where such a project might live: in cloudpathlib or in a separate repository?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions