Skip to content

testing/plugins: migrate cloud fixtures to use fsspec filesystems #8917

Open
@pmrowla

Description

@pmrowla

Cloud fixtures currently depend on each plugin to implement the cloud/path-like utility methods (read_text/bytes, write_text/bytes, etc) and we do this by using the underlying libraries ourselves directly in dvc plugins (i.e. dvc_s3.tests.cloud uses boto directly). It would be easier to add new functionality if the plugin specified an fsspec filesystem to use, and then the generic dvc.testing.Cloud used the fsspec api calls instead. That way adding new functionality for things like retrieving specific version ID's can be added without having to use the underlying aws/gcs/adlfs libraries ourselves.

Basically, since we already added these things in the underlying fsspec implementations, it doesn't make sense for us to be re-implementing them all again in each plugin's Cloud test fixtures.

This would also let us implement a generic Cloud.remove() (currently we use DVC utils.fs.remove() for removing local files, but have no equivalent for the cloud/remote fixtures)

related: #8850

Metadata

Metadata

Assignees

No one assigned

    Labels

    p3-nice-to-haveIt should be done this or next sprinttestingRelated to the tests and the testing infrastructure

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions