[checkpoint] Support fsspec checkpoint paths

[YassineYousfi]

make checkpointing fsspec aware, leave it to DCP to pick the appropriate writer/reader
when fsspec is used, base_folder is not used
also added unit test

[pytorch-bot]

Workflows were awaiting approval. CI has now been triggered for the ciflow labels on this PR.

[fegin]

I don't think it is a good idea to make fsspec as the default (we didn't do this for DCP either). Several reasons, we didn't measure the performance (probably not a big issue), unclear support of mmap, which can be a potential use case in the future.

I understand S3 support is important. A better design is to make fsspec as an option not default.

[YassineYousfi]

I don't think it is a good idea to make fsspec as the default (we didn't do this for DCP either). Several reasons, we didn't measure the performance (probably not a big issue), unclear support of mmap, which can be a potential use case in the future.

I understand S3 support is important. A better design is to make fsspec as an option not default.

Done. DCP already tries fsspec in _storage_setup, so the only changes are the file management/discovery

https://github.com/pytorch/pytorch/blob/main/torch/distributed/checkpoint/_storage_utils.py#L33

[fegin]

The title needs to be changed too.

[tianyu-l]

related PR #3887