Support remote fsspec checkpoint paths (e.g. gs://) via filesystem helpers#3887
Support remote fsspec checkpoint paths (e.g. gs://) via filesystem helpers#3887zhixiangli wants to merge 2 commits into
Conversation
…lpers Add torchtitan/tools/filesystem.py to transparently dispatch path ops to os/shutil for local paths and fsspec for remote URIs, then route checkpoint.py's directory listing/existence/cleanup through it so checkpoints can be written to and read from remote storage like GCS.
zhixiangli
requested review from
fegin,
tianyu-l,
wconstab and
wwwjn
as code owners
|
Hi @zhixiangli! Thank you for your pull request. We require contributors to sign our Contributor License Agreement, and yours needs attention. You currently have a record in our system, but the CLA is no longer valid, and will need to be resubmitted. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
|
The following ciflow label(s) have been added but CI has not been triggered yet because the workflows are awaiting approval:
Once a maintainer approves the workflows (scroll to the bottom of the PR page), the corresponding CI jobs will be triggered automatically. Please ping one of the reviewers if you do not have access to approve and run workflows. |
meta-cla
Bot
added
the
CLA Signed
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
Object stores (GCS, S3) can list a directory-marker entry for the listed directory itself, which os.listdir never returns. Filter it out by full normalized path so a legitimate child sharing the parent's basename is still kept.
2 participants
Fix #3884
Add torchtitan/tools/filesystem.py to transparently dispatch path ops to os/shutil for local paths and fsspec for remote URIs, then route checkpoint.py's directory listing/existence/cleanup through it so checkpoints can be written to and read from remote storage like GCS.