Transparently mount super-lage directory using FUSE

[Busboy3129]

Current Limitation

xet can save storage on the server but when user clone the repo to local disk they becomes normal files

Feature Description

transparently mount super-lage directory using FUSE, so that user can access these files directly via http and only use local disk as a LRU cache

Additional Context

No response

[rajatarya]

Hi @Busboy3129 : We have thought about this and used to offer something similar back in XetHub. However, the user experience for this is difficult considering the current ways users download repositories. There isn't a notion of a repo locally - either downloading a snapshot from a repo or specific files.

If you aren't looking for the entire repo, how about just downloading the files you are interested in:

hf download ...

See docs here: https://huggingface.co/docs/huggingface_hub/guides/cli#hf-download

Also, FUSE has worked reasonably well on Linux but we have experienced many issues with its stability on MacOS. Instead, we implemented this perviously using a local NFSv3 server (repo).

[Busboy3129]

Thanks for reply! However, a repo with many duplicated small files means calling hf download a million times.
I wonder if juicefs's code could help. they have very good fuse support but in golang