Feature Request: Model weight sharing on `dlabscracth1`

[Saibo-creator]

A candidate solution: https://benjijang.com/posts/2024/01/shared-hf-cache/

Current solution: https://github.com/epfl-dlab/dlab-utils

Related issue #2

[Saibo-creator]

Asked Alex (MLO) how they handle this issue, here is his reponse:

basically, in our default cluster setup, we set this env variable:
https://github.com/epfml/getting-started/blob/f80517183c36e0a37e6ee48ebbcdbd3e7206c131/csub.py#L235
GitHubGitHub
getting-started/csub.py at f80517183c36e0a37e6ee48ebbcdbd3e7206c131 · epfml/getting-started
Contribute to epfml/getting-started development by creating an account on GitHub. (43 kB)
https://github.com/epfml/getting-started/blob/f80517183c36e0a37e6ee48ebbcdbd3e7206c131/csub.py#L235

  HF_HOME:
    value: /mloscratch/hf_cache

however, two points:
important to set umask to 007 for group read permissions, e.g. permanently via echo "umask 007" >> /docker/.zshrc
per default, HF_HOME also sets the token inside that cache folder. that means that you can login as a different user (or someone else can reuse your token) 😄 which it’s why it’s important to also set the HF_TOKEN env variable directly with the token, e.g.
HF_TOKEN:
value: {user_cfg['hf_token']}

[Saibo-creator]

One solution for the HF_token polution is to use HF_HUB_CACHE, here is my reply to Alex:

This sounds great! Setting a common cache directory seems to be the best solution indee.d
Upon reviewing the HF cache documentation, it seems that Hugging Face provides a more fine-grained option for sharing only models and datasets via the HF_HUB_CACHE environment variable.
You can find details here: HF_HUB_CACHE Documentation.
This approach might help avoid the token pollution problem you mentioned 🙂

[Saibo-creator]

A bit of thoughts on the permission aspect:

We can set umask for all users on iccluster036/060 and runai image to 00x so that the models/datasets in HF_HUB_CACHE are readable and writeable for all users.

This may bring a questions, if user A download a model X, does user B should have permission to overwrite it ? The overwrite may happen if there is an update for the model(dataset) but I guess it's rare ?

There is a .lock problem explained in this https://benjijang.com/posts/2024/01/shared-hf-cache/