enable auto cache clearing on dlabscratch

[Saibo-creator]

To avoid each user to redownload the model weights everytime, which wastes time and also storage, Julian has made this library:

from dlab_utils.paths import model_path, available_models
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(model_path("meta-llama/Meta-Llama-3-8B-Instruct"))
tokenizer = AutoTokenizer.from_pretrained(model_path("meta-llama/Meta-Llama-3-8B-Instruct"))

print(available_models())

This is really good and simple. Would be even better with autodownloading
What I can do in addition when I get sudo is to periodically delete cache directory so that people need to go for this if they don't want to redownload everytime (edited)

[jkminder]

Minor syntax updates + autodownloading is implemented now

from dlabutils import model_path, available_models
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(model_path("meta-llama/Meta-Llama-3-8B-Instruct"))
tokenizer = AutoTokenizer.from_pretrained(model_path("meta-llama/Meta-Llama-3-8B-Instruct"))

print(available_models())