This test
|
hf upload \ |
|
"metatensor/metatrain-test" \ |
|
"model-32-bit-$TRAIN_ID.ckpt" \ |
|
"model.ckpt" \ |
|
--commit-message="Overwrite test model with new version" \ |
|
--token="$HUGGINGFACE_TOKEN_METATRAIN" |
Will run in every PR. If two PR try to run it at the same time, then we might have the case where PR A upload, then PR B upload and overwrite, then PR A download the checkpoint from PR B. This works fine most of the time, but fails if PR B is trying to do a checkpoint version upgrade (which is what happened with #1135 being PR A and #1107 being PR B).
It would be nice to check if instead of overwriting the same test checkpoint in every CI run, we instead upload to a temporary — maybe even local — "hugging face"-looking repository.
This test
metatrain/tests/resources/run_trainings.sh
Lines 71 to 76 in 8a3ee83
Will run in every PR. If two PR try to run it at the same time, then we might have the case where PR A upload, then PR B upload and overwrite, then PR A download the checkpoint from PR B. This works fine most of the time, but fails if PR B is trying to do a checkpoint version upgrade (which is what happened with #1135 being PR A and #1107 being PR B).
It would be nice to check if instead of overwriting the same test checkpoint in every CI run, we instead upload to a temporary — maybe even local — "hugging face"-looking repository.