Is it safe to copy checkpoint files during training?

[Zhangzhiwei39]

Dear fairchem team,
I am using GemNet in fairchem-1.10.0 for training GemNet. To back up training progress, I have set up a Linux script that automatically runs the command "cp checkpoint.pt backup_checkpoint.pt" every 4 hours to copy the checkpoint file.
However, I later realized that this copy operation might occur at the same time as fairchem is writing the checkpoint. Although none of the checkpoint files I have copied so far have report errors and they can all be successfully loaded into fairchem (and even for perform structures relaxation in ASE), I am concerned that if the file is being written while the copy happens, the resulting backup might be incomplete.
My questions are:
Is my concern unnecessary? That is, does fairchem use some mechanism (such as writing to a temporary file first and then renaming it) to avoid this issue when saving checkpoints?
If there is indeed a risk, should backup checkpoints obtained without ensuring that writing has completed can be used for subsequent tasks (such as perform relaxation to unseen structures in ASE)? Or must training be restarted from scratch?
Thank you for your help!

[rayg1234]

it depends on which checkpoint artifacts you are using, if you are using the DCP (distributed checkpoint) data, it should be safe because the mechanism is controlled by torch DCP, if you are using the "inference_checkpoint", this is probably not safe because we just use torch.save here to write it (instead of write to temp then copy - you can also modify this behavior and PR into the repo if you like to make it safer)

fairchem/src/fairchem/core/units/mlip_unit/mlip_unit.py

Line 128 in 40a7488

torch.save(

[Zhangzhiwei39]

it depends on which checkpoint artifacts you are using, if you are using the DCP (distributed checkpoint) data, it should be safe because the mechanism is controlled by torch DCP, if you are using the "inference_checkpoint", this is probably not safe because we just use torch.save here to write it (instead of write to temp then copy - you can also modify this behavior and PR into the repo if you like to make it safer)

fairchem/src/fairchem/core/units/mlip_unit/mlip_unit.py

Line 128 in 40a7488

torch.save(

Thank you for your reply. I am a graduate student majoring in chemistry. I primarily use the FairChem project and lack strong coding skills, so I am unable to submit a pull request. Thank you again for your information.