Why is checkpointing useful?
Support for checkpointing when running large jobs would be useful for several reasons:
- Sometimes runs get unexpectedly killed before they finalize, due to hardware or human errors
- Runs can be stopped by the queueing system due to limited wall-clock time
Most importantly, running jobs that require longer wall-clock time than offered by the HPC's queueing system is currently not possible.
Solution(s)
Large inference jobs refer to either jobs where model instances need to be initialized many times, possibly in parallel, or jobs with long lead times (rolling out a model for years, for instance).
Many model instances
For many model instances, Bris-inference already supports intermediate files, so starting from where a run ended just means that we need knowledge about the currently generated files, and generate the ones missing.
One issue here is that the slurm job-id is currently used as the run-id, which does not work if we want to continue running with the same run-id, but with a new slurm job-id. An option is to make a separate bris run-id, and use it as the run folder.
Long lead times
Very long lead times is probably harder to checkpoint, as the whole forecast is typically written to one single file. I think we can ignore this for now, as only runs longer than ~20 years will be affected by a 24 hour wall-clock limit.
Why is checkpointing useful?
Support for checkpointing when running large jobs would be useful for several reasons:
Most importantly, running jobs that require longer wall-clock time than offered by the HPC's queueing system is currently not possible.
Solution(s)
Large inference jobs refer to either jobs where model instances need to be initialized many times, possibly in parallel, or jobs with long lead times (rolling out a model for years, for instance).
Many model instances
For many model instances, Bris-inference already supports intermediate files, so starting from where a run ended just means that we need knowledge about the currently generated files, and generate the ones missing.
One issue here is that the slurm job-id is currently used as the run-id, which does not work if we want to continue running with the same run-id, but with a new slurm job-id. An option is to make a separate bris run-id, and use it as the run folder.
Long lead times
Very long lead times is probably harder to checkpoint, as the whole forecast is typically written to one single file. I think we can ignore this for now, as only runs longer than ~20 years will be affected by a 24 hour wall-clock limit.