solutions for very long-running jobs

[tavareshugo]

Could add something to the materials for very long jobs that timeout (e.g. on CSD3).

• checkpoint:
  • https://dmtcp.sourceforge.io/ (need to see how this works exactly)
  • another one: https://criu.org/Main_Page
  • this may sometimes break things -- make sure to test before you do a big job with it
• use cases of breaking workflows into smaller pieces. For example, with GATK can use the new GenomicsDB format to add samples sequentially to the database.
• parallelise your code

[tavareshugo]

This could be part of a separate "advanced" module, together with #18