Add/validate pre-emption support with Submitit and wandb

[lebrice]

Rich support for pre-emption with Submitit, wandb, etc.

Requirements:

• Full support for preemption with Wandb (no duplicate runs).

Potential steps:

• Test out pre-emption support with plain hydra+submitit, identify any issues.
  • Add a (failing) test for pre-emption handling that replicates these issues, if possible.
• Configure or subclass the submitit launcher plugin if needed, to fix issues with the submitit preemption handling
• Make sure that wandb runs are also properly restarted