Fix seed on HPC

[marianovitasari20]

Problem / Goal

The add-seed branch, recently merged into main, works well on the local machine but not on the HPC (Isambard). This needs to be fixed.

What does good look like?

• The feature runs correctly on both the local machine and the HPC

Related context

https://github.com/alan-turing-institute/icenet-mp/tree/256-fix-seed

[jemrobinson]

Do you have any more information about what doesn't work @marianovitasari20 ?

[marianovitasari20]

Do you have any more information about what doesn't work @marianovitasari20 ?

This is the error message:

"RuntimeError: upsample_bilinear2d_aa_backward_out_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option..."

We could use the warn_only=True option, but that would mean not strictly enforcing deterministic behavior. I’m trying to figure out how to disable determinism for just this specific operation while keeping it enabled for everything else.

[marianovitasari20]

I think I’ve found a way to fix this, but I still need to run several experiments before concluding. At the moment, the HPC queue is very slow.

[marianovitasari20]

I have fixed the nondeterministic problem, it is now deterministic in Isambard. The results of tests using the default pipeline settings (before the merge with reprojection, which has recently been merged) with seed 555 across three different runs produce the same results.
https://wandb.ai/turing-seaice/train/runs/4zdg0p8i
https://wandb.ai/turing-seaice/train/runs/nb98deak
https://wandb.ai/turing-seaice/train/runs/bznit645

Here’s what I did:

• Enabled deterministic=True mode in both the Lightning trainer and __init__
• Used warn_only=True in __init__ only, rather than replacing deterministic=True with warn_only=True in the Lightning trainer, so training does not crash on any operations that may lack a deterministic implementation in the future
• Critical fix: Applied a monkey patch to F.interpolate to strip antialias=True, as upsample_bilinear2d_aa (used internally when antialias=True) does not have a deterministic CUDA backward pass. This was the root cause of the nondeterminism.

Please note that I have not applied a land mask or any other changes in those three runs. This is solely to test deterministic behaviour.

[marianovitasari20]

Removing _DeterministicInterpolate, the team agreed not to go with this approach. Keeping a note here for reference: if anyone wants fully deterministic behaviour in future, this function shows one way to do it. It does come with a performance cost, though personally I think it's acceptable for comparison purposes.