Investigate validation loss increasing

[marianovitasari20]

Problem / Goal

The validation loss is consistently increasing when using the current default base / sample_dataset configuration. This behavior needs to be investigated.

What does good look like?

• Validation loss should show a downward trend during training and converge or stabilize, rather than consistently increasing.

Related context

No response

[marianovitasari20]

I’m currently working on this.

[marianovitasari20]

Changing the loss function can affect training and validation loss behavior. For example, in fully deterministic mode for a fair comparison, I compared:
(1) L1 (MAE) loss: https://wandb.ai/turing-seaice/train/runs/4zdg0p8i (the loss increases)
(2) MSE loss: https://wandb.ai/turing-seaice/train/runs/v0u37f1z (the loss decreases)
(3) Weighted MSE loss (with weights from the active grid cell mask):

I need to run more experiments with additional epochs and different seeds before drawing conclusions. I haven’t finished yet. Isambard has just come back online, it was down yesterday.

[marianovitasari20]

1. L1 (MAE) loss

2. MSE loss