Mask out land (and possibly some of the sea too?) in both the loss function and the model outputs

[jemrobinson]

Related issues:

• Generate our own masks #207
• Are the land masks used during training in our encode-process-decode pipeline? #174
• Add land masks to data prep #134

[jemrobinson]

@marianovitasari20 and @IFenton looking at applying mask in the loss function. Will test this locally.

[IFenton]

Currently looking at how to access the land mask data - see comments in #207

[marianovitasari20]

Previously, there was a problem with nonsensical results (https://wandb.ai/turing-seaice/evaluate/runs/0f534mrm) due to a swap between land and ocean values. Now I have fixed it, and it produces correct results.

Results after applying a land mask as sample weights in the loss function:
With: https://wandb.ai/turing-seaice/evaluate/runs/p5kynyr7
Without: https://wandb.ai/turing-seaice/evaluate/runs/q9b55ysz

Adding a land mask appears to improve RMSE and SIE errors (particularly SIE), while slightly increasing MAE, though not significantly.

I have only tested with U-Net on the Southern Hemisphere. I also attempted DDPM, but it was too slow, so I cancelled the run on my local machine (as we do not have HPC at the moment). I will revisit this later once we have our compute resources back. Further testing will be conducted if improved results are observed with the active grid cell mask.

I am currently working on replacing the land mask with an active grid cell mask. This work is still in progress and not yet complete, as we recently had holidays (4–7 April).

[marianovitasari20]

I retrained and re-evaluated the model with and without a landmask (not an active grid), using a naive U-Net, with both different and shared plot scales.

no landmask

• Different scales: https://wandb.ai/turing-seaice/evaluate/runs/lmfq4oe5
• Shared scale: https://wandb.ai/turing-seaice/evaluate/runs/jvfzx5tq

with landmask

• Different scales: https://wandb.ai/turing-seaice/evaluate/runs/wj6e6bm5
• Shared scale: https://wandb.ai/turing-seaice/evaluate/runs/82ig9wqf

In this rerun, it is clear that using a landmask in the loss function improves performance. However, I’m still wondering why the shared-scale plots without a landmask look so bizarre. I’m not sure whether this is a bug in the plotting part of the pipeline or something else. I’m currently rerunning the experiment without a landmask to confirm.

[jemrobinson]

Are your links above the correct way around @marianovitasari20? If so it looks like the thing we think should improve performance (applying the landmask) is making all our metrics worse. I think we really need to understand why this is happening and what it means.

N.B. I think absolute values of MAE and SIEError can't be compared as you're running them over a different number of grid cells. Should we be normalising these by area?

[marianovitasari20]

Hi James, sorry, I misread the number. In the above case (20 epochs), no landmask performs better, but visually, with landmask version is better.
With a different number of epochs, it yields different results. For example, here with more epochs (75 epochs), using landmask, we get better (lower) SIE error.

I’m currently running more experiments to analyse this.

[jemrobinson]

Yeah, I'd expect that more epochs would give a better result, so perhaps we just need a higher n_epochs for baseline comparison?

[marianovitasari20]

With more epochs (91 epochs), the model using landmarks now performs better across all metrics:

[marianovitasari20]

with AGC as loss function, using "sample", 91 epochs: https://wandb.ai/turing-seaice/evaluate/runs/ettjg9ep

[jemrobinson]

What is AGC?

[marianovitasari20]

With AGC as the loss function, but change "sample" to "base", 11 epochs:

• https://wandb.ai/turing-seaice/evaluate/runs/jnufpb4i

Rerun with the same settings to see variation:

• https://wandb.ai/turing-seaice/evaluate/runs/g48npbr2

After changing to 432×432, visualization-wise, those two are actually good, but there is a bit of noise in the upper part of the image. Not sure what caused it.

SIEerror varied too much. I suggest we use the same seeds across different experiments for a fair comparison.

[marianovitasari20]

What is AGC?

Active Grid Cell, sorry :))