[WIP] Standard baseline configs

[jemrobinson]

This PR adds some baseline configs for the model comparisons discussed in #323.

TL;DR

• nothing is beating persistence on a two day forecast (perhaps we want to look at e.g. 7 days?)
• some very weird aliasing features on piecewise and vit that make me feel that we could better-optimise these defaults
• is combining train and validation on the metric plots actually useful or is it unnecessary noise?

Models

00: naive-unet-naive

• 50 epochs
• northern hemisphere: glorious-morning-882
• southern hemisphere: deft-sponge-888

01: persistence

• 50 epochs
• northern hemisphere: frosty-voice-895
• southern hemisphere sunny-bird-897

02: cnn-unet-cnn

• 50 epochs
• northern hemisphere: zesty-cosmos-894
• southern hemisphere sleek-sea-896

03: cnn-vit-cnn

• 50 epochs
• northern hemisphere: clear-violet-885
• southern hemisphere fragrant-paper-891

04: ddpm

• 34/35 epochs (hit 24h walltime limit)
• northern hemisphere: solar-donkey-886
• southern hemisphere toasty-rain-893

05: piecewise-unet-piecewise-5336004.out

• 50 epochs
• northern hemisphere: lemon-deluge-887
• southern hemisphere lyric-gorge-892

Metrics

RMSE

SIE Error

MAE

[IFenton]

Couple of quick thoughts.

nothing is beating persistence on a two day forecast (perhaps we want to look at e.g. 7 days?)

Looking at 7 days seems sensible anyway, as you can then get an idea of how the forecast accuracy develops over time. (And as you say it is more likely to beat persistence).

As we've discussed a bit elsewhere (#323), doing a bit of work to optimise the models before we do this seems sensible. E.g. Erin's trained ViT model (https://wandb.ai/turing-seaice/evaluate/runs/0r9jomqk) does beat persistence

[jemrobinson]

For reference, @erinuclkwon's model was using

n_forecast_steps: 7
n_history_steps: 3

which is what the next round of testing will use.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
icenet_mp
model_service.py					31-33, 43-45, 213-221
icenet_mp/callbacks
ema_weight_averaging_callback.py					29-30, 36-37, 41-42
metric_summary_callback.py					29, 62
icenet_mp/compatibility
__init__.py
icenet_mp/data_processors
data_downloader.py					190-197, 202-212, 231-235, 245-249
icenet_mp/models
base_model.py					95-96
persistence.py
icenet_mp/models/common
conv_block_downsample.py
conv_block_upsample.py
icenet_mp/models/decoders
cnn_decoder.py
icenet_mp/models/encoders
cnn_encoder.py
icenet_mp/models/processors
vit.py					41-42
icenet_mp/types
__init__.py
simple_datatypes.py
Project Total

This report was generated by python-coverage-comment-action