EvalML

Run evaluation pipelines for data-driven weather models built with Anemoi.

Getting started

1. Installation
2. Credentials setup
3. Workspace setup

Features:

• Experiments: compare model performance via standard and diagnostic verification
• Showcasing: produce visual material for specific events
• Sandboxing: generate an isolated inference development environments for any model

Quick example

To launch an experiment, prepare a config file defining your experiment, e.g.

# yaml-language-server: $schema=../workflow/tools/config.schema.json
description: |
  This is an experiment to do blabla.

dates:
  start: 2020-01-01T12:00
  end: 2020-01-10T00:00
  frequency: 60h

runs:
  - forecaster:
      mlflow_id: 2f962c89ff644ca7940072fa9cd088ec
      label: Stage D - N320 global grid with CERRA finetuning
      steps: 0/120/6
  - forecaster:
      mlflow_id: d0846032fc7248a58b089cbe8fa4c511
      label: M-1 forecaster
      steps: 0/120/6


baselines:
  - baseline:
      baseline_id: COSMO-E
      label: COSMO-E
      root: /store_new/mch/msopr/ml/COSMO-E
      steps: 0/120/6

analysis:
  label: COSMO KENDA
  analysis_zarr: /scratch/mch/fzanetta/data/anemoi/datasets/mch-co2-an-archive-0p02-2015-2020-6h-v3-pl13.zarr

locations:
  output_root: output/
  mlflow_uri:
    - https://servicedepl.meteoswiss.ch/mlstore
    - https://mlflow.ecmwf.int

profile:
  executor: slurm
  global_resources:
    gpus: 15
  default_resources:
    slurm_partition: "postproc"
    cpus_per_task: 1
    mem_mb_per_cpu: 1800
    runtime: "1h"
    gpus: 0
  jobs: 50

You can then run it with:

evalml experiment path/to/experiment/config.yaml --report

Installation

This project uses uv. Download and install it with

curl -LsSf https://astral.sh/uv/install.sh | sh

then, install the project and its dependencies with uv sync and activate the virtual environment with source .venv/bin/activate.

Credentials setup

Some experiments are stored on the ECMWF-hosted MLflow server: https://mlflow.ecmwf.int. To access these runs in the evaluation workflow, you need to authenticate using a valid token. Run the following commands once to log in and obtain a token:

uv pip install anemoi-training --no-deps
anemoi-training mlflow login --url https://mlflow.ecmwf.int

You will be prompted to paste a seed token obtained from https://mlflow.ecmwf.int/seed. After this step, your token is stored locally and used for subsequent runs. Tokens are valid for 30 days. Every training or evaluation run within this period automatically extends the token by another 30 days. It’s good practice to run the login command before executing the workflow to ensure your token is still valid.

Workspace setup

By default, data produced by the workflow will be stored under output/ in your working directory. We suggest that you set up a symlink to a directory on your scratch:

mkdir -p $SCRATCH/evalenv/output
ln -s $SCRATCH/evalenv/output output

This way data will be written to your scratch, but you will still be able to browse it with your IDE.

If you are using VSCode, we advise that you install the YAML extension, which will enable config validation, autocompletion, hovering support, and more.