Skip to content

oifs_howto_run_experiments.md

Latest commit

 

History

History
166 lines (120 loc) · 10.8 KB

oifs_howto_run_experiments.md

File metadata and controls

166 lines (120 loc) · 10.8 KB

How to run global forecast experiments

This document describes how to run a global forecast experiment with the OpenIFS model and how to postprocess the fields and diagnostics that are generated by the model during the simulation.

The worked example is one of the forecast experiments of the Storm Karl case study, using a horizontal N128 linear reduced Gaussian model grid (T255) and 91 vertical model levels (ab2a).

Set up the experiment directory

An example forecast experiment has been prepared for OpenIFS 48r1. The experiment ID is ab2a and you can download the tarball from here: https://openifs.ecmwf.int/data/experiments/48r1/2016-09-25_Karl/ab2a.tar.gz

  • Set variable OIFS_EXPT in oifs-config.edit_me.sh to point to a suitable location for your model experiments.
  • Extract the example forecast experiment ab2a.tar.gz to this location. We will refer to directory $OIFS_EXPT/ab2a/2016092500/ as the experiment directory.

NOTE: The OIFS_EXPT location for model experiments will often be in a different location from the model installation OIFS_HOME. In general, you will require more disk space for experiments, depending on the model grid resolution, the duration of the forecast experiment and the output frequency of model results.

source oifs-config.edit_me.sh  # always do this first
cd $OIFS_EXPT
wget https://openifs.ecmwf.int/data/experiments/48r1/2016-09-25_Karl/ab2a.tar.gz
tar -xvzf ab2a.tar.gz
  • Ensure the namelist files for the atmopsheric model (fort.4) and for the wave model (wam_namelist) are found in the experiment directory. Backup copies should remain in the ecmwf subfolder:
cd $OIFS_EXPT/ab2a/2016092500
cp ./ecmwf/fort.4 .
cp ./ecmwf/wam_namelist .
  • Copy the model run scripts and the experiment configuration file into the experiment directory.
    • oifs-run: This is a generic run script which executes the binary model program file.
    • run-oifs.ecmwf-hpc2020.job: This is a wrapper script that calls oifs-run and submits a non-interactive job to the batch scheduler; this script is an example for use on the ECMWF hpc2020 and you may need to adjust it for your local HPC system.
    • exp-config.h: This is the experiment configuration file that determines settings for your experiment; this file is read by oifs-run.
cd $OIFS_EXPT/ab2a/2016092500
cp $OIFS_RUN_SCRIPT/oifs-run .
cp $OIFS_RUN_SCRIPT/exp-config.h .
cp $OIFS_RUN_SCRIPT/run-oifs.ecmwf-hpc2020.job .

Customise the experiment parameters

Here we describe how you can customise the way how OpenIFS will run the forecast experiment.

Customising the namelist:

  • You can edit the atmospheric model namelist file fort.4. It contains Fortran namelists which control model settings and switches.
  • An important switch is CSTOP in namelist NAMRIP. Set this to the desired duration of the forecast experiment.
  • Experiment ab2a can be run for up to 144 hours (6 days) by setting CSTOP='h144'.

Customising the experiment configuration file:

  • You should customise the exp-config.h file which determines the settings used for the experiment.
  • The oifs-run script will read the settings from this file.
  • Alternatively, the settings can be passed to the oifs-run script via command line parameters, which takes precedence over the exp-config.h settings.

NOTE: The preference should be to always set up an exp-config.h for each experiment. If no exp-config.h file is found in the experiment directory (and no command line parameters are provided when calling oifs-run), then oifs-run will revert to its own default values which are not appropriate.

exp-config.h:

#--- required variables for this experiment:
 
# this is specific for each experiment:
OIFS_EXPID="ab2a"       #  your experiment ID
OIFS_RES="255"          #  the spectral grid resolution (here: T255)
OIFS_GRIDTYPE="l"       #  the grid type, either 'l' for linear reduced grid, 

# note: use of the batch job script will overwrite these values:
OIFS_NPROC=8            #  the number of MPI tasks
OIFS_NTHREAD=4          #  the number of OpenMP threads

# postprocessing is optional but recommended:
OIFS_PPROC=true         #  enable postprocessing of model output after the model run
OUTPUT_ROOT=$(pwd)      #  folder where pproc output is created (only used if 
                        #  OIFS_PPROC=true). In this example an output folder is 
                        #  created in the experiment directory.

LFORCE=true             #  overwrite existing symbolic links in the experiment directory
LAUNCH=""               #  the platform specific run command for the MPI environment
                        #  (e.g. "mpirun", "srun", etc). Setting this variable overwrites 
                        #  any platform-specific default run commands defined in oifs-run
 
#--- optional variables that can be set for this experiment:
 
#OIFS_NAMELIST='my-fort.4'               # custom atmospheric model namelist file
#OIFS_EXEC="<custom-path>/ifsMASTER.DP"  # model executable to be used for this experiment

Running the experiment

After all edits to the namelists (fort.4) and to the experiment configuration file (exp-config.h) have been completed the model run can be started. Depending on the available hardware the experiment can either be run interactively or as a batch job.

Running a batch job:

This method is the preferred way to run OpenIFS, as it is more efficient and it allows more flexibility in using the available hardware resources.

  • A job wrapper script needs to be used to call oifs-run; that script must be customised for the locally available batch scheduler and HPC system.
  • We include an example job wrapper script run-oifs.ecmwf-hpc2020.job in $OIFS_RUN_SCRIPT, which is suitable for the ECMWF hpc2020 HPC. This system uses the slurm batch job scheduler.
    • As described above, the job wrapper script is copied to the experiment directory because it needs to be located here to run an experiment.
  • run-oifs.ecmwf-hpc2020.job needs to be edited with the following essential and optional changes:
    • Initially run-oifs.ecmwf-hpc2020.job sets the PLATFORM_CFG variable as follows:
    # set OpenIFS platform environment:
PLATFORM_CFG="/path/to/your/config/oifs-config.edit_me.sh"
    • It is important to change "/path/to/your/config/oifs-config.edit_me.sh" to the actual path for the oifs-config.edit_me.sh, e.g., "$HOME/openifs/oifs-config.edit_me.sh"
    • You will need to adjust the batch scheduler header lines as required for your local system.
    • The run-oifs.ecmwf-hpc2020.job script will update the entries in exp-config.h with the correct settings dermined by the batch job headers.
    • For information, the LAUNCH command for batch job submission is set to "srun" without any further options, because all required parallel environment settings are provided through the slurm script headers.

Once you have made the appropriate changes the job can be submitted:

# run as slurm batch job:
source oifs-config.edit_me.sh
cd $OIFS_EXPT/ab2a/2016092500
sbatch ./run-oifs.ecmwf-hpc2020.job

NOTE: The job wrapper script will read the exp-config.h file and adopt the selected values. The exceptions are LAUNCH, which is set to "srun" for batch jobs, and OIFS_NPROC & OIFS_NTHREAD for which values from the batch job headers are used. The job wrapper script modifies the exp-config.h file accordingly prior to calling the oifs-run script.

Running interactively:

This example is shown for the ECMWF hpc2020, where running the model interactively should work for lower grid resolutions up to T255L91.

  • In order to run the experiment interactively, execute the oifs-run script from the command line in your terminal.
  • If no command line parameters are provided with the oifs-run command, then the values from the exp-config.h will be used.
  • In exp-config.h set OIFS_NPROC=8 and OIFS_NTHREAD=4.
  • In exp-config.h the LAUNCH variable should remain empty, i.e. LAUNCH="" and no --runcmd parameter should be provided in the command line.

The oifs-run script will in this case use its default launch parameters: srun -c${OIFS_NPROC} --mem=64GB --time=60 which will work fine with OIFS_NPROC=8 for experiment ab2a.

# run interactively:
cd $OIFS_EXPT/ab2a/2016092500
./oifs-run

Postprocessing the model output

Postprocessing creates a unique output folder and groups all model output fields and diagnostics into individual GRIB files with ascending forecast time step. Also, a copy of the atmospheric model namelist file fort.4, as well as the ifs.stat and NODE.01_001 log files are moved into the output folder.

Postprocessing can be done either directly by oifs-run or in a separate step by running the run-pproc.ecmwf-hpc2020.job script.

Postprocessing with oifs-run:

If in the exp-config.h file the OIFS_PPROC variable has been set to true (or if the --pproc command line parameter was used) then the model output in the experiment directory is further processed after completing the model run.

  • In this case the script will generate a folder called output_YYYMMDD_HHMMSS, with YYYYMMDD being the current date and HHMMSS the current time.
  • This avoids accidental modification or overwriting of any previous results when the model experiment is repeated.
  • For convenience a symbolic link output is set to the most recently generated model output. If the model run is repeated and a new output_YYYMMDD_HHMMSS folder is generated, the symbolic link will be updated to point to the most recent output folder.
  • The variable OUTPUT_ROOT in exp-config.h determines where this ouput folder will be created. The default location is inside the experiment directory, but when assigning another path to OUTPUT_ROOT this could be created elsewhere.

Postprocessing with a separate script:

The postprocessing task is carried out by a single processor. During this time the batch job's computing resources will remain allocated for oifs-run which can be inefficient.

As an alternative we include the script $OIFS_HOME/scripts/exp3d/run-pproc.ecmwf-hpc2020.job which completes the postprocessing in isolation after the model forecast experiment has ended. This script can be submitted as a serial job.

If you want to use this script follow these steps:

  • In exp-config.h set OIFS_PPROC=false and run the forecast experiment to completion.
  • Copy the run-pproc script to the OIFS_EXPT location: cp $OIFS_HOME/scripts/exp3d/run-pproc.ecmwf-hpc2020.job $OIFS_EXPT
  • Edit the run-pproc script and update the variable PLATFORM_CFG as required for your installation.
  • The script allows postprocessing of multiple experiments in sequence. Ensure the experiment IDs of all experiments you wish to process are included in the EXP_LIST heredoc.

NOTE: In our worked example you need the line entry ab2a.

  • Submit the script with the sbatch command (or as appropriate for your system) as a serial job.