Skip to content

Commit 23b3062

Browse files
uwaguraUtheri WaguraUtheri Wagurayichengt900
authored
Add compile and platform yamls to compile model with FRE/2025.01 (#131)
* Add compile and platform yamls, modify experiment yaml to work with them * Added yaml README and add comments to experiment yaml explaining how to run fre make * Addressed comments in first review of PR #131, removed commented out code * fixed format --------- Co-authored-by: Utheri Wagura <Utheri.Wagura@gaea55.ncrc.gov> Co-authored-by: Utheri Wagura <Utheri.Wagura@gaea53.ncrc.gov> Co-authored-by: Yi-Cheng Teng - NOAA GFDL <143743249+yichengt900@users.noreply.github.com>
1 parent d832842 commit 23b3062

4 files changed

Lines changed: 278 additions & 19 deletions

File tree

‎yamls/NWA12/CEFI_NWA12_cobalt.yaml‎

Lines changed: 69 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,68 @@
1-
# POSTPROCESSING GUIDE
1+
# USAGE
2+
#
3+
# The following yaml is designed to compile and postprocess the ocean_monthly component of the CEFI_NWA12_COBALT_V1 experiment
4+
# Full Documentation is available here: https://noaa-gfdl.github.io/fre-cli/index.html
5+
#
6+
# 0.) Set up your environment by running "module use -a /ncrc/home2/fms/local/modulefiles" followed by "module load fre/2025.01"
7+
#
8+
# For modularity and compatibility with with cylc workflow engine, fre/2025.01 splits up the compile and post processcing process
9+
# into several sub steps.
210
#
3-
# The following yaml is designed to postprocess the ocean_monthly component of the 2024_06 version of the CEFI_NWA12_COBALT_V1 experiment
4-
# if postprocessing data from a different version of this experiment, be sure to change the FRE_STEM variable before postprocessing
11+
# COMPILE GUIDE - BAREMETAL RUN
12+
#
13+
# 1.) Create script that will be used to checkout model components and OPTIONALLY run it with the --execute flag. You can manually run the
14+
# checkout script after this step as well
15+
# fre make create-checkout -y CEFI_NWA12_cobalt.yaml -p ncrc5.intel23 -t prod --execute
16+
#
17+
# 2.) Create Makefile
18+
# fre make create-makefile -y CEFI_NWA12_cobalt.yaml -p ncrc5.intel23 -t prod
519
#
6-
# Quick start instructions and source code can be found here: https://github.com/NOAA-GFDL/fre-cli/tree/main/fre/pp
20+
# 3.) Create script that will compile the model, and OPTIONALLY run it with the --execute flag. You can manually run the
21+
# compile script after this step as well
22+
# fre make create-compile -y CEFI_NWA12_cobalt.yaml -p ncrc5.intel23 -t prod --execute
723
#
8-
# 0.) Load module on GFDL Analysis: module load fre/canopy
24+
# COMPILE GUIDE - CONTAINERS
925
#
10-
# For modularity and compatibility with with cylc workflow engine, fre/canopy splits up the post processcing process into several sub steps:
26+
# 1.) Create checkout script with the "no parallel checkouts option" - this is required for container builds
27+
# fre make create-checkout -y CEFI_NWA12_cobalt.yaml -p hpcme.2023 -t prod -npc
28+
#
29+
# 2.) Create Makefile AND a Docker file. You can then build the container by manually running the createContainer.sh
30+
# script, or by OPTIONALLY including the --execute file when creating the container
31+
# fre make create-makefile -y CEFI_NWA12_cobalt.yaml -p hpcme.2023 -t prod
32+
# fre make create-dockerfile -y CEFI_NWA12_cobalt.yaml -p hpcme.2023 -t prod --execute
1133
#
34+
# POSTPROCESSING GUIDE
35+
#
36+
# 0.) Load module on GFDL Analysis: module load fre/2025.01
37+
#
1238
# 1.) Checkout the git repo containing postprocessing scripts and related files with the folowing command:
13-
# fre pp checkout -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel22 -t prod
39+
# fre pp checkout -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod
1440
#
1541
# 2.) Combine your main yaml and experiment yamls into a single yaml, then set up the cylc-src dir with the configure-yaml command:
16-
# fre yamltools combine-yamls -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel22 -t prod -y CEFI_NWA12_cobalt.yaml
17-
# fre pp configure-yaml -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel22 -t prod -y combined-CEFI_NWA12_COBALT_V1.yaml
42+
# fre yamltools combine-yamls -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod -y CEFI_NWA12_cobalt.yaml
43+
# fre pp configure-yaml -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod -y CEFI_NWA12_cobalt.yaml --use pp
1844
#
19-
# 2a.) Fre/canopy does not automatically create the pp dir for you, so you will have to mkdir this first to pass the validator:
20-
# mkdir /archive/$USER/fre/cefi/NWA/2024_06/CEFI_NWA12_COBALT_V1/gfdl.ncrc5-intel22-prod/pp
45+
# 2a.) fre/2025.01 does not automatically create the pp dir for you, so you will have to mkdir this first to pass the validator:
46+
# mkdir /archive/$USER/fre/cefi/NWA/2024_06/CEFI_NWA12_COBALT_V1/gfdl.ncrc5-intel23-prod/pp
2147
#
2248
# 2b.) (OPTIONAL, BUT RECOMMENDED): Create a list of available tar files within your history tar archives to allow fre to catch a wider variety of errors
23-
# tar -tf /archive/$USER/fre/cefi/NWA/2024_06/CEFI_NWA12_COBALT_V1/gfdl.ncrc5-intel22-prod/history/19930101.nc.tar | grep -v tile[2-6] | sort > /home/$USER/cylc-src/CEFI_NWA12_COBALT_V1__gfdl.ncrc5-intel22__prod/history-manifest
49+
# tar -tf /archive/$USER/fre/cefi/NWA/2024_06/CEFI_NWA12_COBALT_V1/gfdl.ncrc5-intel23-prod/history/19930101.nc.tar | grep -v tile[2-6] | sort > /home/$USER/cylc-src/CEFI_NWA12_COBALT_V1__gfdl.ncrc5-intel23__prod/history-manifest
2450
#
2551
# 3.) Validate that all configuration files are correct
26-
# fre pp validate -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel22 -t prod
52+
# fre pp validate -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod
2753
#
2854
# 4.) Create the cylc-run directory containing the final version of configuration files, scripts, and output directories
29-
# fre pp install -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel22 -t prod
55+
# fre pp install -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod
3056
#
3157
# 5.) Run post processing
32-
# fre pp run -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel22 -t prod
58+
# fre pp run -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod
3359
#
3460
# 6.) To monitor the status of each post processing step, run the following command:
35-
# fre pp status -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel22 -t prod
36-
61+
# fre pp status -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod
3762

3863

3964
fre_properties:
40-
- &FRE_STEM "fre/cefi/NWA/2024_06"
65+
- &FRE_STEM "fre/cefi/NWA/ShortCanopyTest"
4166

4267
- &PP_START_YEAR "1993"
4368
- &PP_END_YEAR "2019"
@@ -47,7 +72,17 @@ fre_properties:
4772

4873
- &GRID_SPEC "/archive/acr/mom6_input/nwa12/nwa12_grid_75z.tar"
4974

50-
75+
# Compile properties
76+
- &FMSincludes "-IFMS/fms2_io/include -IFMS/include -IFMS/mpp/include"
77+
- &F2003_FLAGS " -DINTERNAL_FILE_NML -g "
78+
- &momIncludes "-Imom6/src/MOM6/pkg/CVMix-src/include"
79+
- &sisincludes "-Imom6/src/MOM6/src/framework"
80+
- &MOM6_GIT_FIX "main"
81+
- &FMS_GIT_TAG "2024.03"
82+
- &CPL_GIT_TAG "2024.03"
83+
- &INTEL "intel-classic"
84+
85+
5186
shared:
5287
directories: &shared_directories
5388
history_dir: !join [/archive/$USER/, *FRE_STEM, /, *name, /, *platform, -, *target, /, history]
@@ -77,6 +112,21 @@ shared:
77112
compileYaml: &compile_yaml
78113
platformYaml:
79114

115+
#################################################################################################
116+
# COMPILE #
117+
#################################################################################################
118+
119+
## The location for yamls
120+
build:
121+
platformYaml: platforms.yaml
122+
compileYaml: compile.yaml
123+
## User defined properties for building
124+
#release: "2024.01"
125+
126+
#################################################################################################
127+
# POST-PROCESSING #
128+
#################################################################################################
129+
80130
experiments:
81131
- name: "MOM6_SIS2_GENERIC_4P_compile_symm"
82132
pp:

‎yamls/NWA12/README.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# FRE yamls
2+
3+
The latest version of the FMS Runtime Environment uses yamls instead of xmls to execute the FRE workflow. This folder contains experiment, compile, and platform yamls that can be used with the latest version of `fre` to compile and post process the `CEFI_NWA12_Cobalt_V1` experiment. The compiled experiment can then be run in conjuction with `fre/bronx-23` and the existing xmls in the `xml/NWA12` directory.
4+
5+
The following sections assume that you are compiling and running experiments on Gaea C5, and then conducting postprocessing on GFDL's PPAN system. The `platforms.yaml` does support compilation on C6, but `bronx-23` does not support containerized runs on C6 - baremetal runs, however, are supported on C6.
6+
7+
Full documentation for `Fre/2025.01` is available [here](https://noaa-gfdl.github.io/fre-cli/usage.html)
8+
9+
## Compile the Experiment.
10+
11+
Begin by loading the appropriate modules on Gaea C5:
12+
```
13+
module use -a /ncrc/home2/fms/local/modulefiles
14+
module load fre/2025.01
15+
```
16+
17+
`fre/2025.01` supports both containerized compilation and bare-metal compilation, with very similar steps for both targets.
18+
19+
### Baremetal Compilation
20+
21+
Run the following command to create the script that will be used to checkout the model components, and optionally run the script by passing in the --execute flag.
22+
If you don't provide the --execute flag, be sure to run the script printed at the end of this command manually.
23+
```
24+
fre make create-checkout -y CEFI_NWA12_cobalt.yaml -p ncrc5.intel23 -t prod --execute
25+
```
26+
27+
Create the Makefile for the experiment:
28+
```
29+
fre make create-makefile -y CEFI_NWA12_cobalt.yaml -p ncrc5.intel23 -t prod
30+
```
31+
32+
Create the script that actually compiles the experiment, and optionally run the script by passing in the --execute flag
33+
```
34+
fre make create-compile -y CEFI_NWA12_cobalt.yaml -p ncrc5.intel23 -t prod --execute
35+
```
36+
37+
### Containerized Compilation
38+
39+
Compiling the experiment into a container is a similar process. The only changes are 1.) passing in the -npc flag during the checkout step to prevent parallel checkouts, 2.) changing you platform to `hpcme.2023` and 3.) creating a Dockerfile and container creation script instead of a compilation script. Note that the Dockerfile will compile the model within the container as part of the build process.
40+
41+
```
42+
fre make create-checkout -y CEFI_NWA12_cobalt.yaml -p hpcme.2023 -t prod -npc
43+
fre make create-makefile -y CEFI_NWA12_cobalt.yaml -p hpcme.2023 -t prod
44+
fre make create-dockerfile -y CEFI_NWA12_cobalt.yaml -p hpcme.2023 -t prod --execute
45+
```
46+
47+
## Running the Experiment
48+
49+
`fre/2025.01` does not currently support running experiments with yamls, so in order to run the compiled experiment in fre, you will have to use the existing xmls along with `fre/bronx`. Make sure the `FRE_STEM` set in the xml matches the `FRE_STEM` set in the experiment yaml.
50+
51+
**NOTE**: Since capitilized names are not allowed in Dockerfiles, the current compile yaml names the compilation experiment `mom6_sis2_generic_4p_compile_symm_yaml` instead of `MOM6_SIS2_GENERIC_4P_compile_symm_yaml` like the xml. As a result, `fre` will print a warning that it cannot find your executable when you run `frerun`, and will eventually fail if you submit the job.
52+
53+
To get around this, change the path set in the `executable` variable within the runscript printed by `frerun` to point to the exectable located in `${FRE_STEM}/mom6_sis2_generic_4p_compile_symm_yaml/exec` instead.
54+
55+
### Baremetal Run
56+
57+
Baremetal experiments can be run using the same run [instructions](https://github.com/NOAA-GFDL/CEFI-regional-MOM6/tree/main/xmls) available in the xmls directory. Be sure to load the modules described in that page before running `frerun`.
58+
59+
### Containerized Run
60+
Current xmls can be used for containerized runs, as well, so long as the following changes are made:
61+
62+
1.) Run `module load fre/bronx-23` and change the `FRE_VERSION` property of the xml to `bronx-23`.
63+
2.) Right below the `<experiment name="CEFI_NWA12_COBALT_V1" inherit="MOM6_SIS2_GENERIC_4P_compile_symm">` tag, add a tag with a path to your `.sif` singularity image file:
64+
```
65+
<container file="path/to/container/"/>
66+
```
67+
68+
With these changes, you should be able to run the experiment by calling `frerun` with an extra `--container` flag
69+
```
70+
frerun -x CEFI_NWA12_cobalt.xml -p ncrc5.intel23 -t prod CEFI_NWA12_COBALT_V1 --container
71+
```
72+
73+
74+
## Postprocessing the Experiment
75+
Like `fre make`, postprocessing is split into several subcommands to improve modularity. On PPAN, load the appropriate module:
76+
```
77+
module load fre/2025.01
78+
```
79+
80+
Checkout the git repo containing postprocessing scripts and related files with the folowing command:
81+
```
82+
fre pp checkout -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod
83+
```
84+
85+
Combine your main yaml and experiment yamls into a single yaml, then set up the cylc-src dir with the configure-yaml command:
86+
```
87+
fre yamltools combine-yamls -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod -y CEFI_NWA12_cobalt.yaml --use pp
88+
fre pp configure-yaml -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod -y CEFI_NWA12_cobalt.yaml
89+
```
90+
91+
`fre/2025.01` does not automatically create the pp dir for you, so you will have to mkdir this first to pass the validator:
92+
```
93+
mkdir /archive/$USER/fre/cefi/NWA/2024_06/CEFI_NWA12_COBALT_V1/gfdl.ncrc5-intel23-prod/pp
94+
```
95+
96+
(OPTIONAL, BUT RECOMMENDED): Create a list of available tar files within your history tar archives to allow fre to catch a wider variety of errors
97+
```
98+
tar -tf /archive/$USER/fre/cefi/NWA/2024_06/CEFI_NWA12_COBALT_V1/gfdl.ncrc5-intel23-prod/history/19930101.nc.tar | grep -v tile[2-6] | sort > /home/$USER/cylc-src/CEFI_NWA12_COBALT_V1__gfdl.ncrc5-intel23__prod/history-manifest
99+
```
100+
101+
Validate that all configuration files are correct
102+
```
103+
fre pp validate -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod
104+
```
105+
106+
Create the cylc-run directory containing the final version of configuration files, scripts, and output directories
107+
```
108+
fre pp install -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod
109+
```
110+
111+
Run post processing
112+
```
113+
fre pp run -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod
114+
```
115+
116+
To monitor the status of each post processing step, run the following command:
117+
```
118+
fre pp status -e CEFI_NWA12_COBALT_V1 -p gfdl.ncrc5-intel23 -t prod
119+
```

‎yamls/NWA12/compile.yaml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
compile:
2+
experiment: "mom6_sis2_generic_4p_compile_symm_yaml"
3+
container_addlibs:
4+
baremetal_linkerflags:
5+
src:
6+
- component: FMS
7+
paths: [FMS]
8+
repo: "https://github.com/NOAA-GFDL/FMS.git"
9+
branch: *FMS_GIT_TAG
10+
cppdefs: " -DINTERNAL_FILE_NML -g -Duse_libMPI -Duse_netCDF -Duse_yaml -DMAXFIELDMETHODS_=600" #this uses deprecated IO
11+
otherFlags: *FMSincludes
12+
13+
- component: mom6
14+
requires: [FMS]
15+
repo: "https://github.com/NOAA-GFDL/CEFI-regional-MOM6.git"
16+
branch: *MOM6_GIT_FIX
17+
paths: [mom6/src/MOM6/config_src/memory/dynamic_symmetric,
18+
mom6/src/MOM6/config_src/drivers/FMS_cap,
19+
mom6/src/MOM6/src/*/,
20+
mom6/src/MOM6/src/*/*/,
21+
mom6/src/MOM6/config_src/external/ODA_hooks,
22+
mom6/src/MOM6/config_src/external/stochastic_physics,
23+
mom6/src/MOM6/config_src/external/drifters,
24+
mom6/src/MOM6/config_src/external/database_comms,
25+
mom6/src/ocean_BGC/generic_tracers,
26+
mom6/src/ocean_BGC/mocsy/src,
27+
mom6/src/MOM6/pkg/GSW-Fortran/modules,
28+
mom6/src/MOM6/pkg/GSW-Fortran/toolbox,
29+
mom6/src/MOM6/config_src/infra/FMS2]
30+
#makeOverrides: 'OPENMP=""'
31+
otherFlags: !join [*FMSincludes," ", *momIncludes]
32+
cppdefs: "-DINTERNAL_FILE_NML -g -DINTERNAL_FILE_NML -DMAX_FIELDS_=100 -DUSE_FMS2_IO
33+
-DNOT_SET_AFFINITY -D_USE_MOM6_DIAG -D_USE_GENERIC_TRACER -DUSE_PRECISION=2 "
34+
35+
- component: sis2
36+
requires: [FMS, mom6]
37+
repo: "https://github.com/NOAA-GFDL/ice_param.git"
38+
paths: [mom6/src/SIS2/config_src/dynamic_symmetric,
39+
mom6/src/SIS2/config_src/external/Icepack_interfaces,
40+
mom6/src/SIS2/src,
41+
mom6/src/icebergs/src,
42+
mom6/src/ice_param]
43+
otherFlags: !join [*FMSincludes," ", *momIncludes," ", *sisincludes]
44+
#makeOverrides: 'OPENMP=""'
45+
cppdefs: -DINTERNAL_FILE_NML -g -DUSE_FMS2_IO
46+
47+
- component: land_null
48+
requires: [FMS]
49+
repo: "https://github.com/NOAA-GFDL/land_null.git"
50+
paths: [mom6/src/land_null]
51+
52+
- component: atmos_null
53+
requires: [FMS]
54+
repo: "https://github.com/NOAA-GFDL/atmos_null.git"
55+
paths: [ mom6/src/atmos_null ]
56+
otherFlags: !join [*FMSincludes," ", *momIncludes]
57+
cppdefs: -DINTERNAL_FILE_NML -g
58+
59+
- component: coupler
60+
repo: "https://github.com/NOAA-GFDL/FMScoupler.git"
61+
requires: [FMS, mom6, sis2, land_null, atmos_null]
62+
paths: [mom6/src/coupler/shared,
63+
mom6/src/coupler/full ]
64+
otherFlags: !join [*FMSincludes," ", *momIncludes]
65+
cppdefs: -DINTERNAL_FILE_NML -g -DUSE_FMS2_IO -D_USE_LEGACY_LAND_ -Duse_AM3_physics

‎yamls/NWA12/platforms.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
platforms:
2+
- name: ncrc5.intel23
3+
compiler: intel
4+
modulesInit: [" module use -a /ncrc/home2/fms/local/modulefiles \n","source $MODULESHOME/init/sh \n"]
5+
modules: [!join [*INTEL, /2023.2.0], fre/bronx-22, cray-hdf5, cray-netcdf]
6+
mkTemplate: !join ["/ncrc/home2/fms/local/opt/fre-commands/bronx-22/site/ncrc5/", *INTEL, ".mk"]
7+
modelRoot: !join ["/gpfs/f5/cefi/scratch/${USER}", "/", *FRE_STEM ]
8+
9+
- name: ncrc6.intel23
10+
compiler: intel
11+
modulesInit: [" module use -a /ncrc/home2/fms/local/modulefiles \n","source $MODULESHOME/init/sh \n"]
12+
modules: [!join [*INTEL, /2023.2.0], fre/bronx-22, cray-hdf5, cray-netcdf]
13+
mkTemplate: !join ["/ncrc/home2/fms/local/opt/fre-commands/bronx-22/site/ncrc6/", *INTEL, ".mk"]
14+
modelRoot: !join ["/gpfs/f6/ira-cefi/scratch/${USER}", "/", *FRE_STEM ]
15+
16+
- name: hpcme.2023
17+
compiler: intel
18+
RUNenv: [". /spack/share/spack/setup-env.sh", "spack load libyaml", "spack load netcdf-fortran@4.5.4", "spack load hdf5@1.14.0"]
19+
modelRoot: /apps
20+
container: true
21+
containerBase: "ecpe4s/noaa-intel-prototype:2023.09.25"
22+
containerBuild: "podman"
23+
containerRun: "apptainer"
24+
# NOTE: c5 and c6 make templates are currently the same. Defaulting to c6 for container runs
25+
mkTemplate: "/apps/mkmf/templates/hpcme-intel23.mk"

0 commit comments

Comments
 (0)