Skip to content

README.md

Latest commit

 

History

History
206 lines (145 loc) · 10.7 KB

README.md

File metadata and controls

206 lines (145 loc) · 10.7 KB

CMOR Subtools in fre-cli

These notes describe how to use the refactored fre cmor tool for rewriting climate model output with CMIP-compliant metadata, also known as the process of "CMORization". Both CMIP6 and CMIP7 workflows are supported. The examples below use CMIP6 configuration, but the tool can be used for any MIP era as long as the right configuration files are accessible.

Documentation

Comprehensive documentation on fre cmor is available at the official fre-cli docs.

These subtools are highly dependent on the PCMDI/cmor module. As such, the documentation is good to have nearby, and can be found here.

Getting Started

Below is recommended reading for understanding how to use this tool, which requires ample configuration from outside this repository, and the user directly.

External Configuration

  • CMOR rewrites ("CMORizes") the target input files based on configuration in MIP tables, e.g. the cmip6-cmor-tables.
  • A controlled vocabulary must be present, and is usually associated with the MIP tables, e.g. the CMIP6_CVs.

Required User Configuration

  • A variable list as a JSON dictionary is required, to assist with targeting the right input files for CMORization. Each entry is a key/value pair where the key is the modeler's variable name (used in the filename AND expected inside the file) and the value is the corresponding MIP table variable name. The key and value are often the same, but may differ when the modeler uses a different name than the MIP standard. The variable name in the filename must match the variable name inside the file -- if they differ, fre cmor run will raise an error. An example in the repository is included here. Additionally, see fre cmor varlist and the --opt_var_name flag for more information.

  • An experiment configuration file as a JSON dictionary, an example provided by PCMDI is included in the repository here. The file must contain info on the targeted input files' calendar and grid attributes, as well as the desired output directory structure definition.

  • If desired, a fre.cmor-flavored yaml file to encode many fre cmor run calls in one fre cmor yaml call, with pp and model flavored yaml files nearby. See fre cmor yaml. The exact structure of these configuration files is currently being actively developed, but current working(ish) examples are available in this repository: AM5-oriented ex., ESM4-oriented ex.

Subcommands & Usage Examples

fre cmor is a command group with several subcommands.

# entry-point to all subcommands
fre cmor --help

# main engine of the sub module, does the rewriting
fre cmor run --help

# higher-level routine to encode many 'run' calls across tables, grids, components, etc.
fre cmor yaml --help

# convenience / helper function for exploring external table configuration files
# helpful for code development, not designed for deployment usage
# open an issue on github if you'd like more functionality here
fre cmor find --help

# convenience / helper function for creating variable lists
# helpful for code development, not designed for deployment usage
# open an issue on github if you'd like more functionality here
fre cmor varlist --help

# generate a cmor yaml configuration file from a post-processing directory tree
# scans pp components and cross-references against MIP tables
fre cmor config --help

Below are short descriptions and a practical example for each of the fre cmor subcommands. For more information, consult each respective tool's --help output at the command line, and consult the official fre-cli docs.

fre cmor run

Rewrite climate model output files in a target input directory to be CMIP-compliant. This rewriting process is referred to as "CMORization" in the MIP-package ecosystem. External configuration is required in the form of MIP tables, and a controlled vocabulary to work. Additionally, the user must provide a variable list and certain pieces of metadata associated with the experiment (e.g. calendar). This tool DOES NOT use yaml configuration, and should be considered, i.e. yaml-naive and/or independent.

Example and Description

fre cmor run --run_one --grid_label gr --grid_desc FOO_BAR_PLACEHOLD --nom_res '10000 km' \
             --indir fre/tests/test_files/ocean_sos_var_file/ \
             --varlist fre/tests/test_files/varlist \
             --table_config fre/tests/test_files/cmip6-cmor-tables/Tables/CMIP6_Omon.json \
             --exp_config fre/tests/test_files/CMOR_input_example.json \
             --outdir fre/tests/test_files/outdir

Note- the target input file here is only created after running pytest fre/cmor/tests fre/tests/test_fre_cmor_cli.py. These tests effectively contain this exact example, which is run automatically in as a unit-test in this repository's CI/CD workflows.

Here, fre cmor run will process one file before exiting (--run_one), use the input gridding information metadata provided by the --grid_label, --grid_desc, and --nom_res arguments. --table_config is pointing to a specific external configuration table known as a MIP table, while --exp_config will contain the requisite information on output directory structure, calendar, and more. --varlist specifies a JSON dictionary mapping modeler variable names (keys, used for filename targeting) to MIP table variable names (values, used for CMIP metadata lookups). The output directory structure's final location will be at --outdir.

fre cmor yaml

Target several (or one) directories in a FRE-workflow output directory structure with varying MIP table targets, grids, and variable lists. This tool requires all the configuration files that fre cmor run requires, and more, in the form of FRE-flavored yaml files. An input triplet of --experiment / --platform / --target is required to make sense of the input yaml files.

Example and Description

fre -v cmor yaml --run_one --dry_run --output combined.yaml \
                 --yamlfile fre/yamltools/tests/AM5_example/am5.yaml \
				 --experiment c96L65_am5f7b12r1_amip \
				 --platform ncrc5.intel \
				 --target prod-openmp

Note- this example is also from this repository's unit-tests.

Here, fre cmor yaml, targeting the am5.yaml model-yaml, will seek out relevant configuration from the CMOR-yaml under the AM5_example directory structure, locked within CMOR, grid, and pp yaml files. The combined output dictionary is saved to the -o flag target, combined.yaml, which can be utilized for debugging any issues that come up while the code runs. --dry_run here means the cmor run tool is not called, and instead, prints out the call to run subtool that is created from the input configuration files. --run_one means only the first call is printed, then the routine will finish. The --experiment/-e, --platform/-p, and --target/-t triplet of flags, characteristic to FRE, assist with the selection of configuration information pointed to by -y.

In dry-run mode, the default output shows the equivalent fre cmor run CLI invocation. Pass --no-print_cli_call to instead print the Python cmor_run_subtool(...) call, which can be useful for debugging.

fre cmor find

Search MIP tables for variable definitions and print relevant information to screen. A single variable as a string, or a list of variablescan be accepted as input.

Example and Description

fre -v cmor find --table_config_dir fre/tests/test_files/cmip6-cmor-tables/Tables/ \
                 --opt_var_name sos

Here, if a MIP table under cmip6-cmor-tables/Tables contains a variable entry for sos, then the MIP table and the metadata for sos within will be printed to screen by this call.

fre cmor varlist

Generate a variable list from NetCDF files in a target directory. Only works if the targeted files have names containing the variable in the right spot. Each entry in the output list is a key/value pair mapping the modeler's variable name to itself by default (e.g., "sos": "sos"). Edit the values as needed to map to the correct MIP table variable names.

Example and Description

fre cmor varlist --dir_targ fre/tests/test_files/ocean_sos_var_file/ \
                 --output simple_varlist.txt
cat simple_varlist.txt # shows the result

Here, simple_varlist.txt will be a simple JSON file, containing a dictionary with the variable(s) sos and sosV2 listed, each mapping to itself (e.g., "sos": "sos"). Review and edit the values to map to the correct MIP table variable names. Note that sosV2 is a made-up variable for software testing purposes only.

Optionally, pass --mip_table with a path to a MIP table JSON file to filter the generated variable list so that only variables present in the MIP table are included.

fre cmor config

Generate a CMOR YAML configuration file from a post-processing directory tree. This scans the pp directory for available components and time-series data, cross-references found variables against MIP tables, creates per-component variable list files, and writes the structured YAML that fre cmor yaml expects.

Example and Description

fre cmor config --pp_dir /path/to/pp \
                --mip_tables_dir /path/to/cmip7-cmor-tables/tables \
                --mip_era cmip7 \
                --exp_config /path/to/CMOR_input.json \
                --output_yaml cmor_config.yaml \
                --output_dir /path/to/cmor_output \
                --varlist_dir /path/to/varlists \
                --freq monthly --chunk 5yr --grid gn \
                --calendar noleap

Here, fre cmor config scans --pp_dir for components (e.g. ocean_monthly_1x1deg), finds time-series files under each, and matches their variable names against all MIP tables in --mip_tables_dir. For each component/table combination where variables match, a variable list JSON file is written to --varlist_dir, and a corresponding entry is added to the output YAML at --output_yaml. The generated YAML can then be fed directly to fre cmor yaml. Pass --overwrite to regenerate existing variable list files.