Question on model vs model e3sm_diags task zppy configuration

[chengzhuzhang]

Question criteria

• I added a descriptive title here.
• I searched the zppy GitHub Discussions to find a similar question and didn't find it.
• I searched the zppy documentation.

What is the deadline?

NA

Describe your question

@BunnyVon and I are working on a zppy configuration for compare a BGC run with a v2.amip run. The configuration file can be viewed here: /home/ac.zhang40/test_zppy/test_from_sha_618.cfg

We had this problem that the diurnal_cycle set from e3sm_diags model-vs-model run is not working. The error was '/lcrc/group/e3sm/ac.sfeng1/E3SM_output/v2.LR.amip_0101/post/atm/180x360_aave/clim_diurnal_8xdaily/30yr/v2.LR.amip_0101.eam.h3_*_1985??_2014??_[climo.nc](http://climo.nc/)': No such file or directory
However the diurnal files are saved under a path: /lcrc/group/e3sm/ac.sfeng1/E3SM_output/v2.LR.amip_0101/post/atm/180x360_aave/clim_atm_monthly_diurnal_8xdaily_180x360_aave/30yr, also the prefix for the file should be v2.LR.amip_0101.eam.h4, instead of v2.LR.amip_0101.eam.h3 as zppy is expecting. @forsyth2 not sure if there is a way to point to the proper files in this case?

Are there are any possible answers you came across?

No response

What machine were you running on?

Chrysalis

Environment

e3sm_unified 1.10.

What command did you run?

zppy -c /home/ac.zhang40/test_zppy/test_from_sha_618.cfg
(The script is currently working with `diurnal_cycle` removed from sets.)

Copy your cfg file

[default]
case = v2.LR.BGC-LNDATM.CONTRL
environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
input = "/lcrc/group/e3sm2/ac.sfeng1/E3SM_Simulations/20240315.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis"
output = "/lcrc/group/e3sm2/ac.zhang40/tests/E3SM_Simulations/20240315.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis"
partition = "debug"
# qos = "regular"
qos = "debug"
www = "/lcrc/group/e3sm/public_html/diagnostic_output/ac.zhang40/tests/20240315.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis"
walltime = "4:00:00"

[climo]
active = True
years = "1985:2014:30",

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  mapping_file = map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  frequency = "monthly"

  [[ atm_monthly_diurnal_8xdaily_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h3"
  mapping_file = map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  vars = "PRECT"
  frequency = "diurnal_8xdaily"

  [[ land_monthly_climo ]]
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = archive/lnd/hist
  vars = ""

[ts]
active = True
years = "1985:2014:10",


  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,ICEFRAC,LANDFRAC,OCNFRAC,PS,CLDICE,CLDLIQ,T,AODDUST"
# Needed for mixed-phase partition
#  vars = "LANDFRAC,CLDICE,CLDLIQ,T" 
  ts_fmt = "cmip"

  [[ atm_daily_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h1"
  frequency = "daily"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  # Needed for Wheeler Kiladis
  vars = "FLUT,PRECT,U850"

  [[ atm_monthly_glb ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = "glb"

  [[ land_monthly ]]
  input_subdir = "archive/lnd/hist"
  input_files = "elm.h0"
  frequency = "monthly"
  #mapping_file = ""
  mapping_file = map_r05_to_cmip6_180x360_aave.20231110.nc
  vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR"
  extra_vars = "landfrac"
  ts_fmt = "cmip"
#
  [[ rof_monthly ]]
  input_subdir = "archive/rof/hist"
  input_files = "mosart.h0"
  mapping_file = ""
  frequency = "monthly"
  vars = "RIVER_DISCHARGE_OVER_LAND_LIQ"
  extra_vars = 'areatotal2'

  [[ lnd_monthly_glb ]]
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  mapping_file = "glb"
  vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILWATER_10CM,TSA,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR"

[e3sm_diags]
active = True
multiprocessing = True
partition = "debug"
nodes = 8
years = "1985:2014:30",
ts_num_years = 10
ref_start_yr = 1985
ref_final_yr = 2014

   [[atm_monthly_180x360_aave_mvm]]
   years = "1985-2014", 
   ref_years = "1985-2014",
   ref_start_yr = 1985
   ref_final_yr = 2014
   ts_num_years = 10   
   ts_num_years_ref = 5
   ts_subsection = "atm_monthly_180x360_aave"
   input_files = "eam.h3"
   short_name = 'BGCv2LNDATM'
   grid = '180x360_aave'
   ref_name = ' v2.LR.amip_0101'
   short_ref_name = 'v2.LR.amip'
  #  tag = 'v3alpha04i-COARE_vs_CTL'
   tag = "model_vs_model"
   run_type = "model_vs_model"
   reference_data_path = '/lcrc/group/e3sm/ac.sfeng1/E3SM_output/v2.LR.amip_0101/post/atm/180x360_aave/clim'
   climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
   #climo_diurnal_subsection = "clim_atm_monthly_diurnal_8xdaily_180x360_aave"
   climo_diurnal_frequency = "diurnal_8xdaily"
   climo_subsection = "atm_monthly_180x360_aave"
   #sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","diurnal_cycle","zonal_mean_2d_stratosphere","aerosol_budget"
    sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","zonal_mean_2d_stratosphere","aerosol_budget"
   diff_title = 'Difference'
   output_format_subplot = "pdf",

   [[ atm_monthly_180x360_aave ]]
 #  environment_commands = "source /home/ac.zhang40/y/etc/profile.d/conda.sh; conda activate edv2110"
   short_name = 'BGCv2LNDATM'
   grid = '180x360_aave'
   reference_data_path = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/climatology'
   obs_ts = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series'
   dc_obs_climo = '/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/climatology'
   climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
   climo_diurnal_frequency = "diurnal_8xdaily"
   ts_daily_subsection = "atm_daily_180x360_aave"
   sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","diurnal_cycle","zonal_mean_2d_stratosphere","aerosol_aeronet","tropical_subseasonal",
 #"mp_partition","aerosol_budget",
 #  sets="tropical_subseasonal",
   output_format_subplot = "pdf",

   [[ lnd_monthly_mvm_lnd ]]
   # Test model-vs-model using the same files as the reference
   #environment_commands = "source /home/ac.zhang40/y/etc/profile.d/conda.sh; conda activate edv290"
   grid = 'native'
   climo_subsection = "land_monthly_climo"
   diff_title = "Difference"
   partition = "compute"
   qos = "regular"
   short_name = BGCv2LNDATM
   ref_name = "20230717_CBGCv2.r05.chrysalis.I20TRGSWCNP"
   ref_start_yr = 1985
   ref_final_yr = 2014
   ref_years = "1985-2014",
   reference_data_path = "/lcrc/group/e3sm/ac.sfeng1/E3SM_output/20230717_CBGCv2.r05.chrysalis.I20TRGSWCNP/zppy.5yrs/post/lnd/180x360_aave/clim"
   run_type = "model_vs_model"
   sets = "lat_lon_land",
   short_ref_name = "BGCv2LND.GSW"
   swap_test_ref = False
   tag = "model_vs_model"
   ts_num_years_ref = 30

What jobs are failing?

No response

What stack trace are you encountering?

No response

[forsyth2]

Summary

(scroll left-right)

	Actual	zppy errors because it expects	As defined for [[atm_monthly_180x360_aave_mvm]] subtask in the given cfg	As defined for the[[ atm_monthly_diurnal_8xdaily_180x360_aave ]] subtask in the given cfg
Diurnal files saved under	clim_atm_monthly_diurnal_8xdaily_180x360_aave	clim_diurnal_8xdaily	climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave", climo_diurnal_frequency = "diurnal_8xdaily"	N/A
input file name	eam.h4	eam.h3	input_files = "eam.h3"	input_files = "eam.h3"

The input file name error is easy then. You need to set input_files = "eam.h4" for the [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] subtask (not the [[atm_monthly_180x360_aave_mvm]] subtask).

As for the the file location, it looks like zppy is using climo_diurnal_frequency rather than climo_diurnal_subsection.

Where is climo_diurnal_subsection used?

In https://github.com/E3SM-Project/zppy/blob/main/zppy/e3sm_diags.py:

            if "diurnal_cycle" in c["sets"]:
                if c["dc_obs_climo"] == "":
                    c["dc_obs_climo"] = c["reference_data_path"]
                diurnal_cycle_subtask_name = c["climo_diurnal_subsection"]
                if diurnal_cycle_subtask_name in config["climo"].keys():
                    if (
                        "input_files"
                        in config["climo"][diurnal_cycle_subtask_name].keys()
                    ):
                        diurnal_cycle_input_files = config["climo"][
                            diurnal_cycle_subtask_name
                        ]["input_files"]
                        c["climo_diurnal_input_files"] = diurnal_cycle_input_files

So, climo_diurnal_subsection is used to tell us which climo subtask's input files we should use.

            if "diurnal_cycle" in c["sets"]:
                dependencies.append(
                    os.path.join(
                        scriptDir,
                        "climo_%s_%04d-%04d.status"
                        % (c["climo_diurnal_subsection"], c["year1"], c["year2"]),
                    )
                )

It also is used to tell us which climo subtasks need to finish before diags can run.

Where is climo_diurnal_frequency used?

In https://github.com/E3SM-Project/zppy/blob/main/zppy/templates/e3sm_diags.bash:

{% if run_type == "model_vs_obs" %}
climo_diurnal_dir_primary=climo_{{ climo_diurnal_frequency }}
{% elif run_type == "model_vs_model" %}
climo_diurnal_dir_primary=climo_{{ climo_diurnal_frequency }}_test
{%- endif %}
# Create local links to input diurnal cycle climo files
climo_diurnal_dir_source={{ output }}/post/atm/{{ grid }}/clim_{{ climo_diurnal_frequency }}/{{ '%dyr' % (year2-year1+1) }}
create_links_climo_diurnal ${climo_diurnal_dir_source} ${climo_diurnal_dir_primary} ${case} ${Y1} ${Y2} 3

So, climo_diurnal_frequency is used to tell us where to place the links to the diurnal climatology files, as well as where to get the output from the diurnal climatology subtask.

It seems that this may be a bug. Perhaps the line should be this instead:

climo_diurnal_dir_source={{ output }}/post/atm/{{ grid }}/clim_{{ climo_diurnal_subsection }}/{{ '%dyr' % (year2-year1+1) }}

That is, apparently the diurnal climatology subtask outputs files into a directory based on its subsection name (atm_monthly_diurnal_8xdaily_180x360_aave) rather than its frequency parameter (frequency = "diurnal_8xdaily")

[chengzhuzhang]

@forsyth2 thank you for troubleshooting. I think the complication is that for model vs model case, the test case has diurnal output in h3, however the reference case has output in h4. My understanding is that this configuration file is having difficulty to locate the reference case files which was prepared by another zppy run. The parameters: input_files and climo_diurnal_section/frequency are used for specifying information for the test case, right?

[forsyth2]

@BunnyVon @chengzhuzhang Can you let me know if setting reference_data_path_climo_diurnal = /lcrc/group/e3sm/ac.sfeng1/E3SM_output/v2.LR.amip_0101/post/atm/180x360_aave/clim_atm_monthly_diurnal_8xdaily_180x360_aave for [[atm_monthly_180x360_aave_mvm]] works? The intended use of that parameter is for this situation -- i.e., when zppy can't automatically determine what the path should have been.

I'm doing some testing on a simpler cfg in #608 (I'm trying to create a series of small cfgs to test different zppy options, to replace the one large test that misses many parameter combinations)

[chengzhuzhang]

@forsyth2 I got this error: cp: cannot stat '/lcrc/group/e3sm/ac.sfeng1/E3SM_output/v2.LR.amip_0101/post/atm/180x360_aave/clim_atm_monthly_diurnal_8xdaily_180x360_aave/30yr/v2.LR.amip_0101._*_1985??_2014??_climo.nc': No such file or directory . reference_data_path_climo_diurnal worked to point to the correct path, but zppy expects eam.h3 and there is not a way to specify a different string eam.h4 for the reference case...

[forsyth2]

@chengzhuzhang Thanks for checking! It looks like we'll need to create a new parameter like input_files_ref then...

[BunnyVon]

Thank you @chengzhuzhang for following this up.

@forsyth2 , I agree that is probably the best solution. I tried to use ref_name = ' v2.LR.amip_0101.eam.h4' to get around it, but got another error cp: cannot stat '/lcrc/group/e3sm/ac.sfeng1/E3SM_output/v2.LR.amip_0101/post/atm/180x360_aave/clim_diurnal_8xdaily/30yr/v2.LR.amip_0101.eam.h4.eam.h3_*_1985??_2014??_climo.nc': No such file or directory

I think this parameter is quite necessary, particularly for the comparison across different versions and groups that don't have the same output arrangement.

[forsyth2]

particularly for the comparison across different versions and groups

Yes, I think the problem is that, at the time of implementation, I thought the h number would be consistent. I've since discovered that is not the case. I should probably do a full review of the parameters being passed to diags to see if anything else needs _ref version too...

[forsyth2]

This will be resolved by #612.