[Bug]: Error from parameter checking: raise ParameterNotProvidedError(relevant_parameter)

[chengzhuzhang]

What happened?

When testing v3.0.0 rc2, I ran into error below. Not sure how this problem is introduced, but the checker is expecting ref_end_year for tropical_subseasonal set but what always provided as a parameter is ref_final_year.

Traceback (most recent call last):
  File "/usr/WS1/e3sm/apps/e3sm-unified/base/envs/e3sm_unified_1.11.0rc3_login/bin/zppy", line 8, in <module>
    sys.exit(main())
  File "/usr/WS1/e3sm/apps/e3sm-unified/base/envs/e3sm_unified_1.11.0rc3_login/lib/python3.10/site-packages/zppy/__main__.py", line 61, in main
    _launch_scripts(config, script_dir, job_ids_file, plugins)
  File "/usr/WS1/e3sm/apps/e3sm-unified/base/envs/e3sm_unified_1.11.0rc3_login/lib/python3.10/site-packages/zppy/__main__.py", line 237, in _launch_scripts
    existing_bundles = e3sm_diags(config, script_dir, existing_bundles, job_ids_file)
  File "/usr/WS1/e3sm/apps/e3sm-unified/base/envs/e3sm_unified_1.11.0rc3_login/lib/python3.10/site-packages/zppy/e3sm_diags.py", line 39, in e3sm_diags
    check_parameters_for_bash(c)
  File "/usr/WS1/e3sm/apps/e3sm-unified/base/envs/e3sm_unified_1.11.0rc3_login/lib/python3.10/site-packages/zppy/e3sm_diags.py", line 106, in check_parameters_for_bash
    check_required_parameters(c, set(["tropical_subseasonal"]), "ref_end_yr")
  File "/usr/WS1/e3sm/apps/e3sm-unified/base/envs/e3sm_unified_1.11.0rc3_login/lib/python3.10/site-packages/zppy/utils.py", line 241, in check_required_parameters
    raise ParameterNotProvidedError(relevant_parameter)

What machine were you running on?

Ruby at LC.

Environment

zppy v3.0.0 rc2 within e3sm-unified v1.11.0rc3

What command did you run?

zppy -c

Copy your cfg file

[default]
#input = /p/lustre1/golaz1/E3SM/E3SMv3/20240828.v3.LR.piControl.ruby
output = /p/lustre1/zhang40/E3SMv3/post-processing/extendedOutput.v3.LR.historical_0101_rc3/
input = /usr/workspace/e3sm/zhang40/simulations/extendedOutput.v3.LR.historical_0101
#output = /usr/workspace/e3sm/zhang40/postprocessing/extendedOutput.v3.LR.historical_0101_try2
case = extendedOutput.v3.LR.historical_0101
www = /usr/global/web-pages/lc/www/e3sm/diagnostics_output/zhang40/E3SMv3/0130_rc3
mapping_file = map_ne30pg2_to_cmip6_180x360_traave.20231201.nc
machine = "ruby"
account = "e3smtest"
#partition = compute
partition = pbatch
environment_commands = "source /usr/workspace/e3sm/apps/e3sm-unified/load_latest_e3sm_unified_1.11.0rc3_ruby.sh"
#environment_commands = "source /g/g92/zhang40/miniforge3/etc/profile.d/conda.sh; conda activate e3sm-unified110"
#environment_commands = "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.10.0rc5_chrysalis.sh"
campaign = "water_cycle"

[climo]
active = True
#years = "0001:0100:50", "0001:0100:100"
#years = "1985:2014:30", "1985:2014:15"
years = "2000:2014:15",
#years = "0001:0010:10",
walltime = "1:00:00"
#`bck` is needed because running NCO with `mpi` has error
parallel = "bck"

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  mapping_file = map_ne30pg2_to_cmip6_180x360_traave.20231201.nc
  frequency = "monthly"

  [[ atm_monthly_diurnal_8xdaily_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h3"
  mapping_file = map_ne30pg2_to_cmip6_180x360_traave.20231201.nc
  vars = "PRECT"
  frequency = "diurnal_8xdaily"

[ts]
active = True
years = "2000:2014:15"
walltime = "00:50:00"

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = /g/g92/zhang40/zppy_test/map_ne30pg2_to_cmip6_180x360_traave.20231201.nc
  vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,ICEFRAC,LANDFRAC,OCNFRAC,PS,CLDICE,CLDLIQ,T,AODDUST"
# Needed for mixed-phase partition
#  vars = "LANDFRAC,CLDICE,CLDLIQ,T"
  ts_fmt = "cmip"

  [[ atm_daily_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h1"
  frequency = "daily"
  mapping_file = /g/g92/zhang40/zppy_test/map_ne30pg2_to_cmip6_180x360_traave.20231201.nc
  # Needed for Wheeler Kiladis
  vars = "FLUT,PRECT,U850"

  [[ atm_monthly_glb ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = "glb"
[e3sm_diags]
active = True
walltime = "4:00:00"
#years = "0001:0100:50", "0001:0100:100"
years = "2000:2014:15",
ts_num_years = 15
ref_start_yr = 2000
ref_final_yr = 2014
multiprocessing = True
num_workers = 8

  [[ atm_monthly_180x360_aave ]]
#  environment_commands = "source /home/ac.zhang40/y/etc/profile.d/conda.sh; conda activate edv2110"
  short_name = 'v3.LR.historical_0101'
  grid = '180x360_traave'
  reference_data_path = '/usr/workspace/e3sm/data/diagnostics/observations/Atm/climatology'
  obs_ts = '/usr/workspace/e3sm/data/diagnostics/observations/Atm/time-series'
  dc_obs_climo = '/usr/workspace/e3sm/data/diagnostics/observations/Atm/climatology'
  climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
  climo_diurnal_frequency = "diurnal_8xdaily"
  ts_daily_subsection = "atm_daily_180x360_aave"
  sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","diurnal_cycle","zonal_mean_2d_stratosphere","aerosol_aeronet","tropical_subseasonal",

What jobs are failing?

What stack trace are you encountering?

[forsyth2]

@chengzhuzhang Please refer to https://github.com/E3SM-Project/zppy/blob/main/zppy/defaults/default.ini. These are actually slightly different parameters:

# End year (i.e., the last year to use) for the reference data
# Required for "tropical_subseasonal" runs
ref_end_yr = string(default="")
# Final year (i.e., the last available year) for the reference data
# Required for "qbo" runs
# Required for run_type="model_vs_model" "enso_diags"/"streamflow"/"tc_analysis"/"tropical_subseasonal" runs
ref_final_yr = string(default="")

[chengzhuzhang]

Yes, I checked this file. This seems to be a newly introduced change, or it is not tested before? It is confusing to have two parameters defined for one set, notice that ref_end_yr and ref_final_yr are both required here.

[forsyth2]

It is confusing to have two parameters defined for one set, notice that ref_end_yr and ref_final_yr are both required here.

@chengzhuzhang Yes, I believe in #628 I was just trying to pass through parameters to e3sm_diags as transparently as possible:

Actually looking at https://github.com/E3SM-Project/zppy/blob/main/zppy/templates/e3sm_diags.bash, and check_required_parameters(c, set(["tropical_subseasonal"]), "ref_end_yr") in https://github.com/E3SM-Project/zppy/blob/main/zppy/e3sm_diags.py, it seems like "tropical_subseasonal" doesn't actually use ref_final_yr.

I see you started fixing this in #668, but is there a corresponding change in the e3sm_diags parameter space? This is going to trip up users if everything else is using ref_final_yr, but tropical_subseasonal is using ref_end_yr. Are they in fact interchangeable?

[chengzhuzhang]

This is going to trip up users if everything else is using ref_final_yr, but tropical_subseasonal is using ref_end_yr.

I understand that tropical_subseasonal uses ref_final_yr for model vs model as other sets based on monthly time series. And in zppy's default, for model_vs_obs, tropoical_subseasonal set has defaults to 2001 as start and 2010 as end year for reference data.

[forsyth2]

Oh, I understand now. ref_final_yr isn't a parameter in e3sm_diags at all. e3sm_diags only uses ref_end_yr, and for some reason we've been using ref_final_yr to refer to that in zppy. So when I saw ref_end_yr for tropical_subseasonal I thought it was actually a new parameter on the e3sm_diags side. Ok, that needs to be fixed. #668 seems sufficient for that then. Thanks, sorry for the confusion.

[forsyth2]

@chengzhuzhang See my review at #668 (review)