Jobs are getting skipped because of incorrect dependencies

[forsyth2]

Question criteria

• I added a descriptive title here.
• I searched the zppy GitHub Discussions to find a similar question and didn't find it.
• I searched the zppy documentation.

What is the deadline?

N/A

Describe your question

@mahf708 ran into a few issues using "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh" (zppy v2.5.0):

1. zppy still gets confused if some parameters have , after them — is this still a thing? It's pretty weird it is still happening
2. I think something "misleading" is happening when it comes to e3sm_diags mvm, it looks like zppy expects a different set of climo files to be available in order to process the call? That's pretty wild! See message in-thread

After switching to the latest zppy (v3.0.0rc5), there were still issues with incorrect dependencies:

  e3sm_diags_atm_monthly_180x360_aave_model_vs_model_20241204.ne120pg2_r025_RRSwISC6to18E3r5.F20TR-ORO_GWD.muller-cpu_1997-2006_vs_1997-2006
  ...skipping because of dependency status file missing
     /global/cfs/cdirs/e3sm/forsyth/debugging_20250319/20250313.v3HR.F20TR-ORO-GWD.wSLtraj.tuning.ni14_p3rain20.pm-cpu/post/scripts/climo_atm_monthly_180x360_aave_model_vs_model_1997-2006.status

Are there are any possible answers you came across?

Subsections need to be specified more clearly

What machine were you running on?

Perlmutter

Environment

source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.11.0rc13_pm-cpu.sh (zppy v3.0.0rc5)

What command did you run?

zppy -c debugging_20250319.cfg

Copy your cfg file

[default]
input = /global/cfs/cdirs/e3sm/mahf708/E3SMv3_dev/20250313.v3HR.F20TR-ORO-GWD.wSLtraj.tuning.ni14_p3rain20.pm-cpu
#output = /global/cfs/cdirs/e3sm/mahf708/E3SMv3_dev/20250313.v3HR.F20TR-ORO-GWD.wSLtraj.tuning.ni14_p3rain20.pm-cpu
output = /global/cfs/cdirs/e3sm/forsyth/debugging_20250319/20250313.v3HR.F20TR-ORO-GWD.wSLtraj.tuning.ni14_p3rain20.pm-cpu
case = 20250313.v3HR.F20TR-ORO-GWD.wSLtraj.tuning.ni14_p3rain20.pm-cpu
#www = "/global/cfs/cdirs/e3sm/www/mahf708/E3SMv3_dev"
www = "/global/cfs/cdirs/e3sm/www/forsyth/debugging_20250319"
partition = slurm 
environment_commands = "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.11.0rc13_pm-cpu.sh"
campaign = "water_cycle"
input_subdir = "archive/atm/hist"
qos = "regular"

[climo]
active = True
partition = debug
years = "1997:2006:10",
walltime = "00:30:00"

  [[ atm_monthly_180x360_aave ]]
  mapping_file = /global/cfs/cdirs/e3sm/zender/maps/map_ne120pg2_to_cmip6_180x360_nco.20200901.nc
  frequency = "monthly"

  [[ atm_monthly_diurnal_8xdaily_180x360_aave ]]
  input_files = "eam.h3"
  mapping_file = /global/cfs/cdirs/e3sm/zender/maps/map_ne120pg2_to_cmip6_180x360_nco.20200901.nc
  vars = "PRECT"
  frequency = "diurnal_8xdaily"

[ts]
active = True
partition = debug
years = "1997:2006:5",
walltime = "00:30:00"

  [[ atm_monthly_180x360_aave ]]
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = /global/cfs/cdirs/e3sm/zender/maps/map_ne120pg2_to_cmip6_180x360_nco.20200901.nc
  # Use default vars
  # vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,ICEFRAC,LANDFRAC,OCNFRAC,Mass_bc,Mass_dst,Mass_mom,Mass_ncl,Mass_pom,Mass_so4,Mass_soa,AODALL,AODBC,AODDUST,AODPOM,AODSO4,AODSOA,AODSS,AODVIS,PS"
  vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,ICEFRAC,LANDFRAC,OCNFRAC,PS"

  [[ atm_daily_180x360_aave ]]
  input_files = "eam.h1"
  frequency = "daily"
  mapping_file = /global/cfs/cdirs/e3sm/zender/maps/map_ne120pg2_to_cmip6_180x360_nco.20200901.nc
  # Needed for Wheeler Kiladis
  vars = "FLUT,PRECT,U850"

  [[ atm_monthly_glb ]]
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = "glb"
  # Use default vars

  [[ land_monthly ]]
  input_subdir = "archive/lnd/hist"
  input_files = "elm.h0"
  mapping_file = /global/cfs/cdirs/e3sm/zender/maps/map_r025_to_cmip6_180x360_traave.20241101.nc
  frequency = "monthly"
  vars = "FSH,RH2M"
  extra_vars = "landfrac"

  [[ rof_monthly ]]
  input_subdir = "archive/rof/hist"
  input_files = "mosart.h0"
  mapping_file = ""
  frequency = "monthly"
  vars = "RIVER_DISCHARGE_OVER_LAND_LIQ"
  extra_vars = 'areatotal2'


[e3sm_diags]
active = True
walltime = "02:30:00"
years = "1997:2006:10",
ts_num_years = 5
ref_start_yr = 1997
ref_final_yr = 2006
multiprocessing = True
num_workers = 128

  [[ atm_monthly_180x360_aave ]]
  short_name = '20250313.v3HR.F20TR-ORO-GWD.wSLtraj.tuning.ni14_p3rain20.pm-cpu'
  grid = '180x360_nco'
  reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology'
  obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series'
  dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/TRMM-3B43v-7_3h'
  climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
  climo_diurnal_frequency = "diurnal_8xdaily"
  sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","diurnal_cycle","zonal_mean_2d_stratosphere","aerosol_aeronet","aerosol_budget","enso_diags","tropical_subseasonal","streamflow"
  output_format_subplot = "pdf",

  [[ atm_monthly_180x360_aave_model_vs_model ]]
  # e3sm_diags_atm_monthly_180x360_aave_model_vs_model_20241204.ne120pg2_r025_RRSwISC6to18E3r5.F20TR-ORO_GWD.muller-cpu_1997-2006_vs_1997-2006
  # ...skipping because of dependency status file missing
  #    /global/cfs/cdirs/e3sm/forsyth/debugging_20250319/20250313.v3HR.F20TR-ORO-GWD.wSLtraj.tuning.ni14_p3rain20.pm-cpu/post/scripts/climo_atm_monthly_180x360_aave_model_vs_model_1997-2006.status
  years = "1997-2006",
  short_name = '20250313.v3HR.F20TR-ORO-GWD.wSLtraj.tuning.ni14_p3rain20.pm-cpu'
  grid = '180x360_aave',
  run_type = "model_vs_model"
  ref_name = '20241204.ne120pg2_r025_RRSwISC6to18E3r5.F20TR-ORO_GWD.muller-cpu'
  short_ref_name = '20241204.ne120pg2_r025_RRSwISC6to18E3r5.F20TR-ORO_GWD.muller-cpu'
  tag = '20241204.ne120pg2_r025_RRSwISC6to18E3r5.F20TR-ORO_GWD.muller-cpu'
  reference_data_path = '/pscratch/sd/x/xzheng/E3SM_simulations/E3SMv3_tests/20241204.ne120pg2_r025_RRSwISC6to18E3r5.F20TR-ORO_GWD.muller-cpu/post/atm/180x360_nco/clim'
  # # mvm diurnal_cycle only
  # reference_data_path_climo_diurnal = "/pscratch/sd/x/xzheng/E3SM_simulations/E3SMv3_tests/20241204.ne120pg2_r025_RRSwISC6to18E3r5.F20TR-ORO_GWD.muller-cpu/post/atm/180x360_nco/clim_diurnal_8xdaily"
  # # mvm "enso_diags", "qbo", "area_mean_time_series"
  # reference_data_path_ts = "/pscratch/sd/x/xzheng/E3SM_simulations/E3SMv3_tests/20241204.ne120pg2_r025_RRSwISC6to18E3r5.F20TR-ORO_GWD.muller-cpu/post/atm/180x360_nco/ts/monthly"
  # # mvm tropical_subseasonal only
  # reference_data_path_ts_daily = "/pscratch/sd/x/xzheng/E3SM_simulations/E3SMv3_tests/20241204.ne120pg2_r025_RRSwISC6to18E3r5.F20TR-ORO_GWD.muller-cpu/post/atm/180x360_nco/ts/daily"
  climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
  climo_diurnal_frequency = "diurnal_8xdaily"
  ts_subsection = "atm_monthly_180x360_aave"
  sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","diurnal_cycle","zonal_mean_2d_stratosphere","aerosol_aeronet","aerosol_budget","enso_diags","tropical_subseasonal","streamflow"
  output_format_subplot = "pdf",
  diff_title = 'Difference'


[amwg]
active = False

[mpas_analysis]
active = False
walltime = "6:00:00"
parallelTaskCount = 6
anomalyRefYear = 1
ts_years = "1-40",
enso_years = "1-40",
climo_years = "1-40",
mesh = "EC30to60E2r2"
shortTermArchive = True
input_subdir = "archive/ocn/hist"
#input_subdir not recognized. *_in, stream, and nc files are expected to be under .../Allruns/run
#generate below for cryo config
#generate = 'all', 'no_BGC', 'no_icebergs'

[global_time_series]
# global_time_series_1997-2006
# ...skipping because of dependency status file missing
#    /global/cfs/cdirs/e3sm/forsyth/debugging_20250319/20250313.v3HR.F20TR-ORO-GWD.wSLtraj.tuning.ni14_p3rain20.pm-cpu/post/scripts/mpas_analysis_ts_1997-2006_climo_1997-2006.status
active = True
#If .st_archive not run, specify "run" below. Default using archived path?
# input_subdir = "archive/ocn/hist"
years = "1997-2006",
ts_num_years = 5
figstr= "20250313.v3HR.F20TR-ORO-GWD.wSLtraj.tuning.ni14_p3rain20.pm-cpu"
# moc_file = ""
experiment_name = "20250313.v3HR.F20TR-ORO-GWD.wSLtraj.tuning.ni14_p3rain20.pm-cpu"
ts_years = "1997-2006",
climo_years = "1997-2006",

What jobs are failing?

e3sm_diags_atm_monthly_180x360_aave_model_vs_model_20241204.ne120pg2_r025_RRSwISC6to18E3r5.F20TR-ORO_GWD.muller-cpu_1997-2006_vs_1997-2006

What stack trace are you encountering?

N/A

[forsyth2]

@mahf708 ts_subsection isn't sufficient to specify. Add ts_daily_subsection = "atm_daily_180x360_aave". Because you have tropical_subseasonal in your diags sets, zppy needs to know what the ts subtask name is. Since you haven't specified it, zppy is making an inference that is the same name as the model-vs-model subtask, which turns out to be an incorrect inference.

Similarly, you need climo_subsection = "atm_monthly_180x360_aave" as well.

I compared your cfg above to my working setup in https://github.com/E3SM-Project/zppy/blob/main/tests/integration/template_weekly_comprehensive_v3.cfg. I found that it had 4 extra parameters, 2 of which apparently don't make a difference here (probably because your ref years match up with test years):

  climo_subsection = "atm_monthly_180x360_aave" 
  # ref_years = "1997-2006", # Works with this removed
  ts_daily_subsection = "atm_daily_180x360_aave" 
  # ts_num_years_ref = 5 # Works with this removed

Two other things:

1. I haven't looked into the comma issue yet. I thought we had resolved that in 2022 with Years fix #338, but looking at that more, it appears the validator stopped throwing an error on that?
2. default.ini has specified the following since 2022 (Model vs Model functionality #195), yet I notice you have tag = '20241204.ne120pg2_r025_RRSwISC6to18E3r5.F20TR-ORO_GWD.muller-cpu'. I thought maybe that was the issue, but I guess it may be more just a description string.

# Options are "model_vs_obs" and "model_vs_model"
tag = string(default="model_vs_obs")

[mahf708]

Do you want to look at the parser code together? Or we can ask a bot to look into it?

[forsyth2]

Does zppy offer a validate option?

Yes, set dry_run = True in the cfg.

I think it would help to print specifically the false (i.e., failing) ones.

There is in fact an issue including this already: #281. It's from 2022, so I suppose it's been low priority.

From 2 above is where I knew it was the comma after all. In order for me to figure out the details, I would need to redo this from scratch and then make notes.

Hmm ok it may take some trial and error then to reproduce.

[mahf708]

Thanks for reminding me of the run_dry option! That will help with better debugging/reproducing the parser partss

[mahf708]

Btw, i manage to make it work fine eventually (in part by hacking the bash scripts because of the weird aave/nco thing avbove)

[forsyth2]

Glad you've gotten it working. Thanks for pointing out issues and things that could improve the user experience. It's good to know what the pain points, or failure points, are. zppy testing is quite involved and even then it's only a tiny subset of the possible things that could be tested.