New user; Why is zppy looking for .sgs.h0 file

[evasinha]

I am a new user to zppy and am using the following configuration file on compy to create lat_lon and timeseries plot of land only simulations.

[default]
input = /compyfs/sinh210/e3sm_scratch/20220726_global_f09_g16_I1850GSWELMCNCROP_trans_HIST
input_subdir = run
output = /compyfs/sinh210/e3sm_scratch/20220726_global_f09_g16_I1850GSWELMCNCROP_trans_HIST
case = 20220726_global_f09_g16_I1850GSWELMCNCROP_trans_HIST
www = /compyfs/www/sinh210/E3SM/v2/global/trans
partition = short
model_name = elm

[climo]
active = True
input_files = 'elm.h0'
mapping_file = /qfs/people/zender/data/maps/map_r05_to_cmip6_180x360_aave.20200901.nc
vars = "TLAI,EFLX_LH_TOT,GPP,NPP,NBP,QSOIL,QVEGT,QVEGE,QRUNOFF,TOTVEGC,TOTSOMC,TOTSOMN,TOTSOMP,FPG,FPG_P,TBOT,FSDS,RAIN,CPOOL,SR,RH2M"
years = "1995:2014:20",

[glb]
active = True
weights = 'area'
input_files = 'elm.h0'
mapping_file = /qfs/people/zender/data/maps/map_r05_to_cmip6_180x360_aave.20200901.nc
vars = "TLAI,EFLX_LH_TOT,GPP,NPP,NBP,QSOIL,QVEGT,QVEGE,QRUNOFF,TOTVEGC,TOTSOMC,TOTSOMN,TOTSOMP,FPG,FPG_P,TBOT,FSDS,RAIN,CPOOL,SR,RH2M"
years = "1995:2014:20",

[ts]
active = True
mapping_file = /qfs/people/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
vars = "TLAI,EFLX_LH_TOT,GPP,NPP,NBP,QSOIL,QVEGT,QVEGE,QRUNOFF,TOTVEGC,TOTSOMC,TOTSOMN,TOTSOMP,FPG,FPG_P,TBOT,FSDS,RAIN,CPOOL,SR,RH2M"
years = "1850:2014:12",

[e3sm_diags]
active = False

[e3sm_diags_vs_model]
active = False

[amwg]
active = False

[mpas_analysis]
active = False

I am getting the following error messages:

1. ncclimo: ERROR Unable to find required input file /compyfs/sinh210/e3sm_scratch/20220726_global_f09_g16_I1850GSWELMCNCROP_trans_HIST/run/20220726_global_f09_g16_I1850GSWELMCNCROP_trans_HIST.sgs.h0.1995-01.nc. Why is zppy looking for sgs file instead of elm?

2. ts_only ncclimo: ERROR The 12 files specified (via stdin pipe, positional, or input directory) are expected to contain 1 years of data whereas the date options specify 12 years of data. How can I modify my cfg to address this mistake?

[forsyth2]

I am able to reproduce the errors with a script modified to my directory. (Note that error 1 appears in the climo*.o* file and that error 2 appears in the ts.*o.* files). I will try to debug.

[default]
input = /compyfs/sinh210/e3sm_scratch/20220726_global_f09_g16_I1850GSWELMCNCROP_trans_HIST
input_subdir = run
output = /compyfs/fors729/n305.20220726_global_f09_g16_I1850GSWELMCNCROP_trans_HIST
case = 20220726_global_f09_g16_I1850GSWELMCNCROP_trans_HIST
www = /compyfs/www/fors729/zppy_support
partition = short
model_name = elm

[climo]
active = True
input_files = 'elm.h0'
mapping_file = /qfs/people/zender/data/maps/map_r05_to_cmip6_180x360_aave.20200901.nc
vars = "TLAI,EFLX_LH_TOT,GPP,NPP,NBP,QSOIL,QVEGT,QVEGE,QRUNOFF,TOTVEGC,TOTSOMC,TOTSOMN,TOTSOMP,FPG,FPG_P,TBOT,FSDS,RAIN,CPOOL,SR,RH2M"
years = "1995:2014:20",

[glb]
active = True
weights = 'area'
input_files = 'elm.h0'
mapping_file = /qfs/people/zender/data/maps/map_r05_to_cmip6_180x360_aave.20200901.nc
vars = "TLAI,EFLX_LH_TOT,GPP,NPP,NBP,QSOIL,QVEGT,QVEGE,QRUNOFF,TOTVEGC,TOTSOMC,TOTSOMN,TOTSOMP,FPG,FPG_P,TBOT,FSDS,RAIN,CPOOL,SR,RH2M"
years = "1995:2014:20",

[ts]
active = True
mapping_file = /qfs/people/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
vars = "TLAI,EFLX_LH_TOT,GPP,NPP,NBP,QSOIL,QVEGT,QVEGE,QRUNOFF,TOTVEGC,TOTSOMC,TOTSOMN,TOTSOMP,FPG,FPG_P,TBOT,FSDS,RAIN,CPOOL,SR,RH2M"
years = "1850:2014:12",

[e3sm_diags]
active = False

[e3sm_diags_vs_model]
active = False

[amwg]
active = False

[mpas_analysis]
active = False

[czender]

I'm currently looking into this, and may be interrupted by traveling through 3 airports before I finish...
The "good news" is that I too can reproduce the problem(s). The workaround for one issue with the climos is to add the --mdl_nm=clm option to the run command that @forsyth2 shows above. This causes ncclimo to look for elm.h0 not sgs.h0 files. A different workaround is to use -P elm instead of -P sgs but that will only work with E3SMU 1.7 not 1.6 (which for now is the default on Compy). Unfortunately, neither of those workarounds is sufficient because, as ncclimo prints to its standard output, Each input file assumed to contain statistics for one month. @evasinha your files contain 12 months not 1 month. Allowing ncclimo to handle that is on the list but will take time. @forsyth2 good job deducing this and yes, that is why those error messages occur for the climo command.

The ts command above fails for similar reasons. First there is a typo (change eam to elm) in that command. Then it will break because there are 12 months, not 1 month, per input file. @forsyth2 it is imprecise to say the zppy concatenates the files together and sends those to ncclimo. It is more accurate to say the zppy concatenates the filenames together into a single files, and sends that to ncclimo.

One way to proceed is to split the input files from yearly into monthly files, then re-run the commands with the supplied workarounds. And @evasinha you may want to output one file per month from ELM instead of one per year until ncclimo supports that feature.

[evasinha]

@czender Thanks for examining the error in more detail. Yes, know that I understand what is causing the issue with zppy better, I will output one file per month for all future runs.

[forsyth2]

Thank you @czender!

[czender]

FYI I have fixed (in the latest snapshot) the problem where specifying -P sgs results in an unwanted sgs in the expected input filenames. However, the inability to handle 12 months per file is still there.

[evasinha]

Thank you @czender. I should be able to bypass the 12 months per file issue with the new runs I am generating that outputs one file per month. Has your recent fix regarding -P sgs merged into the main branch? How do I access the fix?

[czender]

It's in the main branch of NCO. There are three ways to access it: 1. Clone and build NCO from source and call ncclimo yourself. That is doable though not trivial. 2. Copy just the ncclimo executable from ~zender/bin/ncclimo to someplace that has precedence over the E3SMU ncclimo. 3. Use the NCO in my personal directories on Compy. You would have to first do something like

source /share/apps/intel/2020/compilers_and_libraries_2020.0.166/linux/bin/compilervars.sh intel64 # Compy
module purge
module load intel/20.0.0
module load intelmpi/2020 # Required for ncclimo/ncremap in MPI-mode
module load netcdf/4.6.3
export PATH="~zender/bin:${PATH}"
export LD_LIBRARY_PATH="~zender/lib:${LD_LIBRARY_PATH}"

[czender]

Thanks, I learned something. Didn't realize GitHub even had Discussion pages. Yes, by all means keep it don't delete it.

[forsyth2]

@czender @evasinha @golaz I want to make sure I'm getting the right take-aways from this thread:

Issue 1: climo -- Why is zppy looking for sgs file instead of elm?

sgs is default -- e.g., https://github.com/E3SM-Project/zppy/blob/main/zppy/templates/climo.bash#L51. Note that this works fine for land ts -- https://github.com/E3SM-Project/zppy/blob/main/zppy/templates/ts.bash#L94; we have a land_monthly ts subtask in our testing: https://github.com/E3SM-Project/zppy/blob/main/tests/integration/template_complete_run.cfg#L51.

While the intended way to invoke ncclimo for ELM is ncclimo -P elm, for a while we had to invoke it as ncclimo -P sgs to workaround an earlier ncclimo bug that was, IIRC, in E3SMU 1.5.

The workaround for one issue with the climos is to add the --mdl_nm=clm option to the run command

A different workaround is to use -P elm instead of -P sgs but that will only work with E3SMU 1.7 not 1.6 (which for now is the default on Compy)

Does this mean the sgs references in climo.bash and ts.bash should be replaced by elm? Or at the least, there should be a check input_files.split(".")[0] == 'elm' in addition to the existing input_files.split(".")[0] == 'eam'? And in the elm case set --mdl_nm=clm or -P elm instead of --prc_typ=sgs? If so, I can make a GitHub issue for that.

your files contain 12 months not 1 month. Allowing ncclimo to handle that is on the list but will take time.

If ncclimo doesn't have that functionality, is there anything zppy can do as a workaround?

Issue 2: ts -- expected to contain 1 years of data whereas the date options specify 12 years of data

First there is a typo (change eam to elm) in that command

I don't think that is a typo. I think that is because nothing in the [ts] task specifies it is for land, not atmosphere. https://github.com/E3SM-Project/zppy/blob/main/tests/integration/template_complete_run.cfg#L54 specifies input_files = "elm.h0" for instance.

Then it will break because there are 12 months, not 1 month, per input file.

It is more accurate to say the zppy concatenates the filenames together into a single files, and sends that to ncclimo.

So, ncclimo is actually getting a list of 12 (one-per-month) filenames in the usual case? And it failed in this case because it was getting 1 (12 months) filename?

One way to proceed is to split the input files from yearly into monthly files, then re-run the commands with the supplied workarounds.

Is there a command zppy could include to do that automatically for the user? If so, I can make a GitHub issue for that.

[evasinha]

@forsyth2 Yes, all identified GitHub issues look correct to me.

[czender]

Does this mean the sgs references in climo.bash and ts.bash should be replaced by elm?

When zppy uses E3SMU 1.7+ then it should invoke ncclimo with -P model_name. That would simplify both climo.bash and ts.bash. Note that model_name is then used to construct the filenames. So if the model_name is elm and the filenames are v1-style and thus use clm2, then ncclimo needs an additional argument (--mdl_nm=clm2) to construct the filenames correctly.

When invoked with -P model_name, ncclimo itself will internally generate the correct SGS settings for the models (ELM, MPAS-Seaice) that require SGS. This is probably more robust than zppy trying to generate the SGS settings.

Or at the least, there should be a check input_files.split(".")[0] == 'elm' in addition to the existing input_files.split(".")[0] == 'eam'? And in the elm case set --mdl_nm=clm or -P elm instead of --prc_typ=sgs?

Something like this (with no else clause) should work in all cases:

{%- if input_files.split(".")[0] == 'cam' or  input_files.split(".")[0] == 'eam' or input_files.split(".")[0] == 'elm' or input_files.split(".")[0] == 'clm2'  %}
--prc_typ={{ input_files.split(".")[0] }}

Of course I recommend testing that to be sure answers are BFB as expected.

If so, I can make a GitHub issue for that.

Sounds good.

If ncclimo doesn't have that functionality, is there anything zppy can do as a workaround?

If the input files contain multiple months rather than single months then zppy could split them up into monthly files as described above. However, if the first month is not January and the last month is not December then ncclimo would need to be invoked with the correct --mth_srt and --mth_end options as well.

So, ncclimo is actually getting a list of 12 (one-per-month) filenames in the usual case? And it failed in this case because it was getting 1 (12 months) filename?

Yes and yes

Is there a command zppy could include to do that automatically for the user? If so, I can make a GitHub issue for that.

See answer above. One command that will extract the month of March, say, from a file containing twelve months (i.e., timesteps) that begins in January, and place it in a separate file is, for example:
ncks -d time,2 in.nc out.nc

[forsyth2]

Thanks @czender. I created #313 and #314 to follow up further on these issues.