Skip to content

add example cfg for v3.LR.historical run on LCRC #694

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Apr 3, 2025
Merged

add example cfg for v3.LR.historical run on LCRC #694

Changes from 4 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
f86a6cb
add example cfg for v3.LR.historical run on LCRC
chengzhuzhang Mar 20, 2025
401b32f
Add explanatory notes to cfg
forsyth2 Mar 21, 2025
2c12941
Update paths
forsyth2 Mar 24, 2025
0917f59
Parameter updates
forsyth2 Apr 1, 2025
b691c5f
Address comments
forsyth2 Apr 2, 2025
2eb9c6c
update mvm
chengzhuzhang Apr 2, 2025
6337ea6
Cfg used for 2025-04-03 run
forsyth2 Apr 3, 2025
a618c64
Cfg used for 2025-04-03 run try 2
forsyth2 Apr 3, 2025
67ede24
ref_name update
forsyth2 Apr 3, 2025
bcad046
Clean up env_cmds and paths
forsyth2 Apr 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
289 changes: 289 additions & 0 deletions examples/post.v3.LR.historical.zppy_v3.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,289 @@
# This is an example zppy v3.0.0 configuration file for Chrysalis.
# v3.0.0 has several non-backward compatible ("breaking") changes.
# These are described here.
# In particular, there are a total of 5 deprecated parameters.
# (Search for "DEPRECATED" in this file to find them.)
# zppy v3.0.0 is included in E3SM Unified 1.11.0.

[default]
# Change these two paths to include your username instead
# You can also change these to match your preferred directory structure
output = /lcrc/group/e3sm/ac.forsyth2/E3SMv3_20250331_try2/v3.LR.historical_0051
www = /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/E3SMv3_20250331_try2
# Once E3SM Unified 1.11.0 is released, you can use this line instead:
# environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
environment_commands = "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.11.0rc13_chrysalis.sh"
Copy link
Collaborator

@forsyth2 forsyth2 Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: before merging, just make this the environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh" line.

# Other parameters:
campaign = "water_cycle"
case = v3.LR.historical_0051
input = /lcrc/group/e3sm2/ac.wlin/E3SMv3/v3.LR.historical_0051
Copy link
Collaborator

@golaz golaz Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would move input near the top before output

Copy link
Collaborator

@forsyth2 forsyth2 Apr 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to keep output and www together at the very top because they are the only 2 lines a user would actually need to change in the cfg.

partition = compute

[climo]
active = True
walltime = "1:00:00"
years = "1985:2014:30",
# Another example of `years`:
# years = "1985:2014:30", "1985:2014:15"

[[ atm_monthly_180x360_aave ]]
# This subtask is a dependency for the e3sm_diags task's atm_monthly_180x360_aave and atm_monthly_180x360_aave_mvm subtasks.
# The following e3sm_diags sets require it:
# "lat_lon", "zonal_mean_xy", "zonal_mean_2d", "polar", "cosp_histogram", "meridional_mean_2d", "annual_cycle_zonal_mean", "zonal_mean_2d_stratosphere" "aerosol_aeronet", "aerosol_budget"
frequency = "monthly"
input_subdir = "archive/atm/hist"
mapping_file = map_ne30pg2_to_cmip6_180x360_aave.20200201.nc

[[ atm_monthly_diurnal_8xdaily_180x360_aave ]]
# This subtask is a dependency for the e3sm_diags task's atm_monthly_180x360 and atm_monthly_180x360_aave_mvm subtasks.
# The following e3sm_diags sets require it:
# "diurnal_cycle"
frequency = "diurnal_8xdaily"
input_files = "eam.h3"
input_subdir = "archive/atm/hist"
mapping_file = map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
vars = "PRECT"

[[ land_monthly_climo ]]
# This subtask is a dependency for the e3sm_diags task's lnd_monthly_mvm_lnd subtask.
# The following e3sm_diags sets require it:
# "lat_lon_land",
frequency = "monthly"
input_files = "elm.h0"
input_subdir = archive/lnd/hist
vars = "" # Setting this as "" will tell zppy to use ALL variables

[ts]
# 2 DEPRECATED PARAMETERS in zppy v3.0.0 (the new e3sm_to_cmip task renders these obsolete):
# e3sm_to_cmip_environment_commands
# ts_fmt (e.g., `ts_fmt = "cmip"`)
active = True
walltime = "00:50:00"
years = "1985:2014:30"
Copy link
Collaborator

@golaz golaz Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change the chunk time to 5 years instead of 30 years (and fix whatever else depends on it).


[[ atm_monthly_180x360_aave ]]
# This subtask is a dependency for the e3sm_diags task's atm_monthly_180x360 and atm_monthly_180x360_aave_mvm subtasks.
# The following e3sm_diags sets require it:
# "enso_diags", "qbo", "area_mean_time_series"
# This subtask is also a dependency for the e3sm_to_cmip task's atm_monthly_180x360_aave subtask.
frequency = "monthly"
input_files = "eam.h0"
input_subdir = "archive/atm/hist"
mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
Copy link
Collaborator

@golaz golaz Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can mache abstract the path to make the cfg file portable?

Copy link
Collaborator

@forsyth2 forsyth2 Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mache operates in Python. There's no way to specify a Mache command in the cfg, short of adding a feature to zppy to eval parameters as code rather than text strings (which has security issues). The alternative is the solution implemented in the tests directory, which is to have template cfgs that then auto-generate cfg's for each machine using Python (including Mache).

Copy link
Collaborator Author

@chengzhuzhang chengzhuzhang Apr 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think mapping_file = map_ne30pg2_to_cmip6_180x360_aave.20200201.nc should work here.

vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,ICEFRAC,LANDFRAC,OCNFRAC,PS,CLDICE,CLDLIQ,T,AODDUST,PSL,U10,QREFHT,LHFLX,FLDS,TGCLDIWP,TGCLDCWP,TMQ,FLNSC,FLUTC,FSDS,FSDSC,SOLIN,FSNSC,FSNTOAC,AODABS,AODVIS,AREL,TREFMNAV,TREFMXAV"
# Needed for mixed-phase partition:
# vars = "LANDFRAC,CLDICE,CLDLIQ,T"

[[ land_monthly ]]
# This subtask is a dependency for the e3sm_to_cmip task's land_monthly subtask.
extra_vars = "landfrac"
Copy link
Collaborator

@golaz golaz Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really would prefer 'extra_vars' to be listed after 'vars'. Same other places.

frequency = "monthly"
input_files = "elm.h0"
input_subdir = "archive/lnd/hist"
mapping_file = map_r05_to_cmip6_180x360_aave.20231110.nc
Copy link
Collaborator

@golaz golaz Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need the full path here? mache takes care of it?

Copy link
Collaborator

@forsyth2 forsyth2 Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, https://github.com/E3SM-Project/zppy/blob/main/zppy/utils.py#L167:

def set_mapping_file(c: Dict[str, Any]) -> None:
    if c["mapping_file"] and (c["mapping_file"] != "glb"):
        directory: str = os.path.dirname(c["mapping_file"])
        if not directory:
            # We use the mapping file from Mache's [diagnostics > base_path].
            # However, new mapping files should be added to Mache's [sync > public_diags].
            # These files will then be synced over.
            c["mapping_file"] = os.path.join(
                c["diagnostics_base_path"], "maps", c["mapping_file"]
            )

vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR"

[[ atm_daily_180x360_aave ]]
# This subtask is a dependency for the e3sm_diags task's atm_monthly_180x360 subtask.
# The following e3sm_diags sets require it:
# "tropical_subseasonal"
frequency = "daily"
input_files = "eam.h1"
input_subdir = "archive/atm/hist"
mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
Copy link
Collaborator

@golaz golaz Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mache inferred path?

Copy link
Collaborator Author

@chengzhuzhang chengzhuzhang Apr 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same: mapping_file = map_ne30pg2_to_cmip6_180x360_aave.20200201.nc should work here.

# Needed for Wheeler Kiladis
vars = "FLUT,PRECT,U850"

[[ rof_monthly ]]
# The following e3sm_diags sets require it:
# "streamflow"
extra_vars = 'areatotal2'
frequency = "monthly"
input_files = "mosart.h0"
input_subdir = "archive/rof/hist"
mapping_file = ""
vars = "RIVER_DISCHARGE_OVER_LAND_LIQ"

[[ atm_monthly_glb ]]
# This subtask is a dependency for the global_time_series task.
frequency = "monthly"
input_files = "eam.h0"
input_subdir = "archive/atm/hist"
mapping_file = "glb"

[[ lnd_monthly_glb ]]
# This subtask is a dependency for the global_time_series task.
frequency = "monthly"
input_files = "elm.h0"
input_subdir = "archive/lnd/hist"
mapping_file = "glb"
# vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILWATER_10CM,TSA,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR"
vars = "" # This will tell zppy to use all available variables
Copy link
Collaborator Author

@chengzhuzhang chengzhuzhang Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider to add a note about elapsed time for this 30 years run for this option, and caution about needing to adjust wall-time limit for longer simulations.


[e3sm_to_cmip]
# This is a new task in zppy v3.0.0!
# This task replaces the `ts_fmt = "cmip"` functionality previously found in the `ts` task.
active = True
frequency = "monthly"
# An alternative to setting `ts_grid` here is setting `ts_atm_grid` and `ts_land_grid` in the respective subtasks below.
ts_grid = "180x360_aave"
ts_num_years = 30
walltime = "00:50:00"
years = "1985:2014:30"

[[ atm_monthly_180x360_aave ]]
# This subtask is a dependency for the ilamb task.
# This subtask depends on the ts task's atm_monthly_180x360_aave subtask.
# Notice this subtask name matches a subtask in the `ts` task.
# If it did not, then the `ts_atm_subsection` parameter would be required here to tell zppy which subtask to use.
input_files = "eam.h0"

[[ land_monthly ]]
# This subtask is a dependency for the ilamb task.
# This subtask depends on the ts task's land_monthly subtask.
# Notice this subtask name matches a subtask in the `ts` task.
# If it did not, then the `ts_land_subsection` parameter would be required here to tell zppy which subtask to use.
input_files = "elm.h0"

[tc_analysis]
# This subtask is a dependency for the e3sm_diags task's atm_monthly_180x360_aave subtask.
# 1 DEPRECATED PARAMETER in zppy v3.0.0:
# scratch
active = True
walltime = "02:00:00"
years = "1985:2014:30",
Copy link
Collaborator Author

@chengzhuzhang chengzhuzhang Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider to add a note about elapsed time for this 30 years run, and caution about needing to adjust wall-time limit for longer simulations.

Copy link
Collaborator

@forsyth2 forsyth2 Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For tc_analysis specifically, or in general at the top of the cfg?

Copy link
Collaborator Author

@chengzhuzhang chengzhuzhang Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For tc_analysis here, can you find the elapsed time from log files, so folks can have a rough estimate.

Copy link
Collaborator

@forsyth2 forsyth2 Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tc_analysis specifically is 3296 seconds = 54.93 minutes

> grep "Elapsed time" *.o*
climo_atm_monthly_180x360_aave_1985-2014.o719881:Elapsed time 0m40s
climo_atm_monthly_180x360_aave_1985-2014.o719881:Elapsed time: 44 seconds
climo_atm_monthly_diurnal_8xdaily_180x360_aave_1985-2014.o719882:Elapsed time 2m2s
climo_atm_monthly_diurnal_8xdaily_180x360_aave_1985-2014.o719882:Elapsed time: 133 seconds
climo_land_monthly_climo_1985-2014.o719883:Elapsed time 1m21s
climo_land_monthly_climo_1985-2014.o719883:Elapsed time: 83 seconds
e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-2014.o719893:Elapsed time: 4473 seconds
e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1985-2014_vs_1985-2014.o719894:Elapsed time: 1460 seconds
e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1985-2014_vs_0051-0100.o719895:Elapsed time: 1855 seconds
e3sm_to_cmip_atm_monthly_180x360_aave_1985-2014-0030.o719890:Elapsed time: 71 seconds
e3sm_to_cmip_land_monthly_1985-2014-0030.o719891:Elapsed time: 74 seconds
global_time_series_1985-2014.o719959:Elapsed time: 24475 seconds
ilamb_1985-2014.o719898:Elapsed time: 1472 seconds
mpas_analysis_ts_1985-2014_climo_1985-2014.o719896:Elapsed time: 1272 seconds
tc_analysis_1985-2014.o719892:Elapsed time: 3296 seconds
ts_atm_daily_180x360_aave_1985-2014-0030.o719886:Elapsed time 1m18s
ts_atm_daily_180x360_aave_1985-2014-0030.o719886:Elapsed time: 84 seconds
ts_atm_monthly_180x360_aave_1985-2014-0030.o719884:Elapsed time 3m1s
ts_atm_monthly_180x360_aave_1985-2014-0030.o719884:Elapsed time: 192 seconds
ts_atm_monthly_glb_1985-2014-0030.o719888:Elapsed time 0m46s
ts_atm_monthly_glb_1985-2014-0030.o719888:Elapsed time: 51 seconds
ts_land_monthly_1985-2014-0030.o719885:Elapsed time 1m43s
ts_land_monthly_1985-2014-0030.o719885:Elapsed time: 114 seconds
ts_lnd_monthly_glb_1985-2014-0030.o719889:Elapsed time 7m24s
ts_lnd_monthly_glb_1985-2014-0030.o719889:Elapsed time: 449 seconds
ts_rof_monthly_1985-2014-0030.o719887:Elapsed time 0m27s
ts_rof_monthly_1985-2014-0030.o719887:Elapsed time: 32 seconds


[e3sm_diags]
active = True
multiprocessing = True
num_workers = 8
ref_final_yr = 2014
ref_start_yr = 1985
ts_num_years = 30
walltime = "4:00:00"
years = "1985:2014:30",

[[ atm_monthly_180x360_aave ]]
Copy link
Collaborator

@golaz golaz Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to have parameters related to obs path grouped together

Copy link
Collaborator

@forsyth2 forsyth2 Apr 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would break the very intentional order of the variables in this cfg explaining which sets which require which extra parameters (this section breaks that down into 6 categories). My interpretation of this example cfg is to teach users what parameters are needed when.

Grouping them in other ways may make sense for users, but I don't think it's necessarily the point of this particular cfg.

Copy link
Collaborator

@forsyth2 forsyth2 Apr 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ @chengzhuzhang

# NOTE: If you want to use the latest development environment of e3sm_diags,
# you can do something like the following:
# environment_commands = "source /home/ac.zhang40/y/etc/profile.d/conda.sh; conda activate e3sm_diags_dev"
# `e3sm_diags` is largely driven by which e3sm_diags sets are requested:
sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","diurnal_cycle","zonal_mean_2d_stratosphere","aerosol_aeronet","tropical_subseasonal","tc_analysis", "tropical_subseasonal",
Copy link
Collaborator

@golaz golaz Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix alignment

Copy link
Collaborator

@forsyth2 forsyth2 Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@golaz I think that is just how it looks in the diff view. If you look at the file in https://github.com/E3SM-Project/zppy/blob/0917f59e9d48a2f7d246ed99b50eae0b352ed9a2/examples/post.v3.LR.historical.zppy_v3.cfg, it is aligned.

# We can categorize e3sm_diags sets into 6 groups, based on dependencies.
# Different e3sm_diags sets require different parameters to be set.
#
# 1. These sets depend on the climo's task's atm_monthly_180x360_aave subtask:
# "lat_lon", "zonal_mean_xy", "zonal_mean_2d", "polar", "cosp_histogram", "meridional_mean_2d", "annual_cycle_zonal_mean", "zonal_mean_2d_stratosphere" "aerosol_aeronet", "aerosol_budget" (not requested)
dc_obs_climo = '/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/climatology'
Copy link
Collaborator

@golaz golaz Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does mache know the path?

Copy link
Collaborator

@forsyth2 forsyth2 Apr 2, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned earlier, we can't use Mache in the cfg itself. That said, zppy does have a very involved parameter inference mechanism. For example, https://github.com/E3SM-Project/zppy/blob/main/zppy/e3sm_diags.py#L192:

        if "diurnal_cycle" in c["sets"]:
            set_value_of_parameter_if_undefined(
                c,
                "dc_obs_climo",
                c["reference_data_path"],
                ParameterInferenceType.PATH_INFERENCE,
            )

That is, if dc_obs_climo is undefined, zppy will attempt to use the reference_data_path. Notice in this case however, that would actually be the incorrect inference. Thus, we have no choice but to set it explicitly.

# 2. These sets depend on the climo's task's atm_monthly_diurnal_8xdaily_180x360_aave subtask:
# "diurnal_cycle"
climo_diurnal_frequency = "diurnal_8xdaily"
# The name of this subtask doesn't match up with the name of the subtask for diurnal data in the `climo` task. We therefore need to explicitly tell zppy which subtask to use.
climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
# 3. These sets depend on the ts task's atm_monthly_180x360_aave subtask:
# "enso_diags" (not requested), "qbo", "area_mean_time_series" (not requested)
obs_ts = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series'
Copy link
Collaborator

@golaz golaz Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mache?

# 4. These sets depend on the ts task's atm_daily_180x360_aave subtask:
# "tropical_subseasonal"
ts_daily_subsection = "atm_daily_180x360_aave"
# 5. These sets depend on the ts task's rof_monthly subtask:
# "streamflow" (not requested)
# 6. These sets depend on the tc_analysis task:
# "tc_analysis"
# No extra parameters required for this set.
#
# Plus several other parameters:
grid = '180x360_aave'
reference_data_path = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/climatology'
short_name = 'v3.LR.historical_0051'

[[atm_monthly_180x360_aave_mvm]]
sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","diurnal_cycle","zonal_mean_2d_stratosphere",
# Using the groups from above:
# Group 1: climo atm monthly
# "lat_lon", "zonal_mean_xy", "zonal_mean_2d", "polar", "cosp_histogram", "meridional_mean_2d", "annual_cycle_zonal_mean", "zonal_mean_2d_stratosphere" "aerosol_aeronet" (not requested), "aerosol_budget" (not requested)
climo_subsection = "atm_monthly_180x360_aave"
reference_data_path = '/lcrc/group/e3sm2/ac.zhang40/E3SMv3/v3.LR.historical_eu11rc13/post/atm/180x360_aave/clim'
# Group 2: climo atm diurnal
# "diurnal_cycle"
climo_diurnal_frequency = "diurnal_8xdaily"
climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
# Group 3: ts atm monthly
# "enso_diags" (not requested), "qbo", "area_mean_time_series" (not requested)
ts_num_years = 30
ts_num_years_ref = 30
ts_subsection = "atm_monthly_180x360_aave"
# Group 4: ts atm daily
# "tropical_subseasonal" (not requested)
# Group 5: ts rof monthly
# "streamflow" (not requested)
# Group 6: tc_analysis
# "tc_analysis" (not requested)
# Other parameters:
diff_title = 'Difference'
grid = '180x360_aave'
ref_final_yr = 2014
ref_name = 'v3.LR.historical_0051'
ref_start_yr = 1985
ref_years = "1985-2014",
run_type = "model_vs_model"
short_name = 'v3.LR.historical'
short_ref_name = 'v3.LR.historical_0051_ref'
tag = 'model_vs_model'

[[ lnd_monthly_mvm_lnd ]]
# Depends on the climo task's land_monthly_climo subtask.
sets = "lat_lon_land",
climo_subsection = "land_monthly_climo"
# Other parameters:
diff_title = "Difference"
grid = 'native'
reference_data_path = "/lcrc/group/e3sm/ac.zhang40/tests/20231209.v3.LR.piControl-spinup.chrysalis_land_diags/post/lnd/native/clim"
ref_final_yr = 0100
ref_name = "20231209.v3.LR.piControl-spinup.chrysalis"
ref_start_yr = 0051
ref_years = "0051-0100",
run_type = "model_vs_model"
short_ref_name = "20231209.v3.LR.piControl-spinup"
short_name = v3.LR.piControl
swap_test_ref = False
tag = "model_vs_model"

[mpas_analysis]
active = True
anomalyRefYear = 1985
climo_years = "1985-2014",
enso_years = "1985-2014",
mesh = "IcoswISC30E3r5"
parallelTaskCount = 6
shortTermArchive = True
ts_years = "1985-2014",
walltime = "4:00:00"

[global_time_series]
# This task depends on the mpas_analysis task and the ts task's atm_monthly_glb and lnd_monthly_glb subtasks.
# 2 DEPRECATED PARAMETERS in zppy v3.0.0 :
# atmosphere_only (now just remove the ocean plots when setting plots_original)
# plot_names (plot names should now be explicitly set via the plots_atm/ice/lnd/ocn parameters)
active = True
climo_years ="1985-2014",
environment_commands = "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_plots_lnd_20250326"
Copy link
Collaborator

@forsyth2 forsyth2 Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: before merging, remove this line.

experiment_name = "v3.LR.historical_0051"
figstr = "v3.LR.historical_0051"
make_viewer = True
moc_file = "mocTimeSeries_1985-2014.nc"
plots_atm = "TREFHT,AODDUST"
Copy link
Collaborator

@golaz golaz Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explain. Does it mean we only get plots for TREFHT,AODDUST? Not the standard plots?

Copy link
Collaborator

@forsyth2 forsyth2 Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the original plots can be found in the single-page PDF but these 2 atm variables are displayed in the atm viewer. Although now I see AODDUST is missing.

> cd /lcrc/group/e3sm/ac.forsyth2/E3SMv3_20250331_try2/v3.LR.historical_0051/post/scripts
> grep AODDUST global_time_series_1985-2014.o719959 
2025-03-31 19:28:19,233 [ERROR]: coupled_global.py(set_var:184) >> "The data variable 'AODDUST' does not exist in the Dataset."
2025-03-31 19:28:19,233 [ERROR]: coupled_global.py(set_var:184) >> "The data variable 'AODDUST' does not exist in the Dataset."
2025-03-31 19:28:19,233 [ERROR]: coupled_global.py(set_var:184) >> "The data variable 'AODDUST' does not exist in the Dataset."
2025-03-31 19:28:19,233 [ERROR]: coupled_global.py(set_var:185) >> globalAnnual failed for AODDUST
2025-03-31 19:28:19,233 [ERROR]: coupled_global.py(set_var:185) >> globalAnnual failed for AODDUST
2025-03-31 19:28:19,233 [ERROR]: coupled_global.py(set_var:185) >> globalAnnual failed for AODDUST
2025-03-31 21:26:49,463 [ERROR]: coupled_global.py(process_data:274) >> glb region globalAnnual could not be computed for these variables: ['AODDUST', 'CH4PROD', 'SABG_PEN', 'QSNWCPICE_NODYNLNDUSE', 'FCOV']
2025-03-31 21:26:49,463 [ERROR]: coupled_global.py(process_data:274) >> glb region globalAnnual could not be computed for these variables: ['AODDUST', 'CH4PROD', 'SABG_PEN', 'QSNWCPICE_NODYNLNDUSE', 'FCOV']
2025-03-31 21:26:49,463 [ERROR]: coupled_global.py(process_data:274) >> glb region globalAnnual could not be computed for these variables: ['AODDUST', 'CH4PROD', 'SABG_PEN', 'QSNWCPICE_NODYNLNDUSE', 'FCOV']

ts > atm_monthly_glb is using default vars, specified in https://github.com/E3SM-Project/zppy/blob/main/zppy/defaults/default.ini

vars = string(default="FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U")

So, it probably makes sense to just remove AODDUST from plots_atm here in this example.

plots_lnd = "all"
ts_num_years = 30
ts_years ="1985-2014",
walltime = "10:00:00"
Copy link
Collaborator

@golaz golaz Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

10 hours? Is this because of the land?

Copy link
Collaborator

@forsyth2 forsyth2 Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From looking at elapsed time, it was closer to 6.7 hours actually needed to cover 30 years of data for Land's 354 variables.

Copy link
Collaborator Author

@chengzhuzhang chengzhuzhang Apr 1, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@forsyth2 I thought the ts-glb task can be time-consuming, but here, this task is only for plotting for land component, right? It is hard to think plotting out global averaged data can take this long...

Copy link
Collaborator

@forsyth2 forsyth2 Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chengzhuzhang Well it also includes the original 8 plots, and a atm plot, but yes the vast majority are land plots.

Looking at some previous results:

From Unified rc13 test on 2025-03-13

> cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_unified_rc13_20250313/v3.LR.historical_0051/post/scripts
> grep "Elapsed" global_time_series_1985-1995*o*
Elapsed time: 251 seconds
> cat emacs global_time_series_1985-1995.bash
# --plots_lnd FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILWATER_10CM,TSA,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR
# --start_yr 1985 --end_yr 1995

That's a little over 4 minutes (240+11 seconds) to cover 25 variables in plots_lnd for 10 years. So, that's 251 seconds / (25 vars * 10 years) ~ 1 second per var*year.

In this example cfg we are running all 353 land variables (row 1 of https://github.com/E3SM-Project/zppy-interfaces/blob/main/zppy_interfaces/global_time_series/zppy_land_fields.csv is the header) on 30 years of data, so that's 10,620 var*years => 10,620 seconds => 177 minutes => 2.95 hours, which is a little less than half the actual 6.7 hours of runtime seen here.

From testing of #697

See viewers_undefined_variables in #697 (comment).

> cd /lcrc/group/e3sm/ac.forsyth2/zppy_min_case_global_time_series_viewers_undefined_variables_output/test_pr697_20250328_try4/v3.LR.historical_0051/post/scripts
> grep -in "Elapsed" global_time_series*.o*
12796:Elapsed time: 8004 seconds
> cat global_time_series_1985-1995.bash
# --plots_lnd all
# --start_yr 1985 --end_yr 1995

That's 133.4 minutes to cover 353 land vars on 10 years of data => 8004 seconds / (353 vars * 10 years) => 8004 seconds / 3530 varyears => 2.26 seconds / varyear

For this example cfg on 30 years of data, we'd expect 353 vars * 30 years * 2.26 sec/var*year => 23,933.4 seconds => 398.89 minutes => 6.64 hours, which aligns with the actual 6.7 hours of runtime seen here.

Copy link
Collaborator

@forsyth2 forsyth2 Apr 1, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can also look at viewers_original_atm_plus_land from the #697 tests:

> cd /lcrc/group/e3sm/ac.forsyth2/zppy_min_case_global_time_series_viewers_original_atm_plus_land_output/test_pr697_20250328_try4/v3.LR.historical_0051/post/scripts
> grep -in "Elapsed" global_time_series*.o*
421:Elapsed time: 306 seconds
> cat global_time_series_1985-1995.bash
# --plots_original net_toa_flux_restom,global_surface_air_temperature,toa_radiation,net_atm_energy_imbalance,net_atm_water_imbalance
# --plots_atm TREFHT
# --plots_lnd FSH,RH2M,LAISHA,LAISUN
# --start_yr 1985 --end_yr 1995

That's 5 original plots, 1 atm plot, 4 land plots. 306 seconds / 10 vars * 10 years = 3.06 seconds/var*year. On this example cfg, we'd expect 353 land vars * 30 years * 3.06 sec = 32,405.4 sec => 540.09 minutes => 9 hours, which is more than 2 hours longer than it actually took.

Copy link
Collaborator

@tomvothecoder tomvothecoder Apr 1, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A possible debugging strategy is to add the performance profiling code, run 1 variable, then isolate the longest run time by percentage. For example, 3 seconds to run the variable, 1.5 seconds to run dataset.temporal.group_average() or another function which means 50% of the runtime is eaten up here.

Copy link
Collaborator

@tomvothecoder tomvothecoder Apr 1, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tomvothecoder thank you for chiming in. I looked at the output log at /lcrc/group/e3sm/ac.forsyth2/E3SMv3_20250331_try2/v3.LR.historical_0051/post/scripts. Roughly, 2 hours are use for computation and more than 4 and half hours are used in plotting line plots. I think we should be able to save time at least on plotting. I will try to share a minimum example here.

Hmmm, I don't recall plotting ever taking this long in e3sm_diags. During refactoring, it would either break, run successfully with incorrect plots, or run successfully with correct plots (all without stalling).

The example would be great to see.

Copy link
Collaborator Author

@chengzhuzhang chengzhuzhang Apr 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the example to reproduce:

zi-global-time-series --use_ocn True --input /lcrc/group/e3sm2/ac.wlin/E3SMv3/v3.LR.historical_0051 --input_subdir archive/ocn/hist --moc_file mocTimeSeries_1985-2014.nc --case_dir /lcrc/group/e3sm/ac.forsyth2/E3SMv3_20250331_try2/v3.LR.historical_0051 --experiment_name v3.LR.historical_0051 --figstr v3.LR.historical_0051 --color Blue --ts_num_years 30 --plots_original net_toa_flux_restom,global_surface_air_temperature,toa_radiation,net_atm_energy_imbalance,change_ohc,max_moc,change_sea_level,net_atm_water_imbalance --plots_atm TREFHT --plots_ice None --plots_lnd all --plots_ocn None --nrows 4 --ncols 2 --results_dir ./zi --regions glb,n,s --make_viewer True --start_yr 1985 --end_yr 2014

To make it a miniumal example, by changing some options:
zi-global-time-series --use_ocn False --input /lcrc/group/e3sm2/ac.wlin/E3SMv3/v3.LR.historical_0051 --input_subdir archive/ocn/hist --moc_file mocTimeSeries_1985-2014.nc --case_dir /lcrc/group/e3sm/ac.forsyth2/E3SMv3_20250331_try2/v3.LR.historical_0051 --experiment_name v3.LR.historical_0051 --figstr v3.LR.historical_0051 --color Blue --ts_num_years 30 --plots_original None --plots_atm TREFHT --plots_ice None --plots_lnd FSH --plots_ocn None --nrows 4 --ncols 2 --results_dir ./zi --regions glb,n,s --make_viewer True --start_yr 1985 --end_yr 2014

Copy link
Collaborator

@forsyth2 forsyth2 Apr 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chengzhuzhang @tomvothecoder Please see my analysis on E3SM-Project/zppy-interfaces#20 (comment)

Copy link
Collaborator Author

@chengzhuzhang chengzhuzhang Apr 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@forsyth2 @tomvothecoder please see my analysis and PR attempt to improve performance. E3SM-Project/zppy-interfaces#19

years = "1985-2014",

[ilamb]
# This task depends on the e3sm_to_cmip task's atm_monthly_180x360_aave and land_monthly subtasks.
active = True
nodes = 1
short_name = 'v3.LR.historical_0051'
ts_num_years = 30
walltime = "2:00:00"
years = "1985:2014:30"
Loading