Skip to content

ELM Example Case VI. ELM running from pre stage restarts

Jump to bottom
Fengming Yuan edited this page May 30, 2023 · 4 revisions

ELM Examples of Setting-up, Configuring, Building, and Running VI.

ELM-ATS case, restarting from pre-run ending at model year 1980, at Portage River outlet nearby-town, Oakharbor, OHIO.

image

NOTE:

  • In this example, there are 3 user-defined ENVs variables of directories: $E3SM_INPUTDATA, $E3SM_ROOT, $PROJECT_E3SM, for inputdata, e3sm model source codes, and root directory for cases and runs, respectively. USERS must have created these, and cloned both data and e3sm codes.

FOR this ELM-ATS example, the source ELM codes are branch EMI-ats as ELM-ATS branch

  • For ATS installed inc/libs, there are at least 2 user-defined ENVs variables of directories: $AMANZI_TPLS_DIR, $ATS_DIR, for TPLs and ATS with elm-ats interface API installed, respectively. USERS must have created these, and BOTH are built successfully.

  • $E3SM_INPUTDATA is cloned repository of a few site-level full datasets to run offline ELM, from pt-e3sm-inputdata.

Under this directory, there is an ats input subdirectory as well $E3SM_INPUTDATA/lnd/clm2/ats/

The 'ELM_USR_NAME' for this example is called 1x1pt_Oakharbor-GRID

  • Upon user's machine, $MACH_NAME is needed for E3SM. In these examples, 'mymac' for MacOS/linux, 'wsl' for Windows Subsystem for Linux, OR, 'cades' for a cluster in ORNL, may be directly used.

HERE we manually customize E3SM Land Model to run with option of 'CPL_BYPASS' (coupler_bypass), developed by Dr. Daniel Ricciuto at ORNL. This configuration will run offline ELM about 3 times faster and vey-slightly different results, although not bit-for-bit

++++++++++++++++++++++++++++++++++++++++++++

There are 4-5 forms of this configuration, upon what meteorological dataset to drive the land model. We will demonstrate 1 of them as following: (2) 'GSWP3' - point metdata extracted from half-degree global datasets , which is CPL_BYPASS version of ELM Example Case II. Point Mode with CPL_BYPASS, but with different forcing.

++++++++++++++++++++++++++++++++++++++++++++

CASE VI: single-point user-defined satm-elm (CPL_BYPASS metdata_type of 'GSWP3) COMPSET of 20TRCNPRDCTCBC.

  • STEP 1. Create a PTCLM case, with --compset ICB20TRCNPRDCTCBC
  1) cd $E3SM_ROOT/cime/scripts

  2) ./create_newcase --case $PROJECT_E3SM/cases/testcase_column_tr --res ELM_USRDAT --mach $MACH_NAME --compiler gnu --compset ICB20TRCNPRDCTCBC --walltime 06:00:00
  • STEP 2. Configure the case created in STEP 1 (extracted 1 grid dataset from half-degree global one, namely '1x1pt_Oakharbor' from 1901-2014)
  3) cd $PROJECT_E3SM/cases/testcase_column_tr

  4) ./xmlchange, (OR, vi env_run.xml)
             --id ELM_USRDAT_NAME --val 1x1pt_Oakharbor-GRID

By default, model will run for 5 model-days, with restart file output by end of simulation. So need to change this setting as following

  5) ./xmlchange --id STOP_OPTION --val nyears
     ./xmlchange --id STOP_N --val 165
     ./xmlchange --id REST_N --val 10  (NOTE: this is optional, meaning output restart files every 10 years)

By default, the model run on machine is upon some pre-configured PE setting. For an example, the case setup by now MAY or MAY NOT run model on 1 node with 8 MPI Task or '-1'. If so, that's not good for 1 PT model, so vi env_mach_pes.xml as following so that model runs on 1 node with 1 MPI Tasks

  6) vi env_mach_pes.xml

    <entry id="NTASKS">
      <type>integer</type>
      <values>
        <value compclass="ATM">1</value>
        <value compclass="CPL">1</value>
        <value compclass="OCN">1</value>
        <value compclass="WAV">1</value>
        <value compclass="GLC">1</value>
        <value compclass="ICE">1</value>
        <value compclass="ROF">1</value>
        <value compclass="LND">1</value>
        <value compclass="ESP">1</value>
        <value compclass="IAC">1</value>
      </values>
      <desc>number of tasks for each component</desc>
    </entry> 

    <entry id="NTASKS_PER_INST">
      <type>integer</type>
      <values>
        <value compclass="ATM">1</value>
        <value compclass="CPL">1</value>
        <value compclass="OCN">1</value>
        <value compclass="WAV">1</value>
        <value compclass="GLC">1</value>
        <value compclass="ICE">1</value>
        <value compclass="ROF">1</value>
        <value compclass="LND">1</value>
        <value compclass="ESP">1</value>
        <value compclass="IAC">1</value>
      </values>
      <desc>number of tasks for each component</desc>
    </entry> 

OR, 
./xmlchange NTASKS=1
./xmlchange NTASKS_PER_INST=1

IF PNETCDF is not available,
./xmlchange PIO_TYPENAME=netcdf

THEN, issue the following command:

  7) ./case.setup 
(./case.setup --clean FOR cleaning-up)

The above command may or may not be successful (mostly with errors relevant to missing input files). AND it's also missing of metdata and setting for CPL_BYPASS. So, modifying user_nl_elm as following - please replace $E3SM_INPUTDATA with actually absoluted path:

 metdata_type = 'gswp3'
 metdata_bypass = '$E3SM_INPUTDATA/atm/datm7/atm_forcing.datm7.GSWP3.0.5d.v2.c180716_Oakharbor-Grid/cpl_bypass_full'
 fsurdat = '$E3SM_INPUTDATA/lnd/clm2/surfdata_map/surfdata_1x1pt_Oakharbor-GRID_simyr1850_c360x720_c20230520.nc'
 flanduse_timeseries = '$E3SM_INPUTDATA/lnd/clm2/ats/testcase_column_sp.elm.r.0601-01-01-00000.nc'
 finidat =
 aero_file = '$E3SM_INPUTDATA//atm/cam/chem/trop_mozart_aero/aero/aerosoldep_monthly_1850_mean_1.9x2.5_c090803.nc'
 CO2_file = '$E3SM_INPUTDATA//atm/datm7/CO2/fco2_datm_1765-2007_c100614.nc'
 nyears_ad_carbon_only = 25
 spinup_mortality_factor = 10

 use_ats = .true.
 ats_inputdir = '$E3SM_INPUTDATA/lnd/clm2/ats'
 ats_inputfile = 'column_elm4ats.xml'

NOTE: (1)the last 3 options are for ATS. If use_ats = .false., ELM-ATS model will run as ELM only (but with ATS coupled codes).

(2) Those options are to instruct model read datasets from indicated directory or files.

In this case, the 'metdata' is located in directory metdata_bypass with 7 files GSWP3_*_1901-2014_z19.nc, and an ascii file zone_mappings.txt. So, make sure those files are existed in those inputdata directories.

TWO (2) surface data, 'fsurdat=...', 'flanduse_timeseries=...', are used for spinups and transient runs. Make sure they are in the specified directories and file names are correct. flanduse_timeseries=' ' implies that no dynamical LULC data for transient stage. IF NOT, edit as NEEDED.

(3) an example user_nl_elm_column_tr file is under $E3SM_INPUTDATA/lnd/clm2/ats

THEN, re-issue command ./case.setup

  • STEP 3. Copy pre-stage restart files to RUN_DIR

Under $E3SM_INPUTDATA/lnd/clm2/ats, there are 6 retart files produced from pre-run until date/time of 1980-01-01-00000: 2 'rpointer.*' files, rpointer.drv, rpointer.lnd, and 4 restart data files: testcase_column_tr.cpl.r.1980-01-01-0000.nc, testcase_column_tr.elm.r.1980-01-01-0000.nc, testcase_column_tr.elm.rh0.1980-01-01-0000.nc, testcase_column_tr.elm.h0.1980-01-01-0000.nc. The first 2 file names are indicated in those 2 rpointer.* files above.

COPY those 5 files to case RUN directory. THEN issue the following command:

./xmlchange CONTINUE_RUN=TRUE
./xmlchange STOP_N=35

This will tell model to run from starting time of 1980-01-01-0000, using those restart files just copied, for another 35 years (until end of 2014).

  • STEP 4. Build the case from STEP 1-2

FOR ELM-ATS coupled, ONE more editing or checking IS required before code compiling.

(for newer version ELM since 2021-Oct), checking CAREFULLY cmake_macros/userdefined_cmake, which IS EXACTLY for ELM-ATS interface:

string(APPEND CONFIG_ARGS " ")
string(APPEND CPPDEFS " -DCPL_BYPASS ")
set(AMANZI_TPLS_DIR "$ENV{AMANZI_TPLS_DIR}")
set(ATS_DIR "$ENV{ATS_DIR}")
if (COMP_CLASS STREQUAL lnd)
  if (NOT ${AMANZI_TPLS_DIR} STREQUAL "")
    string(APPEND FFLAGS " -I${AMANZI_TPLS_DIR}/trilinos-13-0-afc4e525/include")
    string(APPEND FFLAGS " -I${AMANZI_TPLS_DIR}/SEACAS/include ")
    string(APPEND FFLAGS " -I${AMANZI_TPLS_DIR}/petsc-3.16/include -I${AMANZI_TPLS_DIR}/pflotran/src ")
    if (NOT ${ATS_DIR} STREQUAL "")
      string(APPEND CPPDEFS " -DUSE_ATS_LIB ")
      string(APPEND FFLAGS " -I${ATS_DIR}/include ")
    endif()
  endif()
endif()
if (COMP_CLASS STREQUAL cpl)
  string(APPEND LDFLAGS " -lstdc++")
  if (NOT ${AMANZI_TPLS_DIR} STREQUAL "")
    string(APPEND LDFLAGS " -L${AMANZI_TPLS_DIR}/lib")
    string(APPEND LDFLAGS " -L${AMANZI_TPLS_DIR}/trilinos-13-0-afc4e525/lib")
    string(APPEND LDFLAGS " -L${AMANZI_TPLS_DIR}/SEACAS/lib ")
    string(APPEND LDFLAGS " -L${AMANZI_TPLS_DIR}/petsc-3.16/lib -L${AMANZI_TPLS_DIR}/pflotran/src ")
    if (NOT ${ATS_DIR} STREQUAL "")
      string(APPEND LDFLAGS " -L${ATS_DIR}/lib -lerror_handling -latk -lfunctions -lgeometry -lgeochemutil -lgeochemsolvers -lgeochembase -lgeochemrxns -lgeochemistry -lmesh -lmesh_audit -lmesh_simple -lmesh_mstk -lmesh_extracted -lmesh_logical -lmesh_factory -ldbg -lwhetstone -ldata_structures -lmesh_functions -loutput -lstate -lsolvers -ltime_integration -loperators -lpks -lchemistry_pk -ltransport -lshallow_water -lats_operators -lats_eos -lats_surf_subsurf -lats_generic_evals -lats_column_integrator -lats_pks -lats_energy_relations -lats_energy -lats_flow_relations -lats_flow -lats_transport -lats_sed_transport -lats_deform -lats_surface_balance -lats_bgc -lats_mpc_relations -lats_mpc -lelm_ats")
    endif()
  endif()
endif()
  • PAY attention to 3 critical hard-wired path/file names: trilinos-13-0-afc4e525, petsc-3.16, -L${ATS_DIR}/lib -l....... Especially, the link library list (long and varied upon ATS building options). YOU may check a txt file, link_line , under ${ATS_DIR}/lib. (As mentioned before, ${ATS_DIR} is your ATS installation directory)

THEN, build the model:

  8) ./case.build
(./case.build --clean      #cleaning-up ALL E3SM model components
 ./case.build --clean lnd  #cleaning-up LND component
 ./case.build --clean-all  #cleaning-up ALL model components and External components)
  • STEP 4. Run the case
  ./case.submit

NOTES: *(a) the column_elm4ats.xml example under $E3SM_INPUTDATA/lnd/clm2/ats has a few hard-wired PATH/FILE names, So must edit it as what really copied or changed in your case.

*(b) the run direcory is:$PROJECT_E3SM/cases/testcase_column_tr/run

*(c) The run length is set by STOP_N and STOP_OPTION, with monthly output, and every 10-year of restart files.

*(d) If you'd like what-ever time-length of the simulation, editing env_run.xml, for an example, like the following:

  --id STOP_OPTION nyears
  --id STOP_N 5
 (These editings will let model run 5 years, with monthly outputs of default variables, AND will produce 3 restart files and modify files of `rpointer.drv` and `rpointer.lnd`)

++++++++++++++++++++++++++++++++++++++++++++

CASE VI-2: 5-point user-defined satm-elm (CPL_BYPASS metdata_type of 'GSWP3) COMPSET of 20TRCNPRDCTCBC, i.e. a hillslope toy case.

Same STEPs as above, except for:

  • STEP 1 (2): case name shall be testcase_hillslope_tr (or whatever you named), like
./create_newcase --case $PROJECT_E3SM/cases/testcase_hillslope_tr --res ELM_USRDAT --mach $MACH_NAME --compiler gnu --compset ICB20TRCNPRDCTCBC --walltime 06:00:00

  • STEP 2 (4): USER_DATA_NAME is 5x1pt_Oakharbor-Grid

  • STEP 2 (6): (optional) NTASKS any number from 1 - 5 (i.e. parallel setting for MPI cores)

  • STEP 2 (7): user_nl_elm example is user_nl_elm_hillslope or user_nl_elm_hillslope_tr, with restart files named as testcase_hillslope_tr.*.nc

Updated @ 2025-07-02

Clone this wiki locally