Merge branch 'master' into fates_fuel_refactor

Author: adrifoster

.git-blame-ignore-revs

@@ -51,3 +51,4 @@ aa04d1f7d86cc2503b98b7e2b2d84dbfff6c316b
 1a49e547ba3c48fa483f9ae81a8f05adcd6b888c
 045d90f1d80f713eb3ae0ac58f6c2352937f1eb0
 753fda3ff0147837231a73c9c728dd9ce47b5997
+f112ba0bbf96a61d5a4d354dc0dcbd8b0c68145c

.gitignore

@@ -85,9 +85,6 @@ ctsm.input_data_list.previous
 # mksurfdata unit tests
 unit_test_build
 
-# Tools executables
-/tools/mkprocdata_map/mkprocdata_map
-
 # run_neon output directories
 /tools/site_and_regional/listing.csv
 /tools/site_and_regional/????/

README

@@ -1,6 +1,6 @@
-$CTSMROOT/README                                          04/19/2023
+$CTSMROOT/README                                          09/05/2024
 
-Community Terrestrial Systems Model (CTSM) science version 5.2  series -- source code, tools, 
+Community Terrestrial Systems Model (CTSM) science version 5.3 series -- source code, tools, 
 offline-build and test scripts. This gives you everything you need
 to run CTSM with CESM with the CMEPS driver and CDEPS data models to provide CRU NCEP or GSWP3 forcing data in 
 place of a modeled atmosphere.
@@ -51,7 +51,7 @@ tools ------------- CTSM Offline tools to prepare input datasets and process out
 cime_config ------- Configuration files of cime for compsets and CTSM settings
 bin/git-fleximod -- Script to manage the needed sub-component source directories (handled with git submodule)
 py_env_create ----- Script to setup the python environment for CTSM python tools using conda
-python ------------ Python modules used in tools and testing and automated checking of ALL CTSM python scirpts
+python ------------ Python modules used in tools and testing and automated checking of ALL CTSM python scripts
 
 Directory structure only for a CTSM checkout:
 
@@ -66,6 +66,8 @@ components/cmeps -------------------- CESM top level driver (for NUOPC driver [w
 components/cdeps -------------------- CESM top level data model shared code (for NUOPC driver).
 components/cism --------------------- CESM Community land Ice Sheet Model.
 components/mosart ------------------- Model for Scale Adaptive River Transport
+components/mizuRoute ---------------- Reached based river transport model for water routing 
+                                      (allows both gridded river and Hydrologic Responce Unit river grids)
 components/rtm ---------------------- CESM River Transport Model.
 
 Top level documentation ($CTSMROOT):
@@ -74,6 +76,7 @@ README ------------------- This file
 README.md ---------------- File that displays on github under https::/github.com/ESCOMP/CTSM.git
 README.rst --------------- File that displays under the project in github
 README_GITFLEXIMOD.rst --- Information on how to work with git-fleximod for CTSM
+WhatsNewInCTSM5.3.md ----- Overview document of the changes between ctsm5.2.0 and ctsm5.3.0
 CODE_OF_CONDUCT.md ------- Code of Conduct for how to work with each other on the CTSM project
 Copyright ---------------- CESM Copyright file
 doc/UpdateChangeLog.pl --- Script to add documentation on a tag to the
@@ -104,17 +107,16 @@ run_sys_tests --------------- Python script to send the standard CTSM testing of
 parse_cime.cs.status -------- Script to parse test status files cs.status.* created by create_test
                               (can be used along with run_sys_tests)
 doc/Quickstart.GUIDE -------- Quick guide to using NUOPC scripts.
-doc/IMPORTANT_NOTES --------- Some important notes about this version of 
-                                                 CTSM, configuration modes and namelist items 
-                                                 that are not validated or functional.
+doc/IMPORTANT_NOTES.md ------ Some important notes about this version of 
+                              CTSM, configuration modes and namelist items 
+                              that are not validated or functional.
 doc/ChangeLog --------------- Detailed list of changes for each model version.
 doc/ChangeSum --------------- Summary one-line list of changes for each 
                               model version.
 doc/UsersGuide -------------- CTSM Users Guide
-doc/IMPORTANT_NOTES --------- Some important notes on caveats for some configurations/namelist items
 
 bld/README ------------------ Description of how to use the build-namelist scripts.
-bld/build-namelist ---------- Script to build CTSM namelists.
+bld/build-namelist ---------- Lower level script to build CTSM namelists.
 
 cime_config/buildnml ------------- Build the CTSM namelist for CIME
 cime_config/buildlib ------------- Build the CTSM library
@@ -130,8 +132,6 @@ cime_config/usermods_dirs -------- Directories of sets of user-modification subd
 
 tools/mksurfdata_esmf --------- Directory to build program to create surface dataset 
                                 at any resolution.
-tools/mkprocdata_map ---------- Process history data from unstructed grids to a gridded 
-                                format.
 tools/mkmapgrids -------------- NCL script to create a SCRIP grid file for a regular lat/lon grid (deprecated)
 tools/crop_calendars ---------- Tools to process and process and create crop calendar datasets for CTSM
 tools/modify_input_files ------ Script to modify existing CTSM input datasets in standard ways
@@ -155,9 +155,9 @@ src/dyn_subgrid --- Dynamic land unit change
 src/init_interp --- Online interpolation
 scr/fates --------- FATES model and sub-directories 
                     Functionally Assembled Terrestrial Ecosystem Simulator (FATES)
-                    Experimental Ecosystem Demography model
+                    Ecosystem Demography model
 src/utils --------- Utility codes
-src/self_tests ---- Internal testing (unit tests run as a part of a CTSM simulation)
+src/self_tests ---- Internal testing (unit tests run as a part of a CTSM system test)
 src/unit_test_shr - Unit test shared modules for unit testing
 src/unit_test_stubs Unit test stubs that replicate CTSM code simpler
 

README.md

@@ -6,7 +6,7 @@
 
 The Community Terrestrial Systems Model.
 
-This includes the Community Land Model (CLM5.0 and CLM4.5) of the Community Earth System Model.
+This includes the Community Land Model of the Community Earth System Model.
 
 For documentation, quick start, diagnostics, model output and
 references, see
@@ -43,7 +43,7 @@ CTSM code management is provided primarily by:
 Software engineering team:
 - [Erik Kluzek](https://github.com/ekluzek)
 - [Bill Sacks](https://github.com/billsacks)
-- [Sam Levis](https://github.com/slevisconsulting)
+- [Sam Levis](https://github.com/slevis-lmwg)
 - [Adrianna Foster](https://github.com/adrifoster)
 - [Sam Rabin](https://github.com/samsrabin)
 - [Greg Lemieux](https://github.com/glemieux)

README_GITFLEXIMOD.rst

@@ -67,10 +67,10 @@ Switching to a different CTSM branch or tag
 
 If you have already checked out a branch or tag and **HAVE NOT MADE ANY
 MODIFICATIONS** it is simple to change your sandbox. Say that you
-checked out ctsm1.0.0 but really wanted to have ctsm1.1.0;
+checked out ctsm5.2.0 but really wanted to have ctsm5.3.0;
 you would simply do the following::
 
-  git checkout ctsm1.1.0
+  git checkout ctsm5.3.0
   ./bin/git-fleximod update
 
 You should **not** use this method if you have made any source code

WhatsNewInCTSM5.3.md

@@ -0,0 +1,53 @@
+Purpose and description of changes since ctsm5.2.005
+----------------------------------------------------
+
+Bring in updates needed for the CESM3.0 science capability/functionality "chill". Most importantly bringing
+in: CN Matrix to speed up spinup for the BGC model, updated surface datasets, updated Leung 2023 dust emissions,
+explicit Air Conditioning for the Urban model, updates to crop calendars. For clm6_0 physics these options are now
+default turned on in addition to Sturm snow, and excess ice.
+
+Changes to CTSM Infrastructure:
+===============================
+
+ - manage_externals removed and replaced by git-fleximod
+ - Ability to handle CAM7 in LND_TUNING_MODE
+
+Changes to CTSM Answers:
+========================
+
+ Changes to defaults for clm6_0 physics:
+  - Urban explicit A/C turned on
+  - Snow thermal conductivity is now Sturm_1997
+  - New IC file for f09 1850
+  - New crop calendars
+  - Dust emissions is now Leung_2023
+  - Excess ice is turned on
+  - Updates to MEGAN for BVOC's
+  - Updates to BGC fire method
+
+ Changes for all physics versions:
+
+  - Parameter files updated
+  - FATES parameter file updated
+  - Glacier region 1 is now undefined
+  - Update in FATES transient Land use
+  - Pass active glacier (CISM) runoff directly to river model (MOSART)
+  - Add the option for using matrix for Carbon/Nitrogen BGC spinup
+
+New surface datasets:
+=====================
+
+- With new surface datasets the following GLC fields have region "1" set to UNSET:
+     glacier_region_behavior, glacier_region_melt_behavior, glacier_region_ice_runoff_behavior
+- Updates to allow creating transient landuse timeseries files going back to 1700.
+- Fix an important bug on soil fields that was there since ctsm5.2.0. This results in mksurfdata_esmf now giving identical answers with a change in number of processors, as it should.
+- Add in creation of ne0np4.POLARCAP.ne30x4 surface datasets.
+- Add version to the surface datasets.
+- Remove the --hires_pft option from mksurfdata_esmf as we don't have the datasets for it.
+- Remove VIC fields from surface datasets.
+
+New input datasets to mksurfdata_esmf:
+======================================
+
+- Updates in PFT/LAI/soil-color raw datasets (now from the TRENDY2024 timeseries that ends in 2023), as well as two fire datasets (AG fire, peatland), and the glacier behavior dataset.
+

bld/CLMBuildNamelist.pm

@@ -1665,6 +1665,7 @@ sub process_namelist_inline_logic {
   setup_logic_demand($opts, $nl_flags, $definition, $defaults, $nl);
   setup_logic_surface_dataset($opts,  $nl_flags, $definition, $defaults, $nl, $envxml_ref);
   setup_logic_dynamic_subgrid($opts,  $nl_flags, $definition, $defaults, $nl);
+  setup_logic_exice($opts, $nl_flags, $definition, $defaults, $nl, $physv);
   if ( remove_leading_and_trailing_quotes($nl_flags->{'clm_start_type'}) ne "branch" ) {
     setup_logic_initial_conditions($opts, $nl_flags, $definition, $defaults, $nl, $physv);
   }
@@ -1894,7 +1895,7 @@ sub process_namelist_inline_logic {
   #################################
   # namelist group: exice_streams #
   #################################
-  setup_logic_exice($opts, $nl_flags, $definition, $defaults, $nl, $physv);
+  setup_logic_exice_streams($opts, $nl_flags, $definition, $defaults, $nl, $physv);
 
   ##########################################
   # namelist group: clm_temperature_inparm #
@@ -2506,8 +2507,9 @@ sub setup_logic_surface_dataset {
   # consistent with it
   # MUST BE AFTER: setup_logic_demand which is where flanduse_timeseries is set
   #
-  my ($opts, $nl_flags, $definition, $defaults, $nl, $xmlvar_ref) = @_;
+  my ($opts_in, $nl_flags, $definition, $defaults, $nl, $xmlvar_ref) = @_;
 
+  my $opts = $opts_in;
   $nl_flags->{'flanduse_timeseries'} = "null";
   my $flanduse_timeseries = $nl->get_value('flanduse_timeseries');
   if (defined($flanduse_timeseries)) {
@@ -2523,6 +2525,11 @@ sub setup_logic_surface_dataset {
   if ($flanduse_timeseries ne "null" && &value_is_true($nl_flags->{'use_cndv'}) ) {
      $log->fatal_error( "dynamic PFT's (setting flanduse_timeseries) are incompatible with dynamic vegetation (use_cndv=.true)." );
   }
+  # Turn test option off for NEON until after XML is interpreted
+  my $test_files = $opts->{'test'};
+  if ( &value_is_true($nl_flags->{'neon'})) {
+     $opts->{'test'} = 0;
+  }
   #
   # Always get the crop version of the datasets now and let the code turn it into the form desired
   # Provided this isn't with FATES on
@@ -2548,7 +2555,7 @@ sub setup_logic_surface_dataset {
                  'use_crop'=>$nl_flags->{'use_crop'} );
   }
   #
-  # Expand the XML variables for NEON cases so that NEONSITE will be used
+  # Expand the XML variables for NEON cases so that NEONSITE will be used and test for existence
   #
   if ( &value_is_true($nl_flags->{'neon'}) ) {
      my $fsurdat = $nl->get_value($var);
@@ -2557,6 +2564,9 @@ sub setup_logic_surface_dataset {
         my $group = $definition->get_group_name($var);
         $nl->set_variable_value($group, $var, $newval);
         $log->verbose_message( "This is a NEON site and the fsurdat file selected is: $newval" );
+        if ( $test_files and ($newval !~ /null|none/) and (! -f remove_leading_and_trailing_quotes($newval) ) ) {
+          $log->fatal_error("file not found: $var = $newval");
+        }
      }
   }
 }
@@ -2571,10 +2581,12 @@ sub setup_logic_initial_conditions {
   #
   # MUST BE AFTER: setup_logic_demand   which is where flanduse_timeseries is set
   #         AFTER: setup_logic_irrigate which is where irrigate is set
+  #         AFTER: setup_logic_exice    which is where use_excess_ice is set
   my ($opts, $nl_flags, $definition, $defaults, $nl, $physv) = @_;
 
   my $var = "finidat";
   my $finidat = $nl->get_value($var);
+  $nl_flags->{'excess_ice_on_finidat'} = "unknown";
   if ( $nl_flags->{'clm_start_type'} =~ /cold/ ) {
     if (defined $finidat ) {
       $log->warning("setting $var (either explicitly in your user_nl_clm or by doing a hybrid or branch RUN_TYPE)\n is incomptable with using a cold start" .
@@ -2623,7 +2635,7 @@ sub setup_logic_initial_conditions {
        $settings{'sim_year'}     = $st_year;
     }
     foreach my $item ( "mask", "maxpft", "irrigate", "glc_nec", "use_crop", "use_cn", "use_cndv",
-                       "use_fates",
+                       "use_fates", "use_excess_ice",
                        "lnd_tuning_mode",
                      ) {
        $settings{$item}    = $nl_flags->{$item};
@@ -2644,6 +2656,7 @@ sub setup_logic_initial_conditions {
     my $done = 2;
     do {
        $try++;
+       $nl_flags->{'excess_ice_on_finidat'} = $settings{'use_excess_ice'};
        add_default($opts,  $nl_flags->{'inputdata_rootdir'}, $definition, $defaults, $nl, $var, %settings );
        # If couldn't find a matching finidat file, check if can turn on interpolation and try to find one again
        $finidat = $nl->get_value($var);
@@ -2743,6 +2756,13 @@ SIMYR:    foreach my $sim_yr ( @sim_years ) {
         $log->fatal_error("$useinitvar is being set for you but a $var was not found, so $useinitvar, init_interp_attributes, and finidat must not be set correctly for this configuration in the namelist_default file" );
      }
   }
+
+  # this check has to be here and not earlier since use_init_interp is set here and hillslope is already set above in setup_logic_hillslope
+  if ( &value_is_true($nl->get_value($useinitvar)) && value_is_true($nl->get_value("use_hillslope")) ) {
+     $log->warning("WARNING: You have set use_hillslope while $useinitvar is TRUE.\n This means all hillslope columns in a gridcell will read identical values" .
+                   " from initial conditions. If you are sure you want this behaviour:")
+  }
+
 } # end initial conditions
 
 #-------------------------------------------------------------------------------
@@ -4957,18 +4977,39 @@ sub setup_logic_cnmatrix {
 #-------------------------------------------------------------------------------
 sub setup_logic_exice {
   #
-  # excess ice streams
+  # excess ice streams, must be set before initial conditions
   #
   my ($opts, $nl_flags, $definition, $defaults, $nl, $physv) = @_;
   add_default($opts, $nl_flags->{'inputdata_rootdir'}, $definition, $defaults, $nl, 'use_excess_ice', 'phys'=>$physv->as_string()); 
   my $use_exice = $nl->get_value( 'use_excess_ice' );
+  # Put use_exice into nl_flags so can be referenced later
+  if ( value_is_true($use_exice) ) {
+     $nl_flags->{'use_excess_ice'} = ".true.";
+  } else {
+     $nl_flags->{'use_excess_ice'} = ".false.";
+  }
+}
+
+#-------------------------------------------------------------------------------
+sub setup_logic_exice_streams {
+  #
+  # excess ice streams
+  # Run after initial conditions found as well as after setup_logic_exice
+  #
+  my ($opts, $nl_flags, $definition, $defaults, $nl, $physv) = @_;
+  my $use_exice = $nl_flags->{'use_excess_ice'};
+  my $excess_ice_on_finidat = $nl_flags->{'excess_ice_on_finidat'};
   my $use_exice_streams = $nl->get_value( 'use_excess_ice_streams' );
   my $finidat = $nl->get_value('finidat');
   # If coldstart and use_excess_ice is on:
   if ( ( (not defined($use_exice_streams)) && value_is_true($use_exice) ) && string_is_undef_or_empty($finidat) ) {
      $nl->set_variable_value('exice_streams', 'use_excess_ice_streams' , '.true.');
      $use_exice_streams = '.true.';
-     # if excess ice is turned off
+  # If an finidat file was selected and use_excess_ice is on:
+  } elsif ( (not defined($use_exice_streams)) && value_is_true($use_exice) && (not value_is_true($excess_ice_on_finidat)) ) {
+     $nl->set_variable_value('exice_streams', 'use_excess_ice_streams' , '.true.');
+     $use_exice_streams = '.true.';
+  # if excess ice is turned off
   } elsif ( (not defined($use_exice_streams)) && (not value_is_true($use_exice)) ) {
      $use_exice_streams = '.false.';
   # Checking for cold clm_start_type and not finidat here since finidat can be not set set in branch/hybrid runs and

bld/README

@@ -4,16 +4,16 @@ CLM build and configure directory and scripts. Scripts to help
 you prepare to build CLM as a component within CESM, and setup
 a namelist for it.
 
+This is a lower level script called from with CESM/CIME.
+
 Important files/directories:
 
---------- Configure and build scripts
---------- (These scripts are also used by the cesm/cime scripts)
+--------- Namelist build scripts
 
 config_files/clm_phys_vers.pm ------------- Perl module to handle different CLM versions
-config_files/config_definition_ctsm.xml --- XML file defining all CTSM configuration items
+config_files/config_definition_ctsm.xml --- XML file defining CTSM configuration items (mainly physics version)
 
 --------- Scripts to build the namelists
---------- (These scripts are also used by the cesm/cime scripts)
 build-namelist --- Build the namelists needed
 
 
@@ -26,17 +26,17 @@ unit_testers --- Directory of scripts to test scipts in this directory
 ---------- XML Files describing namelists in namelist_files
 namelist_files/namelist_defaults_ctsm.xml --------- List of default values for the ctsm namelist
 namelist_files/namelist_defaults_overall.xml ------ List of default values for overall settings
-namelist_files/namelist_defaults_usr_files.xml ---- List of default values for the user-files
-namelist_files/namelist_definition_ctsm.xml -------- Definition of all namelist items for ctsm
+namelist_files/namelist_defaults_usr_files.xml ---- List of default values for the user-files (deprecated)
+namelist_files/namelist_definition_ctsm.xml ------- Definition of all namelist items for ctsm
 namelist_files/namelist_definition.xsl ------------ Describes how to view the xml file as html
-namelist_files/namelist_defaults_drydep.xml ------- List of default values for the dry deposition module.
 namelist_files/use_cases -------------------------- Specific configurations that build-namelist uses
 namelist_files/use_cases/README ------------------- File explaining the naming convention for use_cases
 
 ---------- Driver namelist files, duplicated information from cime/driver/cime_config
 namelist_files/namelist_defaults_drv.xml ---------- List of default values for driver namelist defaults
-namelist_files/namelist_defaults_drydep.xml ------- List of default values for dry deposition fields 
+namelist_files/namelist_defaults_drydep.xml ------- List of default values for dry deposition and MEGAN fields 
 namelist_files/namelist_defaults_fire_emis.xml ---- List of default values for fire emission fields
+namelist_files/namelist_defaults_dust_emis.xml ---- List of default values for the dust emissions module.
 namelist_files/namelist_definition_drv.xml -------- Definition of all driver namelist items
 namelist_files/namelist_definition_drv_flds.xml --- Definition of add driver fieldsnamelist items