Merge pull request #4749 from ESMCI/doc_update

- Updates PR/issue templates
- Updates function doc strings
- Updates README.md
- Updates API docs generation to sphinx autosummary
- Updates documentation organization into two main sections: `ccs` and `system testing`.
- Updates documentation for model configuration
- Updates documentation for XML variables
- Adds documentation generation for XML schema
- Adds documentation for node failure/mpi retry
- Moves users_guide to ccs/
- Moves build_cpl to ccs/model-configuration/variables/component.rst
- Moves misc_tools/ect.rst to tools/ect.rst
- Moves what_cime to ccs/index.rst
- Moves users_guide/testing.rst to system_testing.rst
- Removes xml_files/, files no longer in CIME
- Removes misc_tools/load-balancer-tool.rst, no longer in CIME

Author: jasonb5

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,31 @@
+## What happened?
+<!--
+Thanks for reporting a bug!
+Please describe what you were trying to do, what happened, what went wrong.
+-->
+
+## What did you expect to happen?
+<!--
+What did you expect to happen?
+-->
+
+## Environment
+<!--
+Describe the environment the issue occurred in e.g. machine, module system, batch system, etc.
+-->
+
+## Describe steps to recreate issue.
+<!--
+Describe the steps to recreate the issue.
+Provide as much detail as possible.
+-->
+
+## Relevant logs
+<!--
+Include as many relevant logs.
+-->
+
+## Anything else we need to know?
+<!--
+Is there anything else that could help?
+-->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,19 @@
+## Describe the problem your experiencing
+<!--
+Please provide a clear and concise description of what the problem is e.g. I'm always frustrated when [...]
+-->
+
+## Describe the solution you'd like
+<!--
+A clear and consise description of what you want to happen.
+-->
+
+## Describe alternative solutions
+<!--
+A clear and concise description of any alternative solutions or features you've considered.
+-->
+
+## Additional context
+<!--
+Add any other context about the feature request here.
+-->

.github/PULL_REQUEST_TEMPLATE

@@ -0,0 +1,15 @@
+## Description
+<!--
+    Please include a summary of the change and which issues it fixed.
+    Please also include relevant motivation and context.
+-->
+
+- Closes #<ISSUE_NUMBER_HERE>
+
+## Checklist
+- [ ] My code follows the style guidlines of this proejct (black formatting)
+- [ ] I have performed a self-review of my own code
+- [ ] My changes generate no new warnings
+- [ ] I have added tests that excerise my feature/fix and existing tests continue to pass
+- [ ] I have commented my code, particularly in hard-to-understand areas
+- [ ] I have made corresponding additions and changes to the documentation

.github/pull_request_template.md

@@ -2,6 +2,14 @@
     "version": "2.0.0",
     "tasks": [
         {
+            "label": "Build documentation",
+            "type": "shell",
+            "command": "source ${workspaceFolder}/.venv/bin/activate && sphinx-autobuild --ignore generated/ . _build/html",
+            "options": {
+                "cwd": "${workspaceFolder}/doc/source"
+            }
+        },
+	{
             "label": "Docker Build (base)",
             "type": "shell",
             "command": "docker build -t ghcr.io/esmci/cime:dev --target base .",

.vscode/tasks.json

@@ -8,18 +8,15 @@
 
 import os, logging
 
-###############################################################################
+
 def append_status_cprnc_log(msg, logfile_name, test_dir):
-    ###############################################################################
     try:
         append_status(msg, logfile_name, caseroot=test_dir)
     except IOError:
         pass
 
 
-###############################################################################
 def compare_namelists(case, baseline_name, baseline_root, logfile_name):
-    ###############################################################################
     log_lvl = logging.getLogger().getEffectiveLevel()
     logging.disable(logging.CRITICAL)
     success = case.case_cmpgen_namelists(
@@ -32,9 +29,7 @@ def compare_namelists(case, baseline_name, baseline_root, logfile_name):
     return success
 
 
-###############################################################################
 def compare_history(case, baseline_name, baseline_root, log_id):
-    ###############################################################################
     real_user = case.get_value("REALUSER")
     with EnvironmentContext(USER=real_user):
         baseline_full_dir = os.path.join(
@@ -54,7 +49,6 @@ def compare_history(case, baseline_name, baseline_root, log_id):
         return result, comments
 
 
-###############################################################################
 def compare_test_results(
     baseline_name,
     baseline_root,
@@ -65,18 +59,16 @@ def compare_test_results(
     namelists_only=False,
     hist_only=False,
 ):
-    ###############################################################################
-    """
-    Compares with baselines for all matching tests
+    """Compares with baselines for all matching tests
 
     Outputs results for each test to stdout (one line per test); possible status
     codes are: PASS, FAIL, SKIP. (A SKIP denotes a test that did not make it to
     the run phase or a test for which the run phase did not pass: we skip
     baseline comparisons in this case.)
 
     In addition, creates files named compare.log.BASELINE_NAME.TIMESTAMP in each
-    test directory, which contain more detailed output. Also creates
-    *.cprnc.out.BASELINE_NAME.TIMESTAMP files in each run directory.
+    test directory, which contain more detailed output. Also creates "*.cprnc.out.BASELINE_NAME.TIMESTAMP
+    files in each run directory.
 
     Returns True if all tests generated either PASS or SKIP results, False if
     there was at least one FAIL result.

CIME/compare_test_results.py

@@ -12,18 +12,18 @@ def create_cs_status(test_root, test_id, extra_args="", filename=None):
     """Create a test suite-specific cs.status file from the template
 
     Arguments:
-    test_root (string): path to test root; the file will be put here. If
-        this directory doesn't exist, it is created.
-    test_id (string): test id for this test suite. This can contain
-        shell wildcards if you want this one cs.status file to work
-        across multiple test suites. However, be careful not to make
-        this too general: for example, ending this with '*' will pick up
-        the *.ref1 directories for ERI and other tests, which is NOT
-        what you want.
-    extra_args (string): extra arguments to the cs.status command
-        (If there are multiple arguments, these should be in a space-delimited string.)
-    filename (string): name of the generated cs.status file. If not
-        given, this will be built from the test_id.
+        test_root (string): path to test root; the file will be put here. If
+            this directory doesn't exist, it is created.
+        test_id (string): test id for this test suite. This can contain
+            shell wildcards if you want this one cs.status file to work
+            across multiple test suites. However, be careful not to make
+            this too general: for example, ending this with '*' will pick up
+            the *.ref1 directories for ERI and other tests, which is NOT
+            what you want.
+        extra_args (string): extra arguments to the cs.status command
+            (If there are multiple arguments, these should be in a space-delimited string.)
+        filename (string): name of the generated cs.status file. If not
+            given, this will be built from the test_id.
     """
     cime_root = CIME.utils.get_cime_root()
     tools_path = os.path.join(cime_root, "CIME", "Tools")

CIME/cs_status_creator.py

@@ -65,10 +65,14 @@ def copy_histfiles(case, suffix, match_suffix=None):
     This can allow you to temporarily "save" these files so they won't be blown
     away if you re-run the case.
 
-    case - The case containing the files you want to save
-    suffix - The string suffix you want to add to saved files, this can be used to find them later.
-
-    returns (comments, num_copied)
+    Args:
+        case (CIME.case.Case): The case containing the files you want to save
+        suffix (str): The string suffix you want to add to saved files, this can be used to find them later.
+
+    Returns:
+        Tuple str, int: (comments, num_copied)
+            comments is a string containing the comments generated by this function
+            num_copied is the number of files copied
     """
     rundir = case.get_value("RUNDIR")
     ref_case = case.get_value("RUN_REFCASE")
@@ -411,18 +415,22 @@ def _compare_hists(
 
 
 def compare_test(case, suffix1, suffix2, ignore_fieldlist_diffs=False):
-    """
-    Compares two sets of component history files in the testcase directory
+    """Compares two sets of component history files in the testcase directory
 
-    case - The case containing the hist files to compare
-    suffix1 - The suffix that identifies the first batch of hist files
-    suffix1 - The suffix that identifies the second batch of hist files
-    ignore_fieldlist_diffs (bool): If True, then: If the two cases differ only in their
+    Args:
+        case (CIME.case.Case): The case containing the hist files to compare
+        suffix1 (str): The suffix that identifies the first batch of hist files
+        suffix2 (str): The suffix that identifies the second batch of hist files
+        ignore_fieldlist_diffs (bool): If True, then: If the two cases differ only in their
         field lists (i.e., all shared fields are bit-for-bit, but one case has some
         diagnostic fields that are missing from the other case), treat the two cases as
         identical.
 
-    returns (SUCCESS, comments, num_compared)
+    Returns:
+        Tuple bool, str, str: (files_match, output_filename, comment)
+        files_match is True if the files matched, False otherwise.
+        output_filename is the name of the cprnc output file or None if outfile_suffix is None.
+        comment is either an empty string or one of the module-level constants beginning with CPRNC (e.g., CPRNC_FIELDLISTS_DIFFER)
     """
     rundir = case.get_value("RUNDIR")
 
@@ -447,24 +455,27 @@ def cprnc(
     ignore_fieldlist_diffs=False,
     cprnc_exe=None,
 ):
-    """
-    Run cprnc to compare two individual nc files
-
-    file1 - the full or relative path of the first file
-    file2 - the full or relative path of the second file
-    case - the case containing the files
-    rundir - the rundir for the case
-    outfile_suffix - if non-blank, then the output file name ends with this
-        suffix (with a '.' added before the given suffix).
-        Use None to avoid permissions issues in the case dir.
-    ignore_fieldlist_diffs (bool): If True, then: If the two cases differ only in their
-        field lists (i.e., all shared fields are bit-for-bit, but one case has some
-        diagnostic fields that are missing from the other case), treat the two cases as
-        identical.
+    """Run the cprnc tool to compare two individual netcdf files.
+
+    Args:
+        file1 (str): The full or relative path of the first file.
+        file2 (str): The full or relative path of the second file.
+        case (CIME.case.Case): the case containing the files.
+        rundir (str): The rundir for the case
+        outfile_suffix (str): If non-blank, then the output file name ends with this
+        suffix (with a '.' added before the given suffix). Use None to avoid
+        permissions issues in the case dir.
+        ignore_fieldlist_diffs (bool): If True, then: If the two cases differ only in
+        their field lists (i.e., all shared fields are bit-for-bit, but one case has
+        some diagnostic fields that are missing from the other case), treat the two
+        cases as identical.
+
+    Returns:
+        Tuple bool, str, str: (files_match, output_filename, comment)
+        files_match is True if the files matched, False otherwise.
+        output_filename is the name of the cprnc output file or None if outfile_suffix is None.
+        comment is either an empty string or one of the module-level constants beginning with CPRNC (e.g., CPRNC_FIELDLISTS_DIFFER)
 
-    returns (True if the files matched, log_name, comment)
-        where 'comment' is either an empty string or one of the module-level constants
-        beginning with CPRNC_ (e.g., CPRNC_FIELDLISTS_DIFFER)
     """
     if not cprnc_exe:
         cprnc_exe = case.get_value("CCSM_CPRNC")
@@ -548,16 +559,15 @@ def cprnc(
 
 
 def compare_baseline(case, baseline_dir=None, outfile_suffix=""):
-    """
-    compare the current test output to a baseline result
+    """Compare the current test output to a baseline result
 
-    case - The case containing the hist files to be compared against baselines
-    baseline_dir - Optionally, specify a specific baseline dir, otherwise it will be computed from case config
-    outfile_suffix - if non-blank, then the cprnc output file name ends with
-        this suffix (with a '.' added before the given suffix). if None, no output file saved.
+    Args:
+        case - The case containing the hist files to be compared against baselines
+        baseline_dir - Optionally, specify a specific baseline dir, otherwise it will be computed from case config
+        outfile_suffix - if non-blank, then the cprnc output file name ends with this suffix (with a '.' added before the given suffix). if None, no output file saved.
 
-    returns (SUCCESS, comments)
-    SUCCESS means all hist files matched their corresponding baseline
+    Returns:
+        Tuple bool, str: (success, comments). Success means all hist files matched their corresponding baseline.
     """
     rundir = case.get_value("RUNDIR")
     if baseline_dir is None:

CIME/hist_utils.py

@@ -1,6 +1,7 @@
 """Module containing tools for dealing with Fortran namelists.
 
 The public interface consists of the following functions:
+
 - `character_literal_to_string`
 - `compress_literal_list`
 - `expand_literal_list`
@@ -62,6 +63,7 @@
 The Fortran standard only applies to the interior of namelist groups, and not to
 text between one namelist group and the next. This module assumes that namelist
 groups are separated by (optional) whitespace and comments, and nothing else.
+
 """
 
 ###############################################################################
@@ -840,18 +842,18 @@ def parse(in_file=None, text=None, groupless=False, convert_tab_to_space=True):
     The `groupless` argument changes namelist parsing in two ways:
 
     1. `parse` allows an alternate file format where no group names or slashes
-       are present. In effect, the file is parsed as if an invisible, arbitrary
-       group name was prepended, and an invisible slash was appended. However,
-       if any group names actually are present, the file is parsed normally.
+    are present. In effect, the file is parsed as if an invisible, arbitrary
+    group name was prepended, and an invisible slash was appended. However,
+    if any group names actually are present, the file is parsed normally.
     2. The return value of this function is not a `Namelist` object. Instead a
-       single, flattened dictionary of name-value pairs is returned.
+    single, flattened dictionary of name-value pairs is returned.
 
     The `convert_tab_to_space` option can be used to force all tabs in the file
     to be converted to spaces, and is on by default. Note that this will usually
     allow files that use tabs as whitespace to be parsed. However, the
     implementation of this option is crude; it converts *all* tabs in the file,
     including those in character literals. (Note that there are many characters
-    that cannot be passed in via namelist in any standard way, including '\n',
+    that cannot be passed in via namelist in any standard way, including '\\n',
     so it is already a bad idea to assume that the namelist will preserve
     whitespace in strings, aside from simple spaces.)
 
@@ -860,6 +862,7 @@ def parse(in_file=None, text=None, groupless=False, convert_tab_to_space=True):
     All names and values returned are ultimately unicode strings. E.g. a value
     of "6*2" is returned as that string; it is not converted to 6 copies of the
     Python integer `2`. Null values are returned as the empty string ("").
+
     """
     expect(
         in_file is not None or text is not None,