Merge pull request #318 from xylar/add-aurora

Add support for Aurora

Author: xylar

docs/developers_guide/api.md

@@ -224,7 +224,7 @@ seaice/api
 
    write_job_script
    get_slurm_options
-   clean_up_whitespace
+   get_pbs_options
 ```
 
 ### logging

docs/developers_guide/machines/index.md

@@ -79,6 +79,10 @@ These are the machines supported by MPAS-Ocean and -Seaice, including the
 +--------------+------------+-----------+-------------------+
 ```
 
+:::{note}
+MPAS components currently do not support Aurora in standalone builds.
+:::
+
 (dev-omega-supported-machines)=
 
 ### Omega Supported Machines
@@ -90,6 +94,8 @@ E3SM default for the given machine an compiler.
 +--------------+------------------+-----------+
 | Machine      | Compiler         | MPI lib.  |
 +==============+==================+===========+
+| aurora       | oneapi-ifx       | mpich     |
++--------------+------------------+-----------+
 | chicoma-cpu  | gnu              | mpich     |
 +--------------+------------------+-----------+
 | chrysalis    | intel            | openmpi   |
@@ -239,10 +245,10 @@ hostname_contains = morpheus
 ```
 
 The `[parallel]` section should describe the type of parallel queuing
-system (currently only `slurm` or `single_node` are supported), the number
+system (currently `slurm`, `pbs` or `single_node` are supported), the number
 of cores per node and the command for running an MPI executable (typically
-`srun` for Slurm and `mpirun` for a "single node" machine like a laptop or
-workstation.
+`srun` for Slurm and `mpirun` for a PBS or "single node" machine like a laptop
+or workstation.
 
 The `[spack]` section has some config options to do with loading system
 modules before or after loading a Spack environment.  On a "single node"

docs/tutorials/dev_add_test_group.md

@@ -669,7 +669,7 @@ and run it:
 ```bash
 $ cd ${PATH_TO_WORKING_DIR}/ocean/yet_another_channel/10km/default
 $ sbatch job_script.sh
-$ cat polaris.o${SLURM_JOBID}
+$ cat polaris.o${JOBID}
 
      Loading conda environment
      Done.

docs/users_guide/config_files.md

@@ -35,7 +35,7 @@ database_root = /home/xylar/data/polaris/data
 # The parallel section describes options related to running tests in parallel
 [parallel]
 
-# parallel system of execution: slurm or single_node
+# parallel system of execution: slurm, pbs or single_node
 system = single_node
 
 # whether to use mpirun or srun to run the model
@@ -47,7 +47,7 @@ cores_per_node = 8
 
 ```
 
-The comments in this example are hopefully pretty self-explanatory. 
+The comments in this example are hopefully pretty self-explanatory.
 You provide the config file to `polaris setup` and `polaris suite` with
 the `-f` flag:
 
@@ -81,12 +81,12 @@ sources:
   [extended interpolation](https://docs.python.org/3/library/configparser.html#configparser.ExtendedInterpolation)
   in the config file to use config options within other config
   options, e.g. `component = ${paths:component_path}/ocean_model`.
-- a config file shared with other similar tasks if one is defined.  For 
-  idealized tests, these often include the size and resolution of the mesh as 
+- a config file shared with other similar tasks if one is defined.  For
+  idealized tests, these often include the size and resolution of the mesh as
   well as (for ocean initial conditions) the number of vertical levels.
 - any number of config files from the task.  There might be different
   config options depending on how the task is configured (e.g. only if a
-  certain feature is enabled.  For example, {ref}`ocean-global-ocean` loads 
+  certain feature is enabled.  For example, {ref}`ocean-global-ocean` loads
   different sets of config options for different meshes.
 - a user's config file described above.
 
@@ -153,7 +153,7 @@ core_path = mpas-ocean
 # source: /home/xylar/code/polaris/customize_config_parser/polaris/default.cfg
 partition_executable = gpmetis
 
-# parallel system of execution: slurm or single_node
+# parallel system of execution: slurm, pbs or single_node
 # source: /home/xylar/code/polaris/customize_config_parser/inej.cfg
 system = single_node
 
@@ -467,6 +467,6 @@ min_layer_thickness = 3.0
 max_layer_thickness = 500.0
 ```
 
-The comments are retained and the config file or python module where they were 
-defined is also included as a a comment for provenance and to make it easier 
+The comments are retained and the config file or python module where they were
+defined is also included as a a comment for provenance and to make it easier
 for users and developers to understand how the config file is built up.

docs/users_guide/invalid_quick_start.md

@@ -276,7 +276,7 @@ database_root = </path/to/root>/polaris/data
 # The parallel section describes options related to running tasks in parallel
 [parallel]
 
-# parallel system of execution: slurm or single_node
+# parallel system of execution: slurm, pbs or single_node
 system = single_node
 
 # whether to use mpirun or srun to run the model

docs/users_guide/machines/anvil.md

@@ -30,6 +30,9 @@ polaris_envs = /lcrc/soft/climate/polaris/anvil/base
 # the compiler set to use for system libraries and MPAS builds
 compiler = intel
 
+# the compiler to use to build software (e.g. ESMF and MOAB) with spack
+software_compiler = intel
+
 # the system MPI library to use for intel compiler
 mpi_intel = impi
 
@@ -51,7 +54,7 @@ Additionally, some relevant config options come from the
 # The parallel section describes options related to running jobs in parallel
 [parallel]
 
-# parallel system of execution: slurm, cobalt or single_node
+# parallel system of execution: slurm, pbs or single_node
 system = slurm
 
 # whether to use mpirun or srun to run a task

docs/users_guide/machines/aurora.md

@@ -0,0 +1,97 @@
+# Aurora
+
+login: `ssh <username>@aurora.alcf.anl.gov`
+
+interactive login:
+
+```bash
+qsub -I -A E3SM_Dec -q debug -l select=1 -l walltime=00:30:00 -l filesystems=home:flare
+```
+
+Here is a link to the
+[Aurora User Guide](https://docs.alcf.anl.gov/aurora/)
+
+## config options
+
+Here are the default config options added when you have configured Polairs on
+a Aurora login node (or specified `./configure_polaris_envs.py -m aurora`):
+
+```cfg
+# The paths section describes paths for data and environments
+[paths]
+
+# A shared root directory where polaris data can be found
+database_root = /lus/flare/projects/E3SM_Dec/polaris
+
+# the path to the base conda environment where polars environments have
+# been created
+polaris_envs = /lus/flare/projects/E3SM_Dec/soft/polaris/aurora/base
+
+
+# Options related to deploying a polaris conda and spack environments
+[deploy]
+
+# the compiler set to use for system libraries and MPAS builds
+compiler = oneapi-ifx
+
+# the compiler to use to build software (e.g. ESMF and MOAB) with spack
+software_compiler = oneapi-ifx
+
+# the system MPI library to use for oneapi-ifx compiler
+mpi_oneapi_ifx = mpich
+
+# the base path for spack environments used by polaris
+spack = /lus/flare/projects/E3SM_Dec/soft/polaris/aurora/spack
+
+# whether to use the same modules for hdf5, netcdf-c, netcdf-fortran and
+# pnetcdf as E3SM (spack modules are used otherwise)
+use_e3sm_hdf5_netcdf = False
+
+
+# Config options related to creating a job script
+[job]
+
+# the filesystems used for the job
+filesystems = home:flare
+```
+
+Additionally, some relevant config options come from the
+[mache](https://github.com/E3SM-Project/mache/) package:
+
+```cfg
+# The parallel section describes options related to running jobs in parallel
+[parallel]
+
+# parallel system of execution: slurm, pbs or single_node
+system = pbs
+
+# whether to use mpirun or srun to run a task
+parallel_executable = mpirun
+
+# cores per node on the machine (with hyperthreading)
+cores_per_node = 208
+
+# account for running diagnostics jobs
+account = E3SM_Dec
+
+# queues (default is the first)
+queues = prod, debug
+
+# Config options related to spack environments
+[spack]
+
+# whether to load modules from the spack yaml file before loading the spack
+# environment
+modules_before = False
+
+# whether to load modules from the spack yaml file after loading the spack
+# environment
+modules_after = False
+```
+
+## Loading and running Polaris on Aurora
+
+Follow the developer's guide at {ref}`dev-machines` to get set up.  There are
+currently no plans to support a different deployment strategy (e.g. a shared
+environoment) for users.
+

docs/users_guide/machines/chicoma.md

@@ -83,16 +83,25 @@ polaris_envs = /usr/projects/e3sm/polaris/chicoma-cpu/conda/base
 # the compiler set to use for system libraries and MPAS builds
 compiler = gnu
 
+# the compiler to use to build software (e.g. ESMF and MOAB) with spack
+software_compiler = gnu
+
 # the system MPI library to use for gnu compiler
 mpi_gnu = mpich
 
+# the system MPI library to use for nvidia compiler
+mpi_nvidia = mpich
+
 # the base path for spack environments used by polaris
 spack = /usr/projects/e3sm/polaris/chicoma-cpu/spack
 
 # whether to use the same modules for hdf5, netcdf-c, netcdf-fortran and
 # pnetcdf as E3SM (spack modules are used otherwise)
 use_e3sm_hdf5_netcdf = True
 
+# location of a spack mirror for polaris to use
+spack_mirror = /usr/projects/e3sm/polaris/chicoma-cpu/spack/spack_mirror
+
 
 # The parallel section describes options related to running jobs in parallel
 [parallel]
@@ -107,6 +116,11 @@ cores_per_node = 128
 # hanging on perlmutter)
 threads_per_core = 1
 
+# quality of service
+# overriding mache because the debug qos also requires --reservaiton debug,
+# which polaris doesn't currently support
+qos = standard
+
 
 # Config options related to creating a job script
 [job]
@@ -125,7 +139,7 @@ Additionally, some relevant config options come from the
 # The parallel section describes options related to running jobs in parallel
 [parallel]
 
-# parallel system of execution: slurm, cobalt or single_node
+# parallel system of execution: slurm, pbs or single_node
 system = slurm
 
 # whether to use mpirun or srun to run a task

docs/users_guide/machines/chrysalis.md

@@ -26,6 +26,9 @@ polaris_envs = /lcrc/soft/climate/polaris/chrysalis/base
 # the compiler set to use for system libraries and MPAS builds
 compiler = intel
 
+# the compiler to use to build software (e.g. ESMF and MOAB) with spack
+software_compiler = intel
+
 # the system MPI library to use for intel compiler
 mpi_intel = openmpi
 
@@ -47,7 +50,7 @@ Additionally, some relevant config options come from the
 # The parallel section describes options related to running jobs in parallel
 [parallel]
 
-# parallel system of execution: slurm, cobalt or single_node
+# parallel system of execution: slurm, pbs or single_node
 system = slurm
 
 # whether to use mpirun or srun to run a task

docs/users_guide/machines/compy.md

@@ -17,7 +17,7 @@ database_root = /compyfs/polaris
 
 # the path to the base conda environment where polaris environments have
 # been created
-polaris_envs = /share/apps/E3SM/polaris/conda/base
+polaris_envs = /share/apps/E3SM/conda_envs/polaris/conda/base
 
 
 # Options related to deploying a polaris conda and spack environments
@@ -26,14 +26,17 @@ polaris_envs = /share/apps/E3SM/polaris/conda/base
 # the compiler set to use for system libraries and MPAS builds
 compiler = intel
 
+# the compiler to use to build software (e.g. ESMF and MOAB) with spack
+software_compiler = intel
+
 # the system MPI library to use for intel compiler
 mpi_intel = impi
 
 # the system MPI library to use for gnu compiler
 mpi_gnu = openmpi
 
 # the base path for spack environments used by polaris
-spack = /share/apps/E3SM/polaris/spack
+spack = /share/apps/E3SM/conda_envs/polaris/spack
 
 # whether to use the same modules for hdf5, netcdf-c, netcdf-fortran and
 # pnetcdf as E3SM (spack modules are used otherwise)
@@ -49,7 +52,7 @@ Additionally, some relevant config options come from the
 # The parallel section describes options related to running jobs in parallel
 [parallel]
 
-# parallel system of execution: slurm, cobalt or single_node
+# parallel system of execution: slurm, pbs or single_node
 system = slurm
 
 # whether to use mpirun or srun to run a task