Merge pull request #303 from xylar/customize-generated-scripts

Use shell scripts generated from `config_machines.xml`

Author: xylar

docs/developers_guide/spack.md

@@ -14,8 +14,11 @@ To enable Spack-based environments for a new machine, you will need to:
    supported compiler and MPI library combination.
 2. Add system-provided (external) packages, saving time and preventing build
    failures if Spack attempts to build them from source.
-3. Optionally, provide shell script templates for module/environment setup if
-   needed.
+3. Prefer automatic shell script generation. Shell snippets used to be
+   maintained as templates in `mache.spack.templates`, but are now derived
+   primarily from the E3SM CIME machine configuration
+   (`mache/cime_machine_config/config_machines.xml`). Only add a minimal
+   template override when strictly necessary (see below).
 
 ## YAML Template Files
 
@@ -82,21 +85,46 @@ all:
     lapack: [[email protected]]
 ```
 
-## Shell Script Templates
+## Automatic shell script generation (preferred)
 
-If the machine requires special module loads or environment variables not handled by Spack, add corresponding shell script templates:
+When downstream packages call
+{py:func}`mache.spack.get_spack_script`, the module loads and
+environment-variable setup are constructed as follows:
 
-- `<machine>.sh` and/or `<machine>_<compiler>_<mpilib>.sh` for bash
-- `<machine>.csh` and/or `<machine>_<compiler>_<mpilib>.csh` for csh/tcsh
+1. Optional: activate Spack and the requested environment.
+2. Auto-generate a shell snippet from the E3SM CIME machine configuration
+   stored in `mache/cime_machine_config/config_machines.xml`, filtered for the
+   requested `(machine, compiler, mpilib)` and rendered for the target shell
+   (`sh` or `csh`).
+3. Append any Jinja2 template override found in
+   `mache/spack/templates/` named either `<machine>.<sh|csh>` or
+   `<machine>_<compiler>_<mpilib>.<sh|csh>`.
 
-These scripts are also Jinja2 templates and can use the same conditional logic as the YAML files.
+This pipeline greatly reduces maintenance and prevents drift between Mache and
+E3SM’s authoritative machine configuration. In most cases, you do not need to
+author or maintain shell script templates in Mache.
+
+### When to add a template override
+
+Only provide a small override in `mache/spack/templates/` if you need to:
+
+- Apply an adjustment that’s not appropriate for the shared E3SM CIME config
+  (machine-local quirk, temporary workaround, etc.).
+- Add conditional behavior toggled by
+  `include_e3sm_lapack` or `include_e3sm_hdf5_netcdf` (both exposed as Jinja
+  booleans in templates) that cannot be expressed in the CIME config.
+
+Templates are Jinja2 files and can use the same conditional logic as YAML
+templates.
 
 ## Testing
 
-After adding or modifying YAML and shell script templates:
+After adding or modifying YAML templates (or an exceptional shell override):
 
-1. Use the `make_spack_env` or `get_spack_script` functions in `mache.spack` to generate and test the environment.
-2. Confirm that all modules load and all external packages are correctly detected by Spack.
+1. Use `make_spack_env` or `get_spack_script` in `mache.spack` to generate and
+  test the environment and load scripts.
+2. Confirm that the generated shell snippet includes the expected module loads
+  from the CIME machine config and that Spack detects all external packages.
 3. Build and run a simple test application to verify the environment.
 
 ## Further Reading

docs/users_guide/spack/build.md

@@ -94,7 +94,8 @@ from mache.spack import get_spack_script
 ```
 
 **Purpose:**
-Generates a shell script snippet to activate a Spack environment and load any required modules or environment variables.
+Generates a shell script snippet to activate a Spack environment and load the
+required modules or environment variables.
 
 **Typical usage in downstream packages:**
 
@@ -113,20 +114,24 @@ spack_script = get_spack_script(
     mpi=mpi,
     shell='sh',  # or 'csh'
     machine=machine,
-    config_file=machine_config,
     include_e3sm_lapack=include_e3sm_lapack,
     include_e3sm_hdf5_netcdf=e3sm_hdf5_netcdf,
-    yaml_template=yaml_template
 )
 ```
 
-**Returns:**
-A string containing shell commands to:
+**How the script is composed:**
+
+The returned snippet is assembled in three steps:
 
-- Load required modules (if any).
-- Source the Spack setup script.
-- Activate the specified Spack environment.
-- Set any additional environment variables.
+1. Optionally source Spack and activate the requested environment.
+2. Auto-generate module loads and environment exports from the E3SM CIME
+  machine configuration (`mache/cime_machine_config/config_machines.xml`) for
+  the given `(machine, compiler, mpi)` and target shell (`sh` or `csh`).
+3. Append any Mache template override present in `mache/spack/templates/` named
+  `<machine>.<sh|csh>` or `<machine>_<compiler>_<mpi>.<sh|csh>`.
+
+This design keeps Mache aligned with E3SM’s authoritative machine
+configuration and minimizes maintenance.
 
 **Usage in activation scripts:**
 
@@ -144,7 +149,8 @@ from mache.spack import get_modules_env_vars_and_mpi_compilers
 ```
 
 **Purpose:**
-Returns the MPI compiler wrappers and a shell snippet to load modules and set environment variables for a given machine, compiler, and MPI library.
+Returns the MPI compiler wrappers and a shell snippet to load modules and set
+environment variables for a given machine, compiler, and MPI library.
 
 **Typical usage in downstream packages:**
 
@@ -161,7 +167,6 @@ mpicc, mpicxx, mpifc, mod_env_commands = get_modules_env_vars_and_mpi_compilers(
     shell='sh',  # or 'csh'
     include_e3sm_lapack=include_e3sm_lapack,
     include_e3sm_hdf5_netcdf=e3sm_hdf5_netcdf,
-    yaml_template=yaml_template
 )
 ```
 
@@ -172,13 +177,18 @@ mpicc, mpicxx, mpifc, mod_env_commands = get_modules_env_vars_and_mpi_compilers(
 - `mpifc`: Name of the MPI Fortran compiler wrapper (e.g., `mpif90` or `ftn`).
 - `mod_env_commands`: Shell commands to load modules and set environment variables.
 
-**Usage in build scripts:**
+**Notes and usage in build scripts:**
 
 ```bash
 {{ mod_env_commands }}
 # Now safe to use $mpicc, $mpicxx, $mpifc for building MPI-dependent software
 ```
 
+- This helper produces a minimal snippet based on Mache’s template overrides
+  (when present). It does not auto-generate content from
+  `config_machines.xml`. If you need the full environment setup used by
+  downstream activation scripts, prefer `get_spack_script`.
+
 ---
 
 ## Example: How Downstream Packages Use These Functions

mache/machines/anvil.cfg

@@ -63,18 +63,6 @@ partitions = acme-small, acme-medium, acme-large
 qos = regular, acme_high
 
 
-# Config options related to spack environments
-[spack]
-
-# whether to load modules from the spack yaml file before loading the spack
-# environment
-modules_before = False
-
-# whether to load modules from the spack yaml file after loading the spack
-# environment
-modules_after = False
-
-
 # config options related to synchronizing files
 [sync]
 

mache/machines/aurora.cfg

@@ -49,18 +49,6 @@ account = E3SM_Dec
 # queues (default is the first)
 queues = prod, debug
 
-# Config options related to spack environments
-[spack]
-
-# whether to load modules from the spack yaml file before loading the spack
-# environment
-modules_before = False
-
-# whether to load modules from the spack yaml file after loading the spack
-# environment
-modules_after = False
-
-
 # config options related to synchronizing files
 [sync]
 

mache/machines/chicoma-cpu.cfg

@@ -40,18 +40,6 @@ partitions = standard, gpu
 qos = standard, debug
 
 
-# Config options related to spack environments
-[spack]
-
-# whether to load modules from the spack yaml file before loading the spack
-# environment
-modules_before = False
-
-# whether to load modules from the spack yaml file after loading the spack
-# environment
-modules_after = False
-
-
 # config options related to synchronizing files
 [sync]
 

mache/machines/chrysalis.cfg

@@ -56,19 +56,6 @@ cores_per_node = 128
 # available partition(s) (default is the first)
 partitions = compute, debug, Shigh
 
-
-# Config options related to spack environments
-[spack]
-
-# whether to load modules from the spack yaml file before loading the spack
-# environment
-modules_before = False
-
-# whether to load modules from the spack yaml file after loading the spack
-# environment
-modules_after = True
-
-
 # config options related to synchronizing files
 [sync]
 

mache/machines/compy.cfg

@@ -63,18 +63,6 @@ partitions = slurm
 qos = regular
 
 
-# Config options related to spack environments
-[spack]
-
-# whether to load modules from the spack yaml file before loading the spack
-# environment
-modules_before = True
-
-# whether to load modules from the spack yaml file after loading the spack
-# environment
-modules_after = False
-
-
 # config options related to synchronizing files
 [sync]
 

mache/machines/dane.cfg

@@ -53,18 +53,6 @@ cores_per_node = 112
 partitions = pbatch, pdebug
 
 
-# Config options related to spack environments
-[spack]
-
-# whether to load modules from the spack yaml file before loading the spack
-# environment
-modules_before = True
-
-# whether to load modules from the spack yaml file after loading the spack
-# environment
-modules_after = False
-
-
 # config options related to synchronizing files
 [sync]
 

mache/machines/frontier.cfg

@@ -56,14 +56,6 @@ partitions = batch
 # Config options related to spack environments
 [spack]
 
-# whether to load modules from the spack yaml file before loading the spack
-# environment
-modules_before = False
-
-# whether to load modules from the spack yaml file after loading the spack
-# environment
-modules_after = False
-
 # whether the machine uses cray compilers
 cray_compilers = True
 

mache/machines/pm-cpu.cfg

@@ -62,17 +62,10 @@ constraints = cpu
 # quality of service (default is the first)
 qos = regular, debug, premium
 
+
 # Config options related to spack environments
 [spack]
 
-# whether to load modules from the spack yaml file before loading the spack
-# environment
-modules_before = False
-
-# whether to load modules from the spack yaml file after loading the spack
-# environment
-modules_after = False
-
 # whether the machine uses cray compilers
 cray_compilers = True