Merge pull request #136 from xylar/fix-docs

xylar · web-flow · commit 636443fca83c · 2025-08-10T19:02:40.000+02:00
Yet more updates and fixes to the docs
diff --git a/docs/releasing/creating-rcs/rc-dependencies.md b/docs/releasing/creating-rcs/rc-dependencies.md
@@ -11,7 +11,15 @@ E3SM-developed dependencies.
 
 ## Step-by-Step: Creating an RC for `e3sm_diags`
 
-### 1. Tag a Release Candidate in the Source Repo
+### 1. Tag a Release Candidate or Make a GitHub Release
+
+#### 1.1 Tag a Release Candidate in the Source Repo
+
+In some repos like MPAS-Analysis, it is desirable to tag a release candidate
+without making a GitHub release page.  This avoids clutter and confusion.
+
+We will use E3SM Diags in this example even though this is not the preferred
+workflow for that repository.
 
 Go to the source repository:
 [E3SM Diags GitHub](https://github.com/E3SM-Project/e3sm_diags)
@@ -36,6 +44,57 @@ git push origin v3.0.0rc1
 
 ---
 
+#### 1.2 Perform at GitHub Release
+
+In other repositories like E3SM Diags and zppy, developers prefer that you
+create release pages for release candidates.  These pages provide clarity and
+provenance for the release candidate just like for a normal release.
+
+Follow these steps to create a release candidate via GitHub:
+
+- **Pull the latest changes on `main`:**
+  ```bash
+  git checkout main
+  git pull origin main
+  ```
+
+- **Create a new branch from `main`:**
+  ```bash
+  git checkout -b bump/0.1.0rc1
+  ```
+
+- **Push the branch to your fork or upstream:**
+  ```bash
+  git push --set-upstream origin bump/0.1.0rc1
+  ```
+
+- **Bump the version in the Python files:**
+  - You can use the `tbump` tool (available in the conda development
+    environments for `e3sm_diags` and `e3sm_to_cmip`):
+    ```bash
+    tbump 0.1.0rc1 --no-tag
+    ```
+    This will automatically update version strings, add, commit, and push
+    changes to remote.
+  - Alternatively, manually update version strings in files such as
+    `pyproject.toml`, `setup.py`, and `<python_package>/__init__.py`. Then
+    add, commit, and push these changes.
+
+- **Open a pull request:**
+  - Use your `bump/0.1.0rc1` branch as the source and merge into `main`.
+  - Example: [Compare · E3SM-Project/e3sm_to_cmip](https://github.com/E3SM-Project/e3sm_to_cmip/compare)
+
+- **Publish a new GitHub Release:**
+  - Go to the [GitHub Releases page](https://github.com/E3SM-Project/e3sm_to_cmip/releases/new).
+  - Click "Choose a tag" and enter `0.1.0rc1` (ideally without a `v`, but
+    follow the repo’s conventions).
+  - For "Release title", use `v0.1.0rc1` (with a `v`).
+  - Click "Generate release notes" and organize the changelog as needed.
+  - Check the box "Set as a pre-release".
+  - Click "Publish release".
+
+---
+
 ### 2. Prepare the Feedstock PR
 
 Go to the conda-forge feedstock for `e3sm_diags`:
@@ -117,7 +176,7 @@ conda install -c conda-forge/label/e3sm_diags_dev e3sm_diags
 
 Creating an RC for a dependency involves:
 
-1. Tagging the source repository
+1. Tagging a Release Candidate or making a GitHub release
 2. Opening a PR on the feedstock targeting the `dev` branch
 3. Waiting for CI to pass, then merging
 
diff --git a/docs/releasing/creating-rcs/rc-e3sm-unified.md b/docs/releasing/creating-rcs/rc-e3sm-unified.md
@@ -35,63 +35,30 @@ Edit `recipes/e3sm-unified/meta.yaml`:
   `e3sm_diags`, `mpas-analysis`, etc.)
 
 ---
+## 3. Run `build_packages.py`
 
-## 3. Regenerate the Build Matrix
+If you are building packages for an RC version of E3SM-Unified, the script
+will build packages for Python 3.10 and the `nompi` and `hpc` versions of MPI
+by default (those needed for login and compute nodes, respectively, on HPC).
+You may wish to use the `--python` and `--mpi` flags to override these.
 
-Run the matrix generator script to define combinations of Python and MPI:
+You should supply the `--conda` flag if not using `~/miniforge3`.
 
 ```bash
-cd recipes/e3sm-unified/configs
-rm *.yaml
-python generate.py
+./build_packages.py --conda ~/minforge3 --python 3.10 --mpi nompi hpc
 ```
 
-This produces matrix files like:
+The script will produce a matrix of files like:
 
-* `mpi_mpich_python3.10.yaml`
-* `mpi_hpc_python3.10.yaml`
+* `configs/mpi_nompi_python3.10.yaml`
+* `configs/mpi_hpc_python3.10.yaml`
 
----
-
-## 4. Edit `build_package.bash`
-
-Update the channel list to include dev labels for any packages still in RC
-form. For example:
-
-```bash
-channels="-c conda-forge/label/chemdyg_dev \
-         -c conda-forge/label/e3sm_diags_dev \
-         -c conda-forge/label/mache_dev \
-         -c conda-forge/label/mpas_analysis_dev \
-         -c conda-forge/label/zppy_dev \
-         -c conda-forge/label/zstash_dev \
-         -c conda-forge"
-```
-
-Then define which matrix files to test. For example:
-
-```bash
-for file in configs/mpi_mpich_python3.10.yaml configs/mpi_hpc_python3.10.yaml
-do
-  conda build -m $file --override-channels $channels .
-done
-```
-
-Make sure:
-
-* You use `--override-channels` to isolate testing to dev packages
-* You only include dev labels for packages with RCs — use stable versions
-  otherwise
+These are then used by `conda build` to build the appropriate variant of
+the conda package.
 
 ---
 
-## 5. Build and Troubleshoot
-
-Run the script:
-
-```bash
-bash build_package.bash
-```
+## 4. Troubleshoot
 
 If builds fail, consult the
 [Troubleshooting Conda Build Failures](rc-troubleshooting.md) guide.
@@ -100,7 +67,7 @@ resolution issues.
 
 ---
 
-## 6. Make a draft PR
+## 5. Make a draft PR
 
 Push the branch to your fork of `e3sm-unified` and make a draft PR to the
 main `e3sm-unified` repo.  Use that PR to document progress and highlight
@@ -110,7 +77,7 @@ acces to E3SM's Confluence pages). See
 
 ---
 
-## 7. Keeping updated on Confluence
+## 6. Keeping updated on Confluence
 
 As deployment and testing progresses, you needs to make sure that the packages
 in your `update-to-<version>` branch match the
@@ -127,7 +94,7 @@ jargon**) what is new in their package that would be of interest to users.
 
 ---
 
-## 8. Tag and Publish the RC
+## 7. Tag and Publish the RC
 
 After test builds are successful:
 
diff --git a/docs/releasing/planning-updates.md b/docs/releasing/planning-updates.md
@@ -73,6 +73,25 @@ using the [Creating Release Candidates](creating-rcs/overview.md) workflow.
 
 ---
 
+## ✍️ Updating `meta.yaml`
+
+Once the package list and versions are finalized, update the
+[`recipes/e3sm-unified/meta.yaml`](https://github.com/E3SM-Project/e3sm-unified/blob/main/recipes/e3sm-unified/meta.yaml)
+file to reflect the new selections. This file is the authoritative source for
+the conda environment and must be kept in sync with the planned versions.
+
+- Edit the `meta.yaml` to add new packages, update versions, or remove
+  deprecated packages as needed.
+- Ensure that all changes are consistent with the decisions made during
+  planning.
+- Commit these changes to your release branch as part of preparing the
+  release candidate.
+
+> **Tip:** Also update `e3sm_supported_machines/default.cfg` to match, unless
+  there is a technical reason to diverge.
+
+---
+
 ## 📦 Managing Version Pins During Conda-Forge Migrations
 
 E3SM-Unified often needs to coordinate with conda-forge's centralized version
diff --git a/docs/releasing/release-workflow.md b/docs/releasing/release-workflow.md
@@ -2,20 +2,19 @@
 
 The release process typically follows this progression:
 
-1. **[How Conda and Spack Work Together in E3SM-Unified](conda-vs-spack.md)**
-2. **[Planning Package Updates](planning-updates.md)**
-3. **[Creating Release Candidates](creating-rcs/overview.md)**
-4. **[Deployment and Testing](testing/overview.md)**
-5. **[Adding a New Machine](adding-new-machines.md)**
-6. **[Finalizing the Release](finalizing-release.md)**
-7. **[Maintaining Past Versionse](maintaining-past-versions.md)**
+1. **[Planning Package Updates](planning-updates.md)**
+2. **[Creating Release Candidates](creating-rcs/overview.md)**
+3. **[Deployment and Testing](testing/overview.md)**
+4. **[Adding a New Machine](adding-new-machines.md)**
+5. **[Finalizing the Release](finalizing-release.md)**
+6. **[Maintaining Past Versionse](maintaining-past-versions.md)**
 
-Each of these steps is detailed in its own page. See below for a high-level
-summary.
+We begin with some background information, then each of these steps is detailed
+in its own page. See below for a high-level summary.
 
 ---
 
-## 1. How Conda and Spack Work Together in E3SM-Unified
+## Backgraound: How Conda and Spack Work Together in Polaris
 
 Why does E3SM-Unified use both Conda and Spack? What roles do they each serve?
 Before you start, it's critical to understand how these two systems work
@@ -25,7 +24,7 @@ together.
 
 ---
 
-## 2. Planning Package Updates
+## 1. Planning Package Updates
 
 Updates are driven by the needs of the E3SM community, typically discussed via
 Confluence or GitHub. This step documents how to propose new packages or
@@ -35,7 +34,7 @@ changes to existing ones.
 
 ---
 
-## 3. Creating Release Candidates
+## 2. Creating Release Candidates
 
 This step covers:
 
@@ -46,13 +45,13 @@ This step covers:
 
 ---
 
-## 4. Deploying and Testing on HPCs
+## 3. Deploying and Testing on HPCs
 
 Before full deployment, release candidates are installed on a subset of HPC
 platforms for iterative testing and validation. This stage often requires
-extensive coordination and may involve debugging and extending the Spack build
-workflow, updating the E3SM Spack fork, and modifying `mache` to support new
-systems or changes in machine configurations.
+updating updating `mache` to support new systems or changes in machine
+configurations, adding package versions to E3SM's Spack fork, and
+troubleshooting deployment scripts.
 
 Testing includes everything from basic imports to full `zppy` workflows. This
 is a collaborative effort, with the full iterative process often spanning
@@ -62,7 +61,7 @@ several weeks to a few months.
 
 ---
 
-## 5. Adding a New Machine
+## 4. Adding a New Machine
 
 Most of the work for adding a new machine takes place in `mache`. Here we
 provide notes on adding new HPCs that are specific to E3SM-Unified.
@@ -71,7 +70,7 @@ provide notes on adding new HPCs that are specific to E3SM-Unified.
 
 ---
 
-## 6. Finalizing the Release
+## 5. Finalizing the Release
 
 Once all RCs pass testing:
 
@@ -84,7 +83,7 @@ Once all RCs pass testing:
 
 ---
 
-## 7. Maintaining Past Versions
+## 6. Maintaining Past Versions
 
 Older versions of E3SM-Unified sometimes require maintenance (repairs or
 deletion).
diff --git a/docs/releasing/testing/deploying-on-hpcs.md b/docs/releasing/testing/deploying-on-hpcs.md
@@ -46,8 +46,9 @@ Other flags are optional and will be discussed below.
   versions of some conda packages required in the installation environment
   (notably `mache`)
 * Version numbers here should match `meta.yaml` unless diverging for a reason
-* A special case is `esmpy = None`, required so ESMPy comes from conda-forge,
-  not Spack.
+* Special case are `esmpy = None` and `xesmf = None`, required so ESMPy and
+  xESMF come from conda-forge, not Spack (for compatibility with xCDAT and
+  E3SM Diags).
 
 ### ⚙️ `shared.py`
 
@@ -125,7 +126,7 @@ used during deployment.
    connection and you should pipe the output to a log file, e.g.:
 
    ```bash
-   ./deploy_e3sm_unified.py --conda ~/miniforge3 | tee deploy.log
+   ./deploy_e3sm_unified.py --conda ~/miniforge3 2>&1 | tee deploy.log
    ```
 
    **Note:** It is not recommended that you try to deploy E3SM-Unified
diff --git a/docs/releasing/testing/troubleshooting-deploy.md b/docs/releasing/testing/troubleshooting-deploy.md
@@ -140,7 +140,7 @@ build.
 ## 📎 Related Tools & Tips
 
 * Use `screen`, `tmux`, or `nohup` for long deployments
-* Always log output: `... | tee deploy.log`
+* Always log output: `... 2>&1 | tee deploy.log`
 * Validate final symlinks and paths manually after deployment
 * Document which system + compiler + MPI variants have been tested
 
