Skip to content

Commit 074d2bb

Browse files
xylarxylar
committed
First draft of releasing documentation
1 parent 3c180a4 commit 074d2bb

File tree

10 files changed

+786
-2
lines changed

10 files changed

+786
-2
lines changed

‎docs/deployment.md‎

Whitespace-only changes.

‎docs/index.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,7 @@ This documentation is for both **users** of the E3SM-Unified environment and
2828

2929
* 💠 [Using E3SM-Unified Tools](using-tools.md)
3030
* 🧪 [Testing Release Candidates](testing-release-candidates.md)
31-
* 🦠 [Release Engineering](release-engineering.md)
32-
* 🚚 [Deployment on HPCs](deployment.md)
31+
* 🚚 [The E3SM-Unified Release Workflow](releasing/release-workflow.md)
3332
* 📦 [Package Catalog](packages.md)
3433
* ❓ [Troubleshooting & FAQs](troubleshooting.md)
3534
* 🤝 [Contributing & Community](contributing.md)

‎docs/release-engineering.md‎

Whitespace-only changes.

‎docs/releasing/conda-vs-spack.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# How Conda and Spack Work Together in E3SM-Unified
2+
3+
E3SM-Unified uses a hybrid approach that combines Conda and Spack to build
4+
and deploy a comprehensive software environment for E3SM analysis and
5+
diagnostics. This page explains the motivation for this strategy, how the
6+
components interact, and the shared infrastructure that supports both
7+
E3SM-Unified and related projects.
8+
9+
---
10+
11+
## Why Combine Conda and Spack?
12+
13+
Each tool solves a different part of the problem:
14+
15+
### ✅ Conda
16+
17+
* Excellent for managing Python packages and their dependencies
18+
* Supports rapid installation and reproducibility
19+
* Compatible with conda-forge and custom channels (e.g., `e3sm`)
20+
* User-friendly interface, especially for scientists and developers
21+
22+
### ✅ Spack
23+
24+
* Designed for building performance-sensitive HPC software
25+
* Allows fine-grained control over compilers, MPI implementations, and system
26+
libraries
27+
* Better suited for tools written in Fortran/C/C++ with MPI dependencies
28+
(e.g., NCO, MOAB, TempestRemap)
29+
30+
### ❗ The Challenge
31+
32+
Neither system alone is sufficient:
33+
34+
* Conda cannot reliably build or run MPI-based binaries across multiple nodes
35+
on HPC systems. In our experience, Conda's MPI implementations often fail
36+
even for multi-task jobs on a single node, making them unsuitable for
37+
high-performance parallel workflows
38+
* Spack lacks strong support for modern Python environments and is generally
39+
harder to use for scientists accustomed to Conda-based workflows. While
40+
conda-forge provides access to tens of thousands of Python packages, Spack
41+
offers far fewer, meaning many familiar scientific tools are not readily
42+
available through Spack alone
43+
44+
---
45+
46+
## Architecture: How They Work Together
47+
48+
E3SM-Unified environments:
49+
50+
1. Use **Conda** to install the core Python tools and lightweight dependencies
51+
2. Rely on **Spack** to build performance-critical tools outside Conda
52+
3. Are bundled into a single workflow that ensures compatibility across both
53+
54+
System-specific setup scripts (e.g., `load_latest_e3sm_unified_<machine>.sh`)
55+
ensure both components are activated correctly.
56+
57+
For MPI-based tools:
58+
59+
* The tools are built with Spack using system compilers and MPI
60+
* Users automatically access these builds when running on compute nodes
61+
62+
---
63+
64+
## Shared Infrastructure
65+
66+
E3SM-Unified, Polaris, and Compass all rely on the same key components:
67+
68+
* [`mache`](https://github.com/E3SM-Project/mache): A configuration library
69+
for detecting machine-specific settings (modules, compilers, paths)
70+
* [E3SM's Spack fork](https://github.com/E3SM-Project/spack): Centralized
71+
control over package versions and build settings
72+
* Conda: Used consistently to install `mache`, lightweight tools, and Python
73+
dependencies
74+
75+
This shared foundation ensures reproducibility and consistency across
76+
workflows, testbeds, and developer tools in the E3SM ecosystem.
77+
78+
---
79+
80+
## Future Alternatives
81+
82+
As complexity grows, other strategies may be worth evaluating:
83+
84+
### Option: **E4S (Extreme-scale Scientific Software Stack)**
85+
86+
* Spack-based stack of curated HPC tools
87+
* E4S environments aim to replace the need for manual Spack+Conda integration
88+
* May offer better long-term sustainability, but lacks Python focus today
89+
90+
🔗 [Explore E4S](https://e4s.io)
91+
92+
### Other Approaches (less suitable currently):
93+
94+
* Pure Spack builds (harder for Python workflows)
95+
* Pure Conda builds (harder for HPC performance tools)
96+
* Containers (portability gains, but complex for HPC integration)
97+
98+
---
99+
100+
## Summary
101+
102+
The hybrid Conda + Spack model in E3SM-Unified balances ease of use with HPC
103+
performance. While more complex to maintain, it provides flexibility,
104+
compatibility, and performance across diverse systems. Shared infrastructure
105+
(like `mache` and E3SM's Spack fork) reduces duplication across projects and
106+
streamlines the release process.

‎docs/releasing/creating-rcs/overview.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Creating Release Candidates
2+
3+
E3SM-Unified and its core dependencies follow a structured release process
4+
that relies on **release candidates (RCs)**. These pre-release versions are
5+
used for testing and validation before an official release is finalized and
6+
deployed.
7+
8+
This section describes how to create RCs for both individual dependencies
9+
(like `e3sm_diags` or `mpas-analysis`) and for the `e3sm-unified` metapackage
10+
itself. It also includes tools and tips for troubleshooting build failures.
11+
12+
---
13+
14+
## What Is a Release Candidate?
15+
16+
A release candidate (RC) is a build intended for testing before a final
17+
release. RCs allow us to validate compatibility across the E3SM analysis stack
18+
and to ensure that tools and environments function correctly on supported HPC
19+
platforms.
20+
21+
RC packages are published to special Conda labels (like `e3sm_diags_dev` or
22+
`e3sm_unified_dev`) to keep them separate from stable releases.
23+
24+
---
25+
26+
## Overview of the Process
27+
28+
There are two major workflows:
29+
30+
### 1. Creating RCs for Dependency Packages
31+
32+
These are individual tools like `e3sm_diags`, `zppy`, or `mpas_analysis` that
33+
are used within the E3SM-Unified environment.
34+
35+
Go to: [Creating RCs for Dependency Packages](rc-dependencies.md)
36+
37+
---
38+
39+
### 2. Creating an RC for E3SM-Unified
40+
41+
This involves assembling a full test environment based on specific versions of
42+
all dependencies — including RCs.
43+
44+
Go to: [Creating an E3SM-Unified RC](rc-e3sm-unified.md)
45+
46+
---
47+
48+
### 3. Troubleshooting Build Failures
49+
50+
Solving Conda environments during builds can fail for complex or subtle
51+
reasons. This section provides detailed strategies for debugging, including
52+
use of `conda_first_failure.py`.
53+
54+
Go to: [Troubleshooting Conda Build Failures](rc-troubleshooting.md)
55+
56+
---
57+
58+
Each page includes step-by-step examples, commands, and best practices
59+
tailored to the E3SM-Unified release workflow.

‎docs/releasing/creating-rcs/rc-dependencies.md‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# Creating RCs for Dependency Packages
2+
3+
This page describes how to create release candidates (RCs) for packages that
4+
are included in the E3SM-Unified environment, such as `e3sm_diags`,
5+
`mpas-analysis`, `zppy`, and `zstash`.
6+
7+
We use `e3sm_diags` as a concrete example, but the process is similar for all
8+
E3SM-developed dependencies.
9+
10+
---
11+
12+
## Step-by-Step: Creating an RC for `e3sm_diags`
13+
14+
### 1. Tag a Release Candidate in the Source Repo
15+
16+
Go to the source repository:
17+
[E3SM Diags GitHub](https://github.com/E3SM-Project/e3sm_diags)
18+
19+
Create a release tag:
20+
21+
```bash
22+
git checkout main
23+
git fetch --all -p
24+
git reset --hard origin/main
25+
git tag v3.0.0rc1
26+
git push origin v3.0.0rc1
27+
```
28+
29+
**Note:**
30+
31+
* `e3sm_diags` uses a `v` prefix in version tags (e.g., `v3.0.0rc1`) as part
32+
of its established convention.
33+
* For new packages, it’s recommended to follow
34+
[Semantic Versioning](https://semver.org/) and omit the `v` prefix (i.e.,
35+
tag as `3.0.0rc1`).
36+
37+
---
38+
39+
### 2. Prepare the Feedstock PR
40+
41+
Go to the conda-forge feedstock for `e3sm_diags`:
42+
[E3SM Diags Feedstock](https://github.com/conda-forge/e3sm_diags-feedstock)
43+
44+
If a `dev` branch does not already exist:
45+
46+
* Clone the feedstock repo locally
47+
* Create a new branch off `main` called `dev`
48+
* Push it to the origin
49+
50+
**Note:** By making no changes from the `main` branch, you ensure that no
51+
new packages will be created when you push the `dev` branch to the origin
52+
53+
### 3. Fork the Feedstock and Create a PR
54+
55+
1. Fork the feedstock repo to your GitHub account.
56+
2. In your fork, create a new branch (e.g., `update-v3.0.0rc1`).
57+
58+
**Important:** Do **not** create branches directly on the main conda-forge
59+
feedstock. All changes should go through a pull request from your personal
60+
fork. Creating a branch on the main feedstock can trigger package builds
61+
before your updates have been properly tested or reviewed. (e.g.,
62+
`update-v3.0.0rc1`).
63+
64+
3. Edit `recipe/meta.yaml`:
65+
66+
* Update the `version` field to match your RC tag (e.g., `v3.0.0rc1`)
67+
* Update dependencies if needed (e.g., pin to RC versions of other tools)
68+
69+
4. If you created the `dev` branch above and no previous release candidates
70+
have been added, you will need to add `recipe/conda_build_config.yaml` with
71+
contents like:
72+
73+
``` yaml
74+
channel_targets:
75+
- conda-forge e3sm_diags_dev
76+
```
77+
78+
The label is the name of the package with any `-` replaced by `_`, followed
79+
by `_dev`.
80+
81+
5. Commit the changes and push them to the branch on your fork (unless editing
82+
on GitHub directly).
83+
84+
6. Open a pull request:
85+
86+
* **Source:** your fork’s RC branch
87+
* **Target:** the `dev` branch on the conda-forge feedstock
88+
89+
---
90+
91+
### 4. Merge the PR Once CI Passes
92+
93+
After CI completes successfully:
94+
95+
* Review the logs if needed
96+
* Merge the PR into the `dev` branch
97+
98+
The RC build will now be published to:
99+
100+
```
101+
conda-forge/label/e3sm_diags_dev
102+
```
103+
104+
You can test the RC by installing it like so:
105+
106+
```bash
107+
conda install -c conda-forge/label/e3sm_diags_dev e3sm_diags
108+
```
109+
110+
---
111+
112+
## Summary
113+
114+
Creating an RC for a dependency involves:
115+
116+
1. Tagging the source repository
117+
2. Opening a PR on the feedstock targeting the `dev` branch
118+
3. Waiting for CI to pass, then merging
119+
120+
This process enables E3SM-Unified maintainers to incorporate the RC version of
121+
your package into a unified test build.
122+
123+
➡ Next: [Creating an RC for E3SM-Unified](rc-e3sm-unified.md)

0 commit comments

Comments
 (0)