E3SM-Project · xylar · Jul 18, 2025 · Jul 14, 2025 · Jul 14, 2025 · Jul 14, 2025
diff --git a/dev-speck.txt → dev-spec.txt b/dev-speck.txt → dev-spec.txt
diff --git a/docs/contributing.md b/docs/contributing.md
@@ -0,0 +1,55 @@
+# Contributing & Community
+
+We welcome contributions and feedback from all users and developers of
+E3SM-Unified. Whether you're updating packages, improving documentation, or
+reporting issues, your input helps strengthen the environment and its
+community.
+
+---
+
+## Ways to Contribute
+
+### ✏️ Documentation
+
+* Suggest improvements to the user guide or technical docs.
+* Fix typos or clarify instructions.
+* Add usage examples for tools you use regularly.
+
+### 🚀 Suggest or Update Packages
+
+* Request new tools or features by opening a GitHub Issue.
+* Propose version updates by:
+  * Editing the E3SM Confluence pages defining the next E3SM-Unified version
+    (if you have access)
+  * Or editing the `meta.yaml` (for conda package) or `defaults.cfg` (for spack
+    pacakges) and making a pull request (if you don't have access to E3SM's
+    Confluence pages).
+
+### ⚙️ Development & Testing
+
+* Help test release candidates on supported platforms.
+* Report issues you encounter.
+* Contribute improvements to tools in the E3SM ecosystem (e.g., `mache`,
+  `mpas-analysis`, `zppy`, `e3sm_diags`).
+
+---
+
+## Getting Started
+
+1. Fork the [e3sm-unified GitHub repository](https://github.com/E3SM-Project/e3sm-unified).
+2. Create a new branch for your changes.
+3. Submit a pull request (PR).
+4. Tag reviewers as needed (e.g., `@xylar`).
+
+We recommend following our naming conventions for release branches (e.g.,
+`update-to-1.12.0`).
+
+---
+
+## Communication
+
+* GitHub Issues: [E3SM-Unified GitHub](https://github.com/E3SM-Project/e3sm-unified/issues)
+* Slack: `#e3sm-help-postproc`
+* Email: [[email protected]](mailto:[email protected])
+
+Have questions about where to start? Just ask on Slack or open an issue!
diff --git a/docs/index.md b/docs/index.md
@@ -1,11 +1,49 @@
+# Welcome to E3SM-Unified Documentation
+
 ```{image} logo/e3sm_unified_logo_200.png
 :align: center
 :width: 200 px
 ```
 
-# E3SM-Unfied
+E3SM-Unified is a unified conda + Spack environment developed to support pre-
+and post-processing workflows for the Energy Exascale Earth System Model
+(E3SM). It bundles commonly used analysis, visualization, and workflow tools
+into a single portable environment.
+
+This documentation is for both **users** of the E3SM-Unified environment and
+**developers** who build, test, and deploy it on supported platforms.
+
+---
+
+## 🚀 Start Here
+
+* 📖 [Introduction](introduction.md): What E3SM-Unified is and why you should
+  use it
+* 🧪 [Quickstart Guide](quickstart.md): Load the environment and start using
+  tools
+
+---
+
+## 📚 Contents
+
+* 💠 [Using E3SM-Unified Tools](using-tools.md)
+* 🧪 [Testing Release Candidates](testing-release-candidates.md)
+* 🚚 [The E3SM-Unified Release Workflow](releasing/release-workflow.md)
+* 📦 [Package Catalog](packages.md)
+* ❓ [Troubleshooting & FAQs](troubleshooting.md)
+* 🤝 [Contributing & Community](contributing.md)
+
+---
+
+## 💬 Get Help
+
+```{admonition} Support
+- Slack: #e3sm-help-postproc
+- GitHub Issues: [E3SM-Unified on GitHub](https://github.com/E3SM-Project/e3sm-unified/issues)
+- Maintainer contact: [email protected]
+```
+
+---
 
-A metapackage for a unified
-[conda environment](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)
-for analysis and other pre- and post-processing for the Energy Exascale Earth
-System Model (E3SM).
+This documentation is maintained by the E3SM Infrastructure Team and
+contributors across the community.
diff --git a/docs/introduction.md b/docs/introduction.md
@@ -0,0 +1,55 @@
+# Introduction to E3SM-Unified
+
+```{image} logo/e3sm_unified_logo_200.png
+:align: center
+:width: 200 px
+```
+
+## What is E3SM-Unified?
+
+E3SM-Unified is a unified conda-based environment that provides pre- and
+post-processing tools for the Energy Exascale Earth System Model (E3SM). It is
+designed to streamline the analysis, visualization, and transformation of model
+output for scientists and developers.
+
+This environment bundles together a curated set of Python and compiled tools
+that work well across supported platforms, particularly on E3SM-managed
+high-performance computing (HPC) systems.
+
+## Key Features
+
+* Combines dozens of packages into one environment, eliminating setup friction.
+* Maintains consistency across HPC platforms (Anvil, Chrysalis, Compy, etc.).
+* Offers both Conda and Spack-installed components for MPI performance.
+* Fully open source and community-maintained via GitHub.
+
+## Common Use Cases
+
+* Diagnostics and evaluation (e.g., `e3sm_diags`, `MPAS-Analysis`)
+* CMIP output conversion (`e3sm_to_cmip`)
+* Time series generation, viewer creation, and archiving (`zppy`, `zstash`)
+* Domain generation and mesh visualization (`cime_gen_domain`, `mosaic`)
+
+## Supported Platforms
+
+E3SM-Unified is available on many E3SM-supported systems:
+
+* Andes
+* Anvil
+* Chrysalis
+* Compy
+* Dane
+* Frontier
+* Perlmutter
+* Polaris (ALCF)
+* Ruby
+
+It can also be installed on Linux or macOS laptops for limited use (see
+[Quickstart Guide](quickstart.md)). Windows is not supported.
+
+## Getting Help
+
+* [Quickstart Guide](quickstart.md)
+* GitHub: [E3SM-Unified repository](https://github.com/E3SM-Project/e3sm-unified)
+* Slack: `#e3sm-help-postproc`
+* Issues/questions: GitHub Issues or contact [[email protected]](mailto:[email protected])
diff --git a/docs/logo/e3sm_unified_logo.png b/docs/logo/e3sm_unified_logo.png
diff --git a/docs/logo/e3sm_unified_logo_200.png b/docs/logo/e3sm_unified_logo_200.png
diff --git a/docs/packages.md b/docs/packages.md
diff --git a/docs/quickstart.md b/docs/quickstart.md
@@ -0,0 +1,98 @@
+# Quickstart Guide
+
+```{note}
+E3SM-Unified is supported only on Linux, OSX and HPC platforms. It is **not**
+supported on Windows.
+```
+
+## Accessing E3SM-Unified on Supported Machines
+
+On most E3SM-supported HPC systems, E3SM-Unified is already installed and
+ready to use via an activation script.
+
+### Example Activation Commands
+
+```bash
+# Andes
+source /ccs/proj/cli115/software/e3sm-unified/load_latest_e3sm_unified_andes.sh
+
+# Anvil
+source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_anvil.sh
+
+# Chrysalis
+source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
+
+# Compy
+source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
+
+# Dane
+source /usr/workspace/e3sm/apps/e3sm-unified/load_latest_e3sm_unified_dane.sh
+
+# Frontier
+source /ccs/proj/cli115/software/e3sm-unified/load_latest_e3sm_unified_frontier.sh
+
+# Perlmutter
+source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
+
+# Polaris (ALCF)
+source /lus/grand/projects/E3SMinput/soft/e3sm-unified/load_latest_e3sm_unified_polaris.sh
+
+# Ruby
+source /usr/workspace/e3sm/apps/e3sm-unified/load_latest_e3sm_unified_ruby.sh
+```
+
+Once the script is sourced, you'll have access to all the tools provided by
+E3SM-Unified in your environment.
+
+## Verifying Installation
+
+After activation, you can verify that the environment is correctly loaded by
+testing if major packages are importable:
+
+```python
+python -c "import xarray, e3sm_diags, mpas_analysis, zppy"
+```
+
+## Running on Compute Nodes (Optional but Recommended)
+
+Many E3SM-Unified tools (e.g., MOAB, MPAS-Analysis, NCO, TempestRemap) benefit
+from running on compute nodes using MPI-enabled system builds.
+
+Check your system documentation for how to launch interactive compute sessions
+(e.g., `srun`, `salloc`, or `qsub`).
+
+## Installing E3SM-Unified on an Unsupported System
+
+E3SM-Unified is not officially supported on Linux or Mac laptops or
+workstations, but users can install it using `miniforge3`.
+
+### Step-by-Step (Linux/macOS):
+
+```bash
+# Install miniforge3
+wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
+bash "Miniforge3-$(uname)-$(uname -m).sh"
+
+# Create a new environment
+conda create -n esm-unified -c conda-forge e3sm-unified
+
+# Activate it
+conda activate e3sm-unified
+```
+
+Note: On macOS with M1/M2 chips, install the x86\_64 version and use Rosetta 2
+for compatibility.
+
+---
+
+## Related Pages
+
+* [Introduction to E3SM-Unified](introduction.md)
+* [Using E3SM-Unified Tools](using-tools.md)
+* [Troubleshooting](troubleshooting.md)
+
+```{admonition} Need Help?
+- Slack: #e3sm-help-postproc
+- GitHub Issues: https://github.com/E3SM-Project/e3sm-unified/issues
+- Email: [email protected]
+```
diff --git a/docs/releasing/conda-vs-spack.md b/docs/releasing/conda-vs-spack.md
@@ -0,0 +1,106 @@
+# How Conda and Spack Work Together in E3SM-Unified
+
+E3SM-Unified uses a hybrid approach that combines Conda and Spack to build
+and deploy a comprehensive software environment for E3SM analysis and
+diagnostics. This page explains the motivation for this strategy, how the
+components interact, and the shared infrastructure that supports both
+E3SM-Unified and related projects.
+
+---
+
+## Why Combine Conda and Spack?
+
+Each tool solves a different part of the problem:
+
+### ✅ Conda
+
+* Excellent for managing Python packages and their dependencies
+* Supports rapid installation and reproducibility
+* Compatible with conda-forge and custom channels (e.g., `e3sm`)
+* User-friendly interface, especially for scientists and developers
+
+### ✅ Spack
+
+* Designed for building performance-sensitive HPC software
+* Allows fine-grained control over compilers, MPI implementations, and system
+  libraries
+* Better suited for tools written in Fortran/C/C++ with MPI dependencies
+  (e.g., NCO, MOAB, TempestRemap)
+
+### ❗ The Challenge
+
+Neither system alone is sufficient:
+
+* Conda cannot reliably build or run MPI-based binaries across multiple nodes
+  on HPC systems. In our experience, Conda's MPI implementations often fail
+  even for multi-task jobs on a single node, making them unsuitable for
+  high-performance parallel workflows
+* Spack lacks strong support for modern Python environments and is generally
+  harder to use for scientists accustomed to Conda-based workflows. While
+  conda-forge provides access to tens of thousands of Python packages, Spack
+  offers far fewer, meaning many familiar scientific tools are not readily
+  available through Spack alone
+
+---
+
+## Architecture: How They Work Together
+
+E3SM-Unified environments:
+
+1. Use **Conda** to install the core Python tools and lightweight dependencies
+2. Rely on **Spack** to build performance-critical tools outside Conda
+3. Are bundled into a single workflow that ensures compatibility across both
+
+System-specific setup scripts (e.g., `load_latest_e3sm_unified_<machine>.sh`)
+ensure both components are activated correctly.
+
+For MPI-based tools:
+
+* The tools are built with Spack using system compilers and MPI
+* Users automatically access these builds when running on compute nodes
+
+---
+
+## Shared Infrastructure
+
+E3SM-Unified, Polaris, and Compass all rely on the same key components:
+
+* [`mache`](https://github.com/E3SM-Project/mache): A configuration library
+  for detecting machine-specific settings (modules, compilers, paths)
+* [E3SM's Spack fork](https://github.com/E3SM-Project/spack): Centralized
+  control over package versions and build settings
+* Conda: Used consistently to install `mache`, lightweight tools, and Python
+  dependencies
+
+This shared foundation ensures reproducibility and consistency across
+workflows, testbeds, and developer tools in the E3SM ecosystem.
+
+---
+
+## Future Alternatives
+
+As complexity grows, other strategies may be worth evaluating:
+
+### Option: **E4S (Extreme-scale Scientific Software Stack)**
+
+* Spack-based stack of curated HPC tools
+* E4S environments aim to replace the need for manual Spack+Conda integration
+* May offer better long-term sustainability, but lacks Python focus today
+
+🔗 [Explore E4S](https://e4s.io)
+
+### Other Approaches (less suitable currently):
+
+* Pure Spack builds (harder for Python workflows)
+* Pure Conda builds (harder for HPC performance tools)
+* Containers (portability gains, but complex for HPC integration)
+
+---
+
+## Summary
+
+The hybrid Conda + Spack model in E3SM-Unified balances ease of use with HPC
+performance. While more complex to maintain, it provides flexibility,
+compatibility, and performance across diverse systems. Shared infrastructure
+(like `mache` and E3SM's Spack fork) reduces duplication across projects and
+streamlines the release process.