Skip to content

Add table of contents and missing pages #132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 20, 2025
Merged

Add table of contents and missing pages #132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
```{toctree}
:hidden:
:maxdepth: 3

introduction
quickstart
using-tools
testing-release-candidates
releasing/release-workflow
packages
troubleshooting
contributing
```

# Welcome to E3SM-Unified Documentation

```{image} logo/e3sm_unified_logo_200.png
Expand Down Expand Up @@ -27,7 +41,6 @@ This documentation is for both **users** of the E3SM-Unified environment and
## 📚 Contents

* 💠 [Using E3SM-Unified Tools](using-tools.md)
* 🧪 [Testing Release Candidates](testing-release-candidates.md)
* 🚚 [The E3SM-Unified Release Workflow](releasing/release-workflow.md)
* 📦 [Package Catalog](packages.md)
* ❓ [Troubleshooting & FAQs](troubleshooting.md)
Expand Down
105 changes: 105 additions & 0 deletions docs/packages.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# 📦 E3SM-Unified Package Catalog

This page provides a complete inventory of packages included in the latest
E3SM-Unified release (`1.11.1`), along with their versions. These packages
come from the `meta.yaml` build file and represent the authoritative source
for what is installed in the environment.

---

## Main Packages and Dependencies

| Package Name | Version Constraint |
|----------------------|-----------------------------|
| blas | |
| bottleneck | |
| cartopy | >=0.17.0 |
| cdat_info | 8.2.1 |
| cdms2 | 3.1.5 |
| cdtime | 3.1.4 |
| cdutil | 8.2.1 |
| cdo | |
| chemdyg | 1.1.0 |
| cime_gen_domain | 6.1.59 |
| cmocean | |
| cython | |
| dask | 2024.8.0 (py<3.10), 2024.11.2 (py>=3.10) |
| dogpile.cache | |
| e3sm_diags | 3.0.0 |
| e3sm-tools | 3.0.2 |
| e3sm_to_cmip | 1.11.3 |
| eofs | |
| esmf | 8.8.0 |
| esmpy | 8.8.0 |
| f90nml | |
| ffmpeg | |
| genutil | 8.2.1 |
| geometric_features | 1.6.1 |
| globus-cli | >=3.15.0 |
| globus-sdk | |
| gsw | |
| hdf5 | 1.14.3 |
| ilamb | 2.7.2 |
| ipygany | |
| ipython | |
| jupyter | |
| libnetcdf | 4.9.2 |
| livvkit | 3.1.0 |
| lxml | |
| mache | 1.28.0 |
| matplotlib-base | 3.9.4 |
| metpy | |
| mkdocs-bibtex | 2.18.0 |
| mkdocs-material | |
| mkdocs-monorepo-plugin | |
| mkdocs-redirects | |
| moab | 5.5.1 |
| mosaic | 1.1.0 (py>=3.10) |
| mpas-analysis | 1.13.0 |
| mpas_tools | 0.36.0 |
| mpi4py | |
| nco | 5.3.2 |
| nc-time-axis | |
| ncview | 2.1.8 |
| ncvis-climate | 2024.01.26 |
| netcdf4 | 1.7.2 nompi_* |
| notebook | <7.0.0 |
| numpy | >=2.0.0 |
| openssh | |
| output_viewer | 1.3.3 |
| pandas | |
| pcmdi_metrics | 3.8.2 |
| pillow | |
| plotly | |
| progessbar2 | |
| proj | 9.5.1 |
| psutil | |
| pyproj | 3.6.1 (py<3.10), 3.7.0 (py>=3.10) |
| pyevtk | |
| pyremap | |
| pytest | |
| pywavelets | |
| r | |
| r-dplyr | 1.1.4 |
| r-ggplot2 | |
| r-tidyverse | |
| scikit-image | |
| scipy | >=0.9.0 |
| shapely | |
| squadgen | 1.2.2 |
| sympy | >=0.7.6 |
| tabulate | |
| tempest-extremes | 2.2.3 |
| tempest-remap | 2.2.0 |
| uxarray | 2024.11.1 (py<3.10), >=2024.12.0 (py>=3.10) |
| windspharm | |
| xarray | 2024.7.0 (py<3.10), 2025.1.1 (py>=3.10) |
| xcdat | 0.8.0 |
| xesmf | 0.8.8 |
| zppy | 3.0.0 |
| zppy-interfaces | 0.1.2 |
| zstash | 1.4.4 |

---

This table reflects the current runtime environment for E3SM-Unified `1.11.1`.
204 changes: 204 additions & 0 deletions docs/using-tools.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
# Using Tools in E3SM-Unified

E3SM-Unified includes a suite of powerful tools for analysis, diagnostics,
file manipulation, and more. This page provides quick summaries and links to
documentation to help you get started with some of the most commonly used
tools.

---

## 📈 E3SM Diags

E3SM Diagnostics (E3SM Diags) enables model-vs-observation and model-vs-model
comparisons using a wide array of climatological diagnostics.

- **Core Use Cases**:
- Climatology comparisons (e.g., model vs. obs, model vs. model)
- Time series comparisons
- Diagnostic sets including ENSO, QBO, streamflow, tropical cyclones, and
more

- **Quickstart**:
1. Clone or download an
[example](https://github.com/E3SM-Project/e3sm_diags/tree/master/examples)
and modify the parameters file
2. Activate the E3SM-Unified environment on an HPC machine (e.g.,
Perlmutter):
```bash
source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
```
3. Run with multiprocessing:
```bash
python ex4.py --multiprocessing --num_workers=32 -d diags.cfg
```
4. View output using the generated HTML viewer

- **CMIP Preprocessing** (for time series obs/model comparisons):
- Rename files to E3SM format: `<variable>_<start_yr>01_<end_yr>12.nc`
- Place all relevant files in the same directory

- **Visualization Examples**:
- Latitude-longitude and polar maps
- Taylor diagrams
- Pressure-latitude and pressure-longitude slices
- Time series and zonal mean plots

🔗 [Documentation Hub](https://docs.e3sm.org/e3sm_diags/_build/html/main/index.html)
🔗 [Run Examples](https://docs.e3sm.org/e3sm_diags/_build/html/main/examples.html)
🔗 [Example Output Viewer](https://portal.nersc.gov/cfs/e3sm/forsyth/examples/ex1-model_ts-vs-model_ts/viewer/)

---

## 📐 MOAB

MOAB provides mesh and geometry tools for analysis workflows using unstructured
grids.

Used for working with unstructured meshes and remapping — useful in
preprocessing, remapping or analyzing EAM and MPAS grids.

🔗 [MOAB Documentation](https://sigma.mcs.anl.gov/moab-library/)

---

## 📊 MPAS-Analysis

MPAS-Analysis is used to evaluate MPAS-Ocean and MPAS-Seaice output.

- **Quickstart**:
- Copy or create a config file (e.g., `myrun.cfg`)
- Run:
```bash
mpas_analysis myrun.cfg
```
- Optional: use `--purge` to remove old results, or `--generate` to run a
subset of tasks
- **Parallel runs**: supported via job scripts for common HPC systems
- **Custom plots**: generate new plots using precomputed climatology and time
series data

🔗 [MPAS-Analysis: Running the Analysis](https://mpas-dev.github.io/MPAS-Analysis/develop/users_guide/quick_start.html#running-the-analysis)
🔗 [Example config files](https://github.com/MPAS-Dev/MPAS-Analysis/tree/develop/configs)

---

## 🧪 NCO

NCO (NetCDF Operators) provides command-line tools for manipulating NetCDF
files.

Includes tools like `ncreamp`, `ncclimo`, `ncks`, `ncra`, `ncrcat`, and
`ncatted` for computing climatologies and extracting time series as well as
remapping, subsetting, concatenating, and editing NetCDF files.

🔗 [NCO Manual](http://nco.sourceforge.net/nco.html)

---

## ⚙️ zppy

[`zppy`](https://github.com/E3SM-Project/zppy) (pronounced *zippee*) is E3SM’s
main post-processing toolchain. It automates key analysis steps like generating
time series, climatologies, and diagnostic plots using **NCO**, **E3SM Diags**,
and **MPAS-Analysis** — especially helpful for long or high-resolution
simulations.

Users control zppy through a single `.cfg` configuration file, specifying:

- Input/output directories
- Model components and time ranges to post-process
- Which analysis tasks to run (e.g., `e3sm_diags`, `mpas_analysis`, `ts`,
`climo`, etc.)

zppy uses this config to generate and submit batch jobs (via SLURM) with
automatic dependency management.

### Example usage

Copy or create a `.cfg` file (e.g., `post.mysimulation.cfg`) containing task
and data settings.

```bash
zppy -c post.mysimulation.cfg
```

- Post-processed outputs (NetCDF + plots) go in the `output` and `www`
directories
- Job logs and submission scripts are written under `<output>/post/scripts`
- Each zppy task can be debugged by editing and resubmitting its `.bash` script

### Configuration Highlights

- Supports custom and campaign-specific settings (e.g.,
`campaign = water_cycle`)
- Each analysis block is defined using `[section]` and `[[subsection]]` syntax
- Works with native-resolution and RRM simulations

🔗 [View the zppy tutorial and examples](https://docs.e3sm.org/zppy/_build/html/main/tutorial.html)

---

## 📦 zstash

[`zstash`](https://github.com/E3SM-Project/zstash) is the
**long-term HPSS archiving** solution used by E3SM to store simulation output
in an efficient and reproducible way.

It is written in Python and designed to be lightweight, fast, and reliable —
emphasizing portability and long-term stability over complex features.

### Key Features

- Archives files into **tarballs** with user-defined max size
- Uses **MD5 checksums** (computed on-the-fly) for both files and tars
- Stores all metadata (paths, sizes, checksums, offsets) in a
**SQLite database**
- Allows for **selective extraction** of files from tarballs
- Verifies **file integrity** during extraction

### Basic Usage

```bash
# Archive files to HPSS
zstash create --hpss=HPSS_PATH --keep input_dir

# Extract files from archive
zstash extract --hpss=HPSS_PATH --files=FILELIST

# Check archive integrity
zstash check --hpss=HPSS_PATH
```

> For best performance, run `zstash` on **data transfer nodes** (DTNs) of
supported machines.

### Installation

`zstash` is included in the E3SM-Unified environment on supported HPC systems:

```bash
source /path/to/load_latest_e3sm_unified_<machine>.sh
```

You can also install the latest release manually:

```bash
mamba create -n zstash_env zstash
conda activate zstash_env
```

### Documentation and Tutorials

- 🔗 [Getting started](https://docs.e3sm.org/zstash/_build/html/main/getting_started.html)
- 🔗 [Install via conda](https://docs.e3sm.org/zstash/_build/html/main/getting_started.html#installation-in-a-conda-environment)
- 🔗 [Best practices](https://docs.e3sm.org/zstash/_build/html/main/best_practices.html)
- 🔗 [zstash command usage](https://docs.e3sm.org/zstash/_build/html/main/usage.html)

---

## 🧰 More Tools

Other included tools (e.g., `e3sm_to_cmip`, `mpas_tools`, `pyremap`, `xcdat`)
have documentation hosted on their respective repos or conda-forge feedstocks.

We welcome suggestions for additional summaries and links!