Skip to content

Commit 17f147a

Browse files
xylarxylar
authored
Merge pull request #132 from xylar/fix-docs
Add table of contents and missing pages
2 parents 5c95541 + 1c8ffc8 commit 17f147a

File tree

3 files changed

+323
-1
lines changed

3 files changed

+323
-1
lines changed

‎docs/index.md‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,17 @@
1+
```{toctree}
2+
:hidden:
3+
:maxdepth: 3
4+
5+
introduction
6+
quickstart
7+
using-tools
8+
testing-release-candidates
9+
releasing/release-workflow
10+
packages
11+
troubleshooting
12+
contributing
13+
```
14+
115
# Welcome to E3SM-Unified Documentation
216

317
```{image} logo/e3sm_unified_logo_200.png
@@ -27,7 +41,6 @@ This documentation is for both **users** of the E3SM-Unified environment and
2741
## 📚 Contents
2842

2943
* 💠 [Using E3SM-Unified Tools](using-tools.md)
30-
* 🧪 [Testing Release Candidates](testing-release-candidates.md)
3144
* 🚚 [The E3SM-Unified Release Workflow](releasing/release-workflow.md)
3245
* 📦 [Package Catalog](packages.md)
3346
* ❓ [Troubleshooting & FAQs](troubleshooting.md)

‎docs/packages.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# 📦 E3SM-Unified Package Catalog
2+
3+
This page provides a complete inventory of packages included in the latest
4+
E3SM-Unified release (`1.11.1`), along with their versions. These packages
5+
come from the `meta.yaml` build file and represent the authoritative source
6+
for what is installed in the environment.
7+
8+
---
9+
10+
## Main Packages and Dependencies
11+
12+
| Package Name | Version Constraint |
13+
|----------------------|-----------------------------|
14+
| blas | |
15+
| bottleneck | |
16+
| cartopy | >=0.17.0 |
17+
| cdat_info | 8.2.1 |
18+
| cdms2 | 3.1.5 |
19+
| cdtime | 3.1.4 |
20+
| cdutil | 8.2.1 |
21+
| cdo | |
22+
| chemdyg | 1.1.0 |
23+
| cime_gen_domain | 6.1.59 |
24+
| cmocean | |
25+
| cython | |
26+
| dask | 2024.8.0 (py<3.10), 2024.11.2 (py>=3.10) |
27+
| dogpile.cache | |
28+
| e3sm_diags | 3.0.0 |
29+
| e3sm-tools | 3.0.2 |
30+
| e3sm_to_cmip | 1.11.3 |
31+
| eofs | |
32+
| esmf | 8.8.0 |
33+
| esmpy | 8.8.0 |
34+
| f90nml | |
35+
| ffmpeg | |
36+
| genutil | 8.2.1 |
37+
| geometric_features | 1.6.1 |
38+
| globus-cli | >=3.15.0 |
39+
| globus-sdk | |
40+
| gsw | |
41+
| hdf5 | 1.14.3 |
42+
| ilamb | 2.7.2 |
43+
| ipygany | |
44+
| ipython | |
45+
| jupyter | |
46+
| libnetcdf | 4.9.2 |
47+
| livvkit | 3.1.0 |
48+
| lxml | |
49+
| mache | 1.28.0 |
50+
| matplotlib-base | 3.9.4 |
51+
| metpy | |
52+
| mkdocs-bibtex | 2.18.0 |
53+
| mkdocs-material | |
54+
| mkdocs-monorepo-plugin | |
55+
| mkdocs-redirects | |
56+
| moab | 5.5.1 |
57+
| mosaic | 1.1.0 (py>=3.10) |
58+
| mpas-analysis | 1.13.0 |
59+
| mpas_tools | 0.36.0 |
60+
| mpi4py | |
61+
| nco | 5.3.2 |
62+
| nc-time-axis | |
63+
| ncview | 2.1.8 |
64+
| ncvis-climate | 2024.01.26 |
65+
| netcdf4 | 1.7.2 nompi_* |
66+
| notebook | <7.0.0 |
67+
| numpy | >=2.0.0 |
68+
| openssh | |
69+
| output_viewer | 1.3.3 |
70+
| pandas | |
71+
| pcmdi_metrics | 3.8.2 |
72+
| pillow | |
73+
| plotly | |
74+
| progessbar2 | |
75+
| proj | 9.5.1 |
76+
| psutil | |
77+
| pyproj | 3.6.1 (py<3.10), 3.7.0 (py>=3.10) |
78+
| pyevtk | |
79+
| pyremap | |
80+
| pytest | |
81+
| pywavelets | |
82+
| r | |
83+
| r-dplyr | 1.1.4 |
84+
| r-ggplot2 | |
85+
| r-tidyverse | |
86+
| scikit-image | |
87+
| scipy | >=0.9.0 |
88+
| shapely | |
89+
| squadgen | 1.2.2 |
90+
| sympy | >=0.7.6 |
91+
| tabulate | |
92+
| tempest-extremes | 2.2.3 |
93+
| tempest-remap | 2.2.0 |
94+
| uxarray | 2024.11.1 (py<3.10), >=2024.12.0 (py>=3.10) |
95+
| windspharm | |
96+
| xarray | 2024.7.0 (py<3.10), 2025.1.1 (py>=3.10) |
97+
| xcdat | 0.8.0 |
98+
| xesmf | 0.8.8 |
99+
| zppy | 3.0.0 |
100+
| zppy-interfaces | 0.1.2 |
101+
| zstash | 1.4.4 |
102+
103+
---
104+
105+
This table reflects the current runtime environment for E3SM-Unified `1.11.1`.

‎docs/using-tools.md‎

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
# Using Tools in E3SM-Unified
2+
3+
E3SM-Unified includes a suite of powerful tools for analysis, diagnostics,
4+
file manipulation, and more. This page provides quick summaries and links to
5+
documentation to help you get started with some of the most commonly used
6+
tools.
7+
8+
---
9+
10+
## 📈 E3SM Diags
11+
12+
E3SM Diagnostics (E3SM Diags) enables model-vs-observation and model-vs-model
13+
comparisons using a wide array of climatological diagnostics.
14+
15+
- **Core Use Cases**:
16+
- Climatology comparisons (e.g., model vs. obs, model vs. model)
17+
- Time series comparisons
18+
- Diagnostic sets including ENSO, QBO, streamflow, tropical cyclones, and
19+
more
20+
21+
- **Quickstart**:
22+
1. Clone or download an
23+
[example](https://github.com/E3SM-Project/e3sm_diags/tree/master/examples)
24+
and modify the parameters file
25+
2. Activate the E3SM-Unified environment on an HPC machine (e.g.,
26+
Perlmutter):
27+
```bash
28+
source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
29+
```
30+
3. Run with multiprocessing:
31+
```bash
32+
python ex4.py --multiprocessing --num_workers=32 -d diags.cfg
33+
```
34+
4. View output using the generated HTML viewer
35+
36+
- **CMIP Preprocessing** (for time series obs/model comparisons):
37+
- Rename files to E3SM format: `<variable>_<start_yr>01_<end_yr>12.nc`
38+
- Place all relevant files in the same directory
39+
40+
- **Visualization Examples**:
41+
- Latitude-longitude and polar maps
42+
- Taylor diagrams
43+
- Pressure-latitude and pressure-longitude slices
44+
- Time series and zonal mean plots
45+
46+
🔗 [Documentation Hub](https://docs.e3sm.org/e3sm_diags/_build/html/main/index.html)
47+
🔗 [Run Examples](https://docs.e3sm.org/e3sm_diags/_build/html/main/examples.html)
48+
🔗 [Example Output Viewer](https://portal.nersc.gov/cfs/e3sm/forsyth/examples/ex1-model_ts-vs-model_ts/viewer/)
49+
50+
---
51+
52+
## 📐 MOAB
53+
54+
MOAB provides mesh and geometry tools for analysis workflows using unstructured
55+
grids.
56+
57+
Used for working with unstructured meshes and remapping — useful in
58+
preprocessing, remapping or analyzing EAM and MPAS grids.
59+
60+
🔗 [MOAB Documentation](https://sigma.mcs.anl.gov/moab-library/)
61+
62+
---
63+
64+
## 📊 MPAS-Analysis
65+
66+
MPAS-Analysis is used to evaluate MPAS-Ocean and MPAS-Seaice output.
67+
68+
- **Quickstart**:
69+
- Copy or create a config file (e.g., `myrun.cfg`)
70+
- Run:
71+
```bash
72+
mpas_analysis myrun.cfg
73+
```
74+
- Optional: use `--purge` to remove old results, or `--generate` to run a
75+
subset of tasks
76+
- **Parallel runs**: supported via job scripts for common HPC systems
77+
- **Custom plots**: generate new plots using precomputed climatology and time
78+
series data
79+
80+
🔗 [MPAS-Analysis: Running the Analysis](https://mpas-dev.github.io/MPAS-Analysis/develop/users_guide/quick_start.html#running-the-analysis)
81+
🔗 [Example config files](https://github.com/MPAS-Dev/MPAS-Analysis/tree/develop/configs)
82+
83+
---
84+
85+
## 🧪 NCO
86+
87+
NCO (NetCDF Operators) provides command-line tools for manipulating NetCDF
88+
files.
89+
90+
Includes tools like `ncreamp`, `ncclimo`, `ncks`, `ncra`, `ncrcat`, and
91+
`ncatted` for computing climatologies and extracting time series as well as
92+
remapping, subsetting, concatenating, and editing NetCDF files.
93+
94+
🔗 [NCO Manual](http://nco.sourceforge.net/nco.html)
95+
96+
---
97+
98+
## ⚙️ zppy
99+
100+
[`zppy`](https://github.com/E3SM-Project/zppy) (pronounced *zippee*) is E3SM’s
101+
main post-processing toolchain. It automates key analysis steps like generating
102+
time series, climatologies, and diagnostic plots using **NCO**, **E3SM Diags**,
103+
and **MPAS-Analysis** — especially helpful for long or high-resolution
104+
simulations.
105+
106+
Users control zppy through a single `.cfg` configuration file, specifying:
107+
108+
- Input/output directories
109+
- Model components and time ranges to post-process
110+
- Which analysis tasks to run (e.g., `e3sm_diags`, `mpas_analysis`, `ts`,
111+
`climo`, etc.)
112+
113+
zppy uses this config to generate and submit batch jobs (via SLURM) with
114+
automatic dependency management.
115+
116+
### Example usage
117+
118+
Copy or create a `.cfg` file (e.g., `post.mysimulation.cfg`) containing task
119+
and data settings.
120+
121+
```bash
122+
zppy -c post.mysimulation.cfg
123+
```
124+
125+
- Post-processed outputs (NetCDF + plots) go in the `output` and `www`
126+
directories
127+
- Job logs and submission scripts are written under `<output>/post/scripts`
128+
- Each zppy task can be debugged by editing and resubmitting its `.bash` script
129+
130+
### Configuration Highlights
131+
132+
- Supports custom and campaign-specific settings (e.g.,
133+
`campaign = water_cycle`)
134+
- Each analysis block is defined using `[section]` and `[[subsection]]` syntax
135+
- Works with native-resolution and RRM simulations
136+
137+
🔗 [View the zppy tutorial and examples](https://docs.e3sm.org/zppy/_build/html/main/tutorial.html)
138+
139+
---
140+
141+
## 📦 zstash
142+
143+
[`zstash`](https://github.com/E3SM-Project/zstash) is the
144+
**long-term HPSS archiving** solution used by E3SM to store simulation output
145+
in an efficient and reproducible way.
146+
147+
It is written in Python and designed to be lightweight, fast, and reliable —
148+
emphasizing portability and long-term stability over complex features.
149+
150+
### Key Features
151+
152+
- Archives files into **tarballs** with user-defined max size
153+
- Uses **MD5 checksums** (computed on-the-fly) for both files and tars
154+
- Stores all metadata (paths, sizes, checksums, offsets) in a
155+
**SQLite database**
156+
- Allows for **selective extraction** of files from tarballs
157+
- Verifies **file integrity** during extraction
158+
159+
### Basic Usage
160+
161+
```bash
162+
# Archive files to HPSS
163+
zstash create --hpss=HPSS_PATH --keep input_dir
164+
165+
# Extract files from archive
166+
zstash extract --hpss=HPSS_PATH --files=FILELIST
167+
168+
# Check archive integrity
169+
zstash check --hpss=HPSS_PATH
170+
```
171+
172+
> For best performance, run `zstash` on **data transfer nodes** (DTNs) of
173+
supported machines.
174+
175+
### Installation
176+
177+
`zstash` is included in the E3SM-Unified environment on supported HPC systems:
178+
179+
```bash
180+
source /path/to/load_latest_e3sm_unified_<machine>.sh
181+
```
182+
183+
You can also install the latest release manually:
184+
185+
```bash
186+
mamba create -n zstash_env zstash
187+
conda activate zstash_env
188+
```
189+
190+
### Documentation and Tutorials
191+
192+
- 🔗 [Getting started](https://docs.e3sm.org/zstash/_build/html/main/getting_started.html)
193+
- 🔗 [Install via conda](https://docs.e3sm.org/zstash/_build/html/main/getting_started.html#installation-in-a-conda-environment)
194+
- 🔗 [Best practices](https://docs.e3sm.org/zstash/_build/html/main/best_practices.html)
195+
- 🔗 [zstash command usage](https://docs.e3sm.org/zstash/_build/html/main/usage.html)
196+
197+
---
198+
199+
## 🧰 More Tools
200+
201+
Other included tools (e.g., `e3sm_to_cmip`, `mpas_tools`, `pyremap`, `xcdat`)
202+
have documentation hosted on their respective repos or conda-forge feedstocks.
203+
204+
We welcome suggestions for additional summaries and links!

0 commit comments

Comments
 (0)