diff --git a/docs/index.md b/docs/index.md index 863fcdc8..5f610635 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,3 +1,17 @@ +```{toctree} +:hidden: +:maxdepth: 3 + +introduction +quickstart +using-tools +testing-release-candidates +releasing/release-workflow +packages +troubleshooting +contributing +``` + # Welcome to E3SM-Unified Documentation ```{image} logo/e3sm_unified_logo_200.png @@ -27,7 +41,6 @@ This documentation is for both **users** of the E3SM-Unified environment and ## ๐Ÿ“š Contents * ๐Ÿ’  [Using E3SM-Unified Tools](using-tools.md) -* ๐Ÿงช [Testing Release Candidates](testing-release-candidates.md) * ๐Ÿšš [The E3SM-Unified Release Workflow](releasing/release-workflow.md) * ๐Ÿ“ฆ [Package Catalog](packages.md) * โ“ [Troubleshooting & FAQs](troubleshooting.md) diff --git a/docs/packages.md b/docs/packages.md index e69de29b..22b5e7c4 100644 --- a/docs/packages.md +++ b/docs/packages.md @@ -0,0 +1,105 @@ +# ๐Ÿ“ฆ E3SM-Unified Package Catalog + +This page provides a complete inventory of packages included in the latest +E3SM-Unified release (`1.11.1`), along with their versions. These packages +come from the `meta.yaml` build file and represent the authoritative source +for what is installed in the environment. + +--- + +## Main Packages and Dependencies + +| Package Name | Version Constraint | +|----------------------|-----------------------------| +| blas | | +| bottleneck | | +| cartopy | >=0.17.0 | +| cdat_info | 8.2.1 | +| cdms2 | 3.1.5 | +| cdtime | 3.1.4 | +| cdutil | 8.2.1 | +| cdo | | +| chemdyg | 1.1.0 | +| cime_gen_domain | 6.1.59 | +| cmocean | | +| cython | | +| dask | 2024.8.0 (py<3.10), 2024.11.2 (py>=3.10) | +| dogpile.cache | | +| e3sm_diags | 3.0.0 | +| e3sm-tools | 3.0.2 | +| e3sm_to_cmip | 1.11.3 | +| eofs | | +| esmf | 8.8.0 | +| esmpy | 8.8.0 | +| f90nml | | +| ffmpeg | | +| genutil | 8.2.1 | +| geometric_features | 1.6.1 | +| globus-cli | >=3.15.0 | +| globus-sdk | | +| gsw | | +| hdf5 | 1.14.3 | +| ilamb | 2.7.2 | +| ipygany | | +| ipython | | +| jupyter | | +| libnetcdf | 4.9.2 | +| livvkit | 3.1.0 | +| lxml | | +| mache | 1.28.0 | +| matplotlib-base | 3.9.4 | +| metpy | | +| mkdocs-bibtex | 2.18.0 | +| mkdocs-material | | +| mkdocs-monorepo-plugin | | +| mkdocs-redirects | | +| moab | 5.5.1 | +| mosaic | 1.1.0 (py>=3.10) | +| mpas-analysis | 1.13.0 | +| mpas_tools | 0.36.0 | +| mpi4py | | +| nco | 5.3.2 | +| nc-time-axis | | +| ncview | 2.1.8 | +| ncvis-climate | 2024.01.26 | +| netcdf4 | 1.7.2 nompi_* | +| notebook | <7.0.0 | +| numpy | >=2.0.0 | +| openssh | | +| output_viewer | 1.3.3 | +| pandas | | +| pcmdi_metrics | 3.8.2 | +| pillow | | +| plotly | | +| progessbar2 | | +| proj | 9.5.1 | +| psutil | | +| pyproj | 3.6.1 (py<3.10), 3.7.0 (py>=3.10) | +| pyevtk | | +| pyremap | | +| pytest | | +| pywavelets | | +| r | | +| r-dplyr | 1.1.4 | +| r-ggplot2 | | +| r-tidyverse | | +| scikit-image | | +| scipy | >=0.9.0 | +| shapely | | +| squadgen | 1.2.2 | +| sympy | >=0.7.6 | +| tabulate | | +| tempest-extremes | 2.2.3 | +| tempest-remap | 2.2.0 | +| uxarray | 2024.11.1 (py<3.10), >=2024.12.0 (py>=3.10) | +| windspharm | | +| xarray | 2024.7.0 (py<3.10), 2025.1.1 (py>=3.10) | +| xcdat | 0.8.0 | +| xesmf | 0.8.8 | +| zppy | 3.0.0 | +| zppy-interfaces | 0.1.2 | +| zstash | 1.4.4 | + +--- + +This table reflects the current runtime environment for E3SM-Unified `1.11.1`. diff --git a/docs/using-tools.md b/docs/using-tools.md new file mode 100644 index 00000000..051892fa --- /dev/null +++ b/docs/using-tools.md @@ -0,0 +1,204 @@ +# Using Tools in E3SM-Unified + +E3SM-Unified includes a suite of powerful tools for analysis, diagnostics, +file manipulation, and more. This page provides quick summaries and links to +documentation to help you get started with some of the most commonly used +tools. + +--- + +## ๐Ÿ“ˆ E3SM Diags + +E3SM Diagnostics (E3SM Diags) enables model-vs-observation and model-vs-model +comparisons using a wide array of climatological diagnostics. + +- **Core Use Cases**: + - Climatology comparisons (e.g., model vs. obs, model vs. model) + - Time series comparisons + - Diagnostic sets including ENSO, QBO, streamflow, tropical cyclones, and + more + +- **Quickstart**: + 1. Clone or download an + [example](https://github.com/E3SM-Project/e3sm_diags/tree/master/examples) + and modify the parameters file + 2. Activate the E3SM-Unified environment on an HPC machine (e.g., + Perlmutter): + ```bash + source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh + ``` + 3. Run with multiprocessing: + ```bash + python ex4.py --multiprocessing --num_workers=32 -d diags.cfg + ``` + 4. View output using the generated HTML viewer + +- **CMIP Preprocessing** (for time series obs/model comparisons): + - Rename files to E3SM format: `_01_12.nc` + - Place all relevant files in the same directory + +- **Visualization Examples**: + - Latitude-longitude and polar maps + - Taylor diagrams + - Pressure-latitude and pressure-longitude slices + - Time series and zonal mean plots + +๐Ÿ”— [Documentation Hub](https://docs.e3sm.org/e3sm_diags/_build/html/main/index.html) +๐Ÿ”— [Run Examples](https://docs.e3sm.org/e3sm_diags/_build/html/main/examples.html) +๐Ÿ”— [Example Output Viewer](https://portal.nersc.gov/cfs/e3sm/forsyth/examples/ex1-model_ts-vs-model_ts/viewer/) + +--- + +## ๐Ÿ“ MOAB + +MOAB provides mesh and geometry tools for analysis workflows using unstructured +grids. + +Used for working with unstructured meshes and remapping โ€” useful in +preprocessing, remapping or analyzing EAM and MPAS grids. + +๐Ÿ”— [MOAB Documentation](https://sigma.mcs.anl.gov/moab-library/) + +--- + +## ๐Ÿ“Š MPAS-Analysis + +MPAS-Analysis is used to evaluate MPAS-Ocean and MPAS-Seaice output. + +- **Quickstart**: + - Copy or create a config file (e.g., `myrun.cfg`) + - Run: + ```bash + mpas_analysis myrun.cfg + ``` + - Optional: use `--purge` to remove old results, or `--generate` to run a + subset of tasks +- **Parallel runs**: supported via job scripts for common HPC systems +- **Custom plots**: generate new plots using precomputed climatology and time + series data + +๐Ÿ”— [MPAS-Analysis: Running the Analysis](https://mpas-dev.github.io/MPAS-Analysis/develop/users_guide/quick_start.html#running-the-analysis) +๐Ÿ”— [Example config files](https://github.com/MPAS-Dev/MPAS-Analysis/tree/develop/configs) + +--- + +## ๐Ÿงช NCO + +NCO (NetCDF Operators) provides command-line tools for manipulating NetCDF +files. + +Includes tools like `ncreamp`, `ncclimo`, `ncks`, `ncra`, `ncrcat`, and +`ncatted` for computing climatologies and extracting time series as well as +remapping, subsetting, concatenating, and editing NetCDF files. + +๐Ÿ”— [NCO Manual](http://nco.sourceforge.net/nco.html) + +--- + +## โš™๏ธ zppy + +[`zppy`](https://github.com/E3SM-Project/zppy) (pronounced *zippee*) is E3SMโ€™s +main post-processing toolchain. It automates key analysis steps like generating +time series, climatologies, and diagnostic plots using **NCO**, **E3SM Diags**, +and **MPAS-Analysis** โ€” especially helpful for long or high-resolution +simulations. + +Users control zppy through a single `.cfg` configuration file, specifying: + +- Input/output directories +- Model components and time ranges to post-process +- Which analysis tasks to run (e.g., `e3sm_diags`, `mpas_analysis`, `ts`, + `climo`, etc.) + +zppy uses this config to generate and submit batch jobs (via SLURM) with +automatic dependency management. + +### Example usage + +Copy or create a `.cfg` file (e.g., `post.mysimulation.cfg`) containing task +and data settings. + +```bash +zppy -c post.mysimulation.cfg +``` + +- Post-processed outputs (NetCDF + plots) go in the `output` and `www` + directories +- Job logs and submission scripts are written under `/post/scripts` +- Each zppy task can be debugged by editing and resubmitting its `.bash` script + +### Configuration Highlights + +- Supports custom and campaign-specific settings (e.g., + `campaign = water_cycle`) +- Each analysis block is defined using `[section]` and `[[subsection]]` syntax +- Works with native-resolution and RRM simulations + +๐Ÿ”— [View the zppy tutorial and examples](https://docs.e3sm.org/zppy/_build/html/main/tutorial.html) + +--- + +## ๐Ÿ“ฆ zstash + +[`zstash`](https://github.com/E3SM-Project/zstash) is the +**long-term HPSS archiving** solution used by E3SM to store simulation output +in an efficient and reproducible way. + +It is written in Python and designed to be lightweight, fast, and reliable โ€” +emphasizing portability and long-term stability over complex features. + +### Key Features + +- Archives files into **tarballs** with user-defined max size +- Uses **MD5 checksums** (computed on-the-fly) for both files and tars +- Stores all metadata (paths, sizes, checksums, offsets) in a + **SQLite database** +- Allows for **selective extraction** of files from tarballs +- Verifies **file integrity** during extraction + +### Basic Usage + +```bash +# Archive files to HPSS +zstash create --hpss=HPSS_PATH --keep input_dir + +# Extract files from archive +zstash extract --hpss=HPSS_PATH --files=FILELIST + +# Check archive integrity +zstash check --hpss=HPSS_PATH +``` + +> For best performance, run `zstash` on **data transfer nodes** (DTNs) of + supported machines. + +### Installation + +`zstash` is included in the E3SM-Unified environment on supported HPC systems: + +```bash +source /path/to/load_latest_e3sm_unified_.sh +``` + +You can also install the latest release manually: + +```bash +mamba create -n zstash_env zstash +conda activate zstash_env +``` + +### Documentation and Tutorials + +- ๐Ÿ”— [Getting started](https://docs.e3sm.org/zstash/_build/html/main/getting_started.html) +- ๐Ÿ”— [Install via conda](https://docs.e3sm.org/zstash/_build/html/main/getting_started.html#installation-in-a-conda-environment) +- ๐Ÿ”— [Best practices](https://docs.e3sm.org/zstash/_build/html/main/best_practices.html) +- ๐Ÿ”— [zstash command usage](https://docs.e3sm.org/zstash/_build/html/main/usage.html) + +--- + +## ๐Ÿงฐ More Tools + +Other included tools (e.g., `e3sm_to_cmip`, `mpas_tools`, `pyremap`, `xcdat`) +have documentation hosted on their respective repos or conda-forge feedstocks. + +We welcome suggestions for additional summaries and links!