v1.11.1

A bug-fix release that updates zppy-interfaces to v0.1.2. This was inadvertently missed in 1.11.0.

Full Changelog: 1.11.0...1.11.1

v1.11.0

In this version

• Many packages have been updated, including:
  • cdo
  • cime_gen_domain 6.1.59
  • chemdyg 1.1.0
  • e3sm_diags 3.0.0
  • e3sm_to_cmip 1.11.3
  • e3sm-tools 3.0.2
  • geometric_features 1.6.1
  • ilamb 2.7.2
  • livvkit 3.1.0
  • mache 1.28.0
  • mosaic 1.1.0
  • mpas-analysis 1.13.0
  • mpas_tools 0.36.0
  • nco 5.3.2
  • pcmdi_metrics 3.8.2
  • squadgen 1.2.2
  • xcdat 0.8.0
  • zppy 3.0.0
  • zppy-interfaces 0.1.1
  • zstash 1.4.4

New and improved

• cime_gen_domain:

  • The gen_domain tool as part of the CIME v6.1.59 tag, now separate from e3sm-tools

• ChemDyg:

  • Updates to be compatible with zppy (v3.0.0)
  • Some minor fixes

• E3SM-Diags:

  • v3.0.0 marks a major milestone after nearly two years of work by the core development team. This release introduces a completely new back-end, replacing CDAT with Xarray and xCDAT. The user-facing API for running E3SM Diagnostics remains backward-compatible between v2 and v3. The modernization improves performance, usability, and maintainability, unit tests coverage and regression testing framework.

  • User-facing Enhancements

    • Add support for most EAMxx variables.
    • Replaced SciPy cwt module with PyWavelets cwt for the added wavelet analysis in QBO set.
    • The viewer is now at the top level of results directory
    • Revamped logger module for additional features and bug fixes.
    • View the full changelog [TBA] for more information.

• e3sm_to_cmip:

  • Bug Fixes to fix table/freq ambiguity for multi-frequency variables; Remove positive from hfsifrazil; Fix masking of 3D MPAS-Ocean variables.
  • Additional refactoring of cmor setup for specific variables (areacella, clisccp, orog, sftlf).
  • Consolidate cmor setup code to simplify implementation.

• e3sm-tools

  • Updates to cube_to_target, interpinic, mksurfdata_map as part of the E3SM v3.0.2 tag

• livvkit:

  • Handles missing performance files more gracefully
  • Improves file diff section header titles

• mosaic:

  • Enables visualization of unstructured MPAS data on the native mesh with matplotlib, through drop in replacements for pcolormesh / pcolor.
  • Visualization of fields at all mesh locations (i.e. Cells, Edges, and Vertices) is supported.
  • Supports map projected figures through cartopy for both planar and spherical meshes.
    • Currently only a subset of map projections are supported for spherical meshes, but we are currently working on expanding support to all cartopy projections.

• MPAS-Analysis:

  • Maps of surface mass and heat fluxes
  • Maps of velocity
  • Maps of numerous new sea-ice fields
  • Major bug fix to the horizontal streamfunction

• NCO:

  • Vertical interpolation works on latest ERA5, MERRA2 formats
  • ncclimo timeseries will output regional statistics (scaled sums or means)
  • ELM restart file gridification allows selecting output landunit/column type, re-orders snow layers
  • ncks --chk_map diagnoses grid resolution
  • ncremap regridder improvements, MPAS-A support

• pcmdi_metrics:

• squadgen:

• xcdat:

  • Improve performance of the Regrid2 API.
  • Significantly improved the performance of temporal group averaging by addressing an Xarray bug discovered during debugging. The bug was subsequently fixed by the Xarray team.
  • Enhanced support for temporal averaging with custom seasons
  • Many bug fixes and documentation updates.
  • View the full changelog for more information.

• zppy:

  • zppy 3.0.0 is a major release; there are non-backwards-compatible ("breaking") changes. It is therefore highly recommended to read the release notes (INSERT LINK HERE), in particular for users of e3sm_to_cmip, tc_analysis, global_time_series, and ilamb functionalities.
  • The e3sm_to_cmip functionality is now accessed via its own task rather than via the ts task with ts_fmt = cmip.
  • The global_time_series task is now implemented with a new package, zppy-interfaces, which is meant for Python scripts that extend external packages for easy use by E3SM users, whereas zppy itself is meant for workflow automation (i.e., calling code written outside zppy).
  • global_time_series has also been extended to work for Land variables and to generate viewers similar to those in e3sm_diags.
  • Parameter inference and validation have been improved and formalized. Users now have the ability to specify if they don't want zppy to infer parameters (for a file path or for the section name of a task dependency).
  • CDAT is no longer a dependency for zppy. xCDAT is now user for handling global_time_series and e3sm_diags.
  • The tc_analysis task underwent a number of improvements to be more robust.

• zppy-interfaces:

  • As noted in the zppy section above, this is a new package meant for Python scripts that extend external packages for easy use by E3SM users. It is currently home to the global_time_series plotting functionality. Future additions include the PCMDI metrics plotting.

• zstash:

  • Important changes were made to the zstash code base to make it more robust:
    • --follow-symlinks will now only create a symlink in the destination directory, not also the source directory.
    • --non-blocking functionality (i.e., tars are created while a Globus transfer is ongoing) now occurs if and only if this option is used, as opposed to always occurring.

What's Changed

• Bump evv version for cime-env by @mkstratos in #120
• Set CIME_MODEL to nonsense to prevent building E3SM by @xylar in #119
• Add LICENSE by @xylar in #122
• Drop unsupported python versions in conda build recipe by @altheaden in #123
• Increment EVV4ESM version to latest release by @mkstratos in #124
• Update to v1.11.0 by @xylar in #125

New Contributors

• @altheaden made their first contribution in #123

Full Changelog: 1.10.0...1.11.0

v1.10.0

In this version

• A few packages have been updated, including:
  • chemdyg 1.0.1
  • e3sm_diags 2.12.0
  • e3sm-tools 3.0.0
  • mache 1.22.0
  • mpas-analysis 1.11.0
  • mpas_tools 0.33.0
  • nco 5.2.4
  • r
  • zppy 2.5.0

New and improved

• ChemDyg:

  • The vertical pressure plots are available for different model layers (not for 72 layers only).
  • New QBO analysis includes three types of plots.
  • O3 STE flux monthly lat-lon plots are added.
  • High-level summary table includes not only the global mean but also the northern and southern hemispheres.

• E3SM-Diags:

  • Add Wheeler and Kiladis diagram for tropical subseasonal diags contributed from SciDAC QBO team.

• e3sm-tools

  • A set of tools used to integrate new meshes into E3SM
  • cube_to_target, gen_domain, interpinic, mksurfdata_map, squadgen

• mache:

  • Update OpenMPI modules on Chrysalis

• MPAS-Analysis:

  • Updates to support newer ESMF and scipy

• MPAS-Tools:

  • Add support for a spherical (rather than elliptical) latitude and longitude points on MALI meshes and grow mask to flood-fill mask creation.

• NCO:

  • User-visible NCO changes relative to the previous E3SM-Unified are minor, and consist mainly of improved error messages and helpful suggestions, and reduced metadata propagated into the regional timeseries history attribute.

• R:

  • R is a free software environment for statistical computing and graphics.

• UXarray:

  • UXarray provides Xarray-styled functionality for working with unstructured grids build around the UGRID conventions.

• zppy:

  • Ability to generate custom global_time_series plots for any variable, rather than being restricted to the built-in set.
  • Wave number frequency plot added in e3sm_diags.
  • New reservation parameter added to allow use of a specific reservation on the job scheduler.
  • See full release notes for more.

v1.9.3

In this version

• A few packages have been updated, including:
  • chemdyg 0.1.5
  • e3sm_diags 2.11.0
  • e3sm_to_cmip 1.11.2
  • geometric_features 1.3.0
  • mache 1.20.0
  • mpas-analysis 1.10.0
  • mpas_tools 0.32.0
  • nco 5.2.1
  • zppy 2.4.0

New and improved

• ChemDyg:

  • Save plotting data in netcdf format
  • Available plotting the results with 80 model layers
  • Provide ChemDyg input data sources on zedono ChemDyg input data

• E3SM-Diags:

  • Initial support for variable naming convention for EAMxx
  • Support comparing monthly mean climatology (in addition to seasonal and annual mean)

• e3sm_to_cmip:

  • Update parallel=False in open_mfdataset() to avoid NetCDF: Not a Valid ID error
  • Revised e3sm_to_cmip --info handling for clearer outputs based on frequency and CMIP table

• geometric_features:

  • Added two new critical land blockages to block the ocean from reaching the interior of Greenland using the GEBCO topography dataset.

• mache:

  • Fixed bugs with Perlmutter and Intel spack builds, updates the paths for ALCF Polaris, and updates spack builds on Compy and Chrysalis.

• MPAS-Analysis:

  • Added melt rate analysis from Paolo et al. (2023) and a new ocean conservation task, and fixes an out-of-memory bug in very long (>1000 year) sea-ice time series, as well as other bug fixes and clean up.

• MPAS-Tools:

  • Fixed building the grid_rotate tool and adds new functions for culling MPAS datasets to the conda package
  • Some clean up and bug fixes to the MPAS cell culler and jigsaw to NetCDF converter as well as land-ice tools.

• NCO:

  • The timeseries splitter in ncclimo can now automatically generate monthly filenames, and so can be invoked analogously to climo-generation with specified start/end dates, rather than via stdin
  • ncks can now generate gridded datasets from ELM initial condition and restart datasets. ncks --s1d rolls up 1D input landunit arrays into gridded output arrays with, e.g., PFT or MEC dimensions

• zppy:

  • Add native EAMxx support
  • Add lat_lon_land from e3sm_diags for land diagnostics
  • Remove waves and eke from MPAS-Analysis generate option
  • Add e3sm_to_cmip_environment_commands parameter

v1.9.2

In this version

• A few packages have been updated, including:
  • e3sm_diags 2.10.1
  • e3sm_to_cmip 1.11.1
  • ilamb 2.7
  • mache 1.18.0
  • mpas-analysis 1.9.1
  • mpas_tools 0.30.0
  • xcdat 0.6.1
  • zstash 1.4.2

New and improved

• E3SM-Diags:

  • 2.10.0
    • Add more aerosol metrics contributed by Aerosol Working Group.
  • 2.10.1
    • Fix bug introduced in 2.10.0 UnboundLocalError for params_results and type conditional check
    • Fix bug introduced in 2.10.0 default diagnostics attrs not being copied to parent CoreParameter

• e3sm_to_cmip:

  • 1.11.0
    • Refactor significant portions of the codebase and refactor some land-atm handlers, including replacing CDAT dependencies with Xarray
    • Add lat and lon bounds if missing in input datasets
  • 1.11.1
    • Fix segmentation fault introduced in 1.11.0 due to early cmor.close() call that wipes cmor file ID mapping after every cmor.write() call

• ILAMB:

  • v2.7 has many bug fixes and enhancements. Many datasets are updated and added. See Release notes for details.

• mache:

  • New diagnostics paths on Andes following removal of Alpine file system
  • Improved host discovery

• MPAS-Analysis:

  • A bug fix to stats printed on global climatology maps

• MPAS-Tools:

  • Bug fix to limit lon to -180 to 180 in lon-lat region masks

• xcdat:

  • 0.6.0

    • New vertical regridding API (extension of xgcm), functions for producing accurate time bounds, and improving the usability of the create_grid API.
    • Bug fixes to preserve attributes when using regrid2 horizontal regridder and fixing multi-file datasets spatial average orientation and weights when lon bounds span prime meridian

  • 0.6.1

    • Adds a default value to the axes argument in ds.bounds.add_missing_bounds() (axes=["X", "Y", "T"]). The axes argument was added in v0.6.0 and did not have a default value, which inadvertently introduced a breaking change to the API.
    • xesmf is now a required dependency because its core library, ESMF, supports Windows as of Feb/2023. More information can be found here.

• zstash:

  • The LCRC endpoint aliases in zstash now use the newest LCRC endpoint.

v1.9.1

In this version

• 5 packages built with Spack for use on compute nodes:
  • esmf 8.4.2
  • moab 5.5.1
  • nco 5.1.9
  • tempestextremes 2.2.1
  • tempestremap 2.2.0
• A few packages have been updated. In addition to moab and nco listed above, notable updates include:
  • mpas_tools 0.27.0
  • zppy 2.3.1
  • zstash 1.4.1

New and improved

• MOAB:

  • New ESMF format NC reader support
  • Improved support for reading MPAS files directly
  • Improved API documentation now available
  • Fix several outstanding issues in mbtempest tool
    • Propagate grid_dims correctly for RLL grids
    • Store history metadata as attributes for provenance
  • A new halo-exchange example to measure performance of nearest-neighbor halo-exchange communication kernels
  • Several configuration and build updates for running on ALCF and OLCF systems

• NCO:

  • See full release notes at https://github.com/nco/nco/blob/5.1.9/doc/ANNOUNCE
  • ncremap now, by default, fills with 0.0 (instead of FillValue) gridcells where sub-gridscale fraction is zero. Previous behavior (filling with FillValue) can be restored with new --mpt_mss option.
  • ncremap supports new TR bilinear and integrated bilinear algorithm types
  • ncremap supports new standard names for old algorithms, e.g., esmfaave, traave, ncoaave
  • ncra, ncrcat fix behavior of dimension interleave (ILV) parameter NCO 5.1.9 User Guide
  • All operator fix possible path separator issue in MS Windows
  • Improved inferral of MALI grids (radians rather than degrees automatically detected)
  • Support for attributes stored as NC_STRING rather than NC_CHAR

• zppy: A new parameter keep_mvm_case_name_in_fig enables turning on and off ref_name in e3sm_diags figure names in model vs model case. The current IICE comparison viewer will work seamlessly again with e3sm_diags output generated with keep_mvm_case_name_in_fig=False for model vs model runs. Additionally, the default set order for E3SM Diags has been returned to its original order, and the machine name can now be discovered on compute nodes while using E3SM Unified.

• zstash: Globus endpoints are beginning to require extra consents. This latest version of zstash addresses this issue by automatically prompting the user for authentication credentials to provide these consents, if required.

v1.9.0

In this version

• 5 packages built with Spack for use on compute nodes:
  • esmf 8.4.2
  • moab 5.5.0
  • nco 5.1.7
  • tempestextremes 2.2.1
  • tempestremap 2.2.0
• Many packages have been updated. In addition to esmf, moab, nco and tempestremap listed above, notable updates include:
  • chemdyg 0.1.4
  • e3sm_diags 2.9.0
  • e3sm_to_cmip 1.10.0
  • geometric_features 1.2.0
  • mache 1.17.0
  • mpas-analysis 1.9.0
  • mpas_tools 0.21.0
  • xcdat 0.5.0
  • zppy 2.3.0
  • zstash 1.4.0

New and improved

• ChemDyg: This new package provides diagnostics for the interactive gas-phase chemistry, newly available in E3SM, to support routine testing and evaluation of the chemistry performance.

• E3SM Diags: This new release adds one new set: mixed phase partition diagnostics (mp_partition), also known as T5050 diagnostics. The ARM diagnostics (arm_diags) set is enhanced with new aerosol-cloud-interaction and aerosol activation metrics developed by ARM infrastructure team. The aerosol budget diagnostics (aerosol_budget) is updated by including global mean burdens, source/sink budgets and lifetimes for aerosol species contributed by Aerosol Working Group. GPCP v3.2 precipitation data is added to replace an older version (v2.2). Surface relative humidity and wind speed data set derived from ERA5 are added as standard evaluation sets.

• e3sm_to_cmip: The latest release includes a major revamp in the implementation of land and atm variable handlers. Handlers are now defined a single yaml file rather than as individual Python modules. More information can be found in the Variable Handlers documentation page. Other improvements include better error handling, a smoother development experience and updated documentation and examples.

• MPAS-Analysis: This release adds new waves analysis and switches from the older Rignot et al. (2013) to the newer Adusumilli et al. (2020) for Antarctic melt-rate observations as well as numerous minor enhancements and bug fixes.

• MOAB: Significant improvements in both robustness and performance for parallel, conservative map generation, and extensions to support bilinear map projections through TempestRemap v2.2.0.

• ncvis: A newly developed graphical tool that supports visualizing NetCDF data on unstructured grid.

• NCO: Simplified invocation of quantization and codecs. CF-compliant longitude boundaries. Fixes for Intel compilers. NCZarr features. Vertical interpolation improvements. EAMxx support.

• xcdat: The latest release includes feature updates to support an optional user-specified climatology reference period when calculating climatologies and departures, and improved support for using custom time coordinates in temporal APIs. It also includes a bug fix for singleton coordinates breaking the swap_lon_axis() function. Additionally, Jupyter Notebooks for presentations and demos have been added to the documentation.

• zppy: This release introduces plugin functionality – users can now create plugins to add custom tasks to zppy. Additionally, users can now 1) choose to generate all variables in climo or ts by setting vars="", 2) directly choose the value for the ncclimo --parallel option, and 3) generate hemispheric time-series plots.

• zstash: This release adds a couple new options. The --include option allows users to specify which files to include. The --follow-symlinks option allows users to copy symlinks, which is helpful in preventing broken links.

v1.7.0

In this version

• 5 packages built with Spack for use on compute nodes:
  • esmf 8.2.0
  • moab 5.3.1
  • nco 5.1.0
  • tempestextremes 2.2.1
  • tempestremap 2.1.1
• Many packages have been updated. In addition to esmf, nco and tempestextremes listed above, notable updates include:
  • e3sm_diags 2.7,9
  • e3sm_to_cmip 1.8.2
  • geometric_features 0.6.0
  • livvkit 3.0.1
  • mache 1.5.0
  • mpas-analysis 1.7.0
  • mpas_tools 0.14.0
  • zstash 1.2.1
  • zppy 2.1.0
• Newly added:
  • pcmdi_metrics 2.3.1
  • xcdat 0.3.0

New and improved

• E3SM Diags: Add cmip6 comparison box whisker plot and updated Taylor Diagram with (E3SM v1 v2 amip and historical runs as baseline) in standard lat_lon set; Addded AOD and AAOD comparison with AERONET measurements; Support ELM and MOSART variables in model vs model comparison; Support model-only diags when reference is not present (i.g. AODDUST); Replace ERA-Interim with ERA5; Added MACv2 for AOD obs; Fixed diurnal cycle phase plot. More updates can be found here.
• MPAS-Analysis: New density contour plots for validating eddy parameterizations; A smarter config parser that preserves comments and provenance in the output directory. More updates can be found here.
• NCO: NCO now supports all new compression features offered by the latest netCDF, 4.9.0, including transparent access to modern lossless (Bzip2, Zstandard) and lossy (BitRound, Granular BitRound) compression and quantization algorithms. Documentation is here. Regridding landunit-specific fields in ELM is fixed.
• pcmdi_metrics (PMP): The PCMDI Metrics Package (PMP) provides multi-model comparison for mean physical climate, climate variability metrics. More information can be found here.
• xcdat: xcdat (Xarray Climate Data Analysis Tools) is an extension of xarray for climate data analysis on structured grids. The features in xcdat v0.3.0 include: wrapper for xarray’s open_dataset() and open_mfdataset() with post-processing options, temporal averaging (time series averages including single snapshot and grouped, climatologies, departures), geospatial weighted averaging, and horizontal structured regridding. Documentation is here.
• zppy: Support running ILAMB tasks, which included writing out CMIP formated time-series using e3sm_to_cmip as a preparation step; Added support to model vs model E3SM Diags runs; Add option to define last year. More updates can be found here.

v1.6.0

This update includes:

• 5 packages built with Spack for use on compute nodes:
  • esmf 8.1.1
  • moab 5.3.1
  • nco 5.0.6
  • empestextremes 2.2
  • tempestremap 2.1.1
• Many packages have been updated. In addition to those listed above, notable updates include:
  • e3sm_diags 2.6.1
  • e3sm_to_cmip 1.8.1
  • mache 1.2.0
  • mpas-analysis 1.6.1
  • mpas_tools 0.12.0
  • zstash 1.2.0
  • zppy 2.0.0

The mache package is now used to keep track of many of the details specific to each E3SM supported machine.

v1.5.1

Version updates to main packages:

• e3sm_diags 2.6.0
• e3sm_to_cmip 1.8.0
• mache 1.1.2
• mpas-analysis 1.5.0
• mpas_tools 0.11.0
• nco 5.0.3
• zppy 1.1.0

The mache package is now used to keep track of many of the details specific to each E3SM supported machine.

As in the previous release, in the environment loaded on compute nodes, the following packages are built with system MPI and compilers:

• ESMF
• mpi4py
• ILAMB
• TempestExtremes