Skip to content

Conversation

@tomvothecoder
Copy link
Collaborator

@tomvothecoder tomvothecoder commented Jun 10, 2025

Description

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • My changes generate no new warnings
  • Any dependent changes have been merged and published in downstream modules

If applicable:

  • New and existing unit tests pass with my changes (locally and CI/CD build)
  • I have added tests that prove my fix is effective or that my feature works
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

@chengzhuzhang
Copy link
Contributor

chengzhuzhang commented Jun 10, 2025

@tomvothecoder thanks for working on this. For reference, my current run with pinning python and several packages following e3sm_unified seems to be promissing, here is the basic packages in my test env (changes in bold):

  **- python =3.10**
  - pip
  - setuptools
  - beautifulsoup4
  - cartopy >=0.17.0
  - cartopy_offlinedata
  - cf-units
  **- dask =2024.11.2**
  - lxml
  - mache >=0.15.0
  - matplotlib-base >=3.8.2
  - netcdf4
  - output_viewer >=1.3.0
  - numpy >=2.0.0,<3.0.0
  - pywavelets
  - scipy
  **- xarray 2025.1.1**
  - xcdat >=0.7.3,<1.0.0
  **- xesmf =0.8.8**
  - xskillscore >=0.0.20
  **- esmf 8.8.0**
  **- esmpy 8.8.0**

@chengzhuzhang
Copy link
Contributor

I spoke too soon. The run with the new env as commended #986 (comment) still exceed 4 hour time limit. I haven't pinned numpy, maybe that is the source if the slowness, but in any case, I will try to work on a smaller example today.

@tomvothecoder
Copy link
Collaborator Author

@tomvothecoder thanks for working on this. For reference, my current run with pinning python and several packages following e3sm_unified seems to be promissing, here is the basic packages in my test env (changes in bold):

  **- python =3.10**
  - pip
  - setuptools
  - beautifulsoup4
  - cartopy >=0.17.0
  - cartopy_offlinedata
  - cf-units
  **- dask =2024.11.2**
  - lxml
  - mache >=0.15.0
  - matplotlib-base >=3.8.2
  - netcdf4
  - output_viewer >=1.3.0
  - numpy >=2.0.0,<3.0.0
  - pywavelets
  - scipy
  **- xarray 2025.1.1**
  - xcdat >=0.7.3,<1.0.0
  **- xesmf =0.8.8**
  - xskillscore >=0.0.20
  **- esmf 8.8.0**
  **- esmpy 8.8.0**

Thanks Jill. Here is the diff between E3SM Unified and the latest dev environment.

Maybe we can rule out Python version differences being an issue. I also recall that the dev environment used to be Python >= 3.9 (now 3.10), which meant our dev environments would often build with Python 3.12 and potentially 3.13.

Based on #911, we should focus on dask first.

Dependency e3sm_diags_dev_985 e3sm_unified
python 3.13.3 3.10.16
numpy 2.2.6 2.2.4
dask 2025.5.1 2024.11.2
xarray 2025.4.0 2025.1.1
esmf 8.8.1 8.8.0
xesmf 0.8.10 0.8.8

@chengzhuzhang
Copy link
Contributor

@tomvothecoder I added a minimum example. To run on chrysalis:

  1. get an interactive compute node
    srun -N 1 -t 01:00:00 --pty bash
  2. activate latest e3sm-unified or the latest e3sm_diags env:
  3. run with:
    time srun python e3sm_diags_extended.py -d core_set.cfg

in e3sm-unified, the run finished in 2.5 mins, it hangs in the new e3sm_diags env.

@tomvothecoder
Copy link
Collaborator Author

@tomvothecoder I added a minimum example. To run on chrysalis:

1. get an interactive compute node
   srun -N 1 -t 01:00:00 --pty bash

2. activate latest e3sm-unified or the latest e3sm_diags env:

3. run with:
   time srun python e3sm_diags_extended.py -d core_set.cfg

in e3sm-unified, the run finished in 2.5 mins, it hangs in the new e3sm_diags env.

Thanks @chengzhuzhang. I will investigate today.

@tomvothecoder tomvothecoder self-assigned this Jun 12, 2025
@tomvothecoder tomvothecoder added the bug Bug fix (will increment patch version) label Jun 12, 2025
@tomvothecoder
Copy link
Collaborator Author

tomvothecoder commented Jun 12, 2025

@tomvothecoder I added a minimum example. To run on chrysalis:

1. get an interactive compute node
   srun -N 1 -t 01:00:00 --pty bash

2. activate latest e3sm-unified or the latest e3sm_diags env:

3. run with:
   time srun python e3sm_diags_extended.py -d core_set.cfg

in e3sm-unified, the run finished in 2.5 mins, it hangs in the new e3sm_diags

Note: The LCRC filesystem is experiencing issues this morning, which is affecting how efficient I can debug (dropped SSH connections, really laggy).

This script runs successfully on the login node. On the compute node, the run stalls and I noticed slurmstepd: error: mpi/pmi2: invalid pmi1 command received: 'init' during regridding (only shows in console output, not log file).

I will pin xesmf/esmf/esmpy to the E3SM Unified version and try running again on the compute node. [UPDATE: Same results as the latest versions]

1. Login Node (latest dependencies)

Result -- Success

This script successfully completed in ~2 minutes with the latest e3sm_diags dev env on the login node.

There is no slurmstepd: error: mpi/pmi2: invalid pmi1 command received: 'init' in the log file, which makes sense since the login node does not use slurm/MPI/mpi2.

Log file: https://web.lcrc.anl.gov/public/e3sm/cdat-migration-fy24/985-perf-degrade/prov/e3sm_diags_run.log
Core dependencies:

python                    3.13.3          hf636f53_101_cp313    conda-forge
dask                      2025.5.1           pyhd8ed1ab_0    conda-forge
numpy                     2.2.6           py313h17eae1a_0    conda-forge
pandas                    2.3.0           py313ha87cce1_0    conda-forge
xesmf                     0.8.10             pyhd8ed1ab_0    conda-forge
esmf                      8.8.1           mpi_mpich_h10ba997_101    conda-forge
esmpy                     8.8.1              pyhec

2. Compute Node (latest dependencies)

Result -- Stalls with Slurm/MPI error

The run stalls and there is an slurmstepd error related to MPI (not captured in the log file, only in the console log) that happens during regridding (subset_and_align_datasets()).

2025-06-12 12:52:22,733 [INFO]: regrid.py(subset_and_align_datasets:69) >> Selected region: global
slurmstepd: error: mpi/pmi2: invalid pmi1 command received: 'init'

Full console log:

================================================================================
E3SM Diagnostics Run
--------------------
Timestamp: 2025-06-12 12:52:15
Version Info: version v3.0.0
Results Path: /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node
Log Path: /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/prov/e3sm_diags_run.log
Parameter Files Path: /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/prov/core_set.cfg
Python Script Path: /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/prov/e3sm_diags_extended.py
Environment YML Path: /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/prov/environment.yml
Provenance Index HTML Path: /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/prov/index.html
================================================================================

2025-06-12 12:52:20,660 [INFO]: lat_lon_driver.py(run_diag:69) >> Variable: PRECT
2025-06-12 12:52:20,661 [INFO]: zonal_mean_2d_driver.py(run_diag:42) >> Variable: T
2025-06-12 12:52:21,478 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the test climatology variable using the source variables: ('T',)
2025-06-12 12:52:21,519 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the test climatology variable using the source variables: ('PRECC', 'PRECL')
2025-06-12 12:52:21,719 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the ref climatology variable using the source variables: ('sat_gauge_precip',)
2025-06-12 12:52:21,847 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the ref climatology variable using the source variables: ('ta',)
2025-06-12 12:52:22,053 [INFO]: zonal_mean_2d_driver.py(_run_diags_3d:87) >> Selected pressure level: [50.0, 100.0, 150.0, 200.0, 250.0, 300.0, 350.0, 400.0, 450.0, 500.0, 550.0, 600.0, 650.0, 700.0, 750.0, 800.0, 850.0, 900.0, 950.0, 1000.0]
2025-06-12 12:52:22,143 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-06-12 12:52:22,414 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-06-12 12:52:22,733 [INFO]: regrid.py(subset_and_align_datasets:69) >> Selected region: global
slurmstepd: error: mpi/pmi2: invalid pmi1 command received: 'init'
2025-06-12 12:52:30,931 [INFO]: io.py(_save_data_metrics_and_plots:85) >> Metrics saved in /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/zonal_mean_2d/MERRA2/MERRA2-T-SON-global.json
2025-06-12 12:52:31,717 [INFO]: utils.py(_save_plot:92) >> Plot saved in: /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/zonal_mean_2d/MERRA2/MERRA2-T-SON-global.png
2025-06-12 12:52:31,718 [INFO]: zonal_mean_2d_driver.py(run_diag:42) >> Variable: T
2025-06-12 12:52:32,320 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the test climatology variable using the source variables: ('T',)
2025-06-12 12:52:32,551 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the ref climatology variable using the source variables: ('ta',)
2025-06-12 12:52:32,760 [INFO]: zonal_mean_2d_driver.py(_run_diags_3d:87) >> Selected pressure level: [50.0, 100.0, 150.0, 200.0, 250.0, 300.0, 350.0, 400.0, 450.0, 500.0, 550.0, 600.0, 650.0, 700.0, 750.0, 800.0, 850.0, 900.0, 950.0, 1000.0]
2025-06-12 12:52:32,847 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-06-12 12:52:33,114 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-06-12 12:52:40,788 [INFO]: io.py(_save_data_metrics_and_plots:85) >> Metrics saved in /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/zonal_mean_2d/MERRA2/MERRA2-T-JJA-global.json
2025-06-12 12:52:41,252 [INFO]: utils.py(_save_plot:92) >> Plot saved in: /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/zonal_mean_2d/MERRA2/MERRA2-T-JJA-global.png
2025-06-12 12:52:41,252 [INFO]: zonal_mean_2d_driver.py(run_diag:42) >> Variable: T
2025-06-12 12:52:41,902 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the test climatology variable using the source variables: ('T',)
2025-06-12 12:52:42,230 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the ref climatology variable using the source variables: ('ta',)
2025-06-12 12:52:42,442 [INFO]: zonal_mean_2d_driver.py(_run_diags_3d:87) >> Selected pressure level: [50.0, 100.0, 150.0, 200.0, 250.0, 300.0, 350.0, 400.0, 450.0, 500.0, 550.0, 600.0, 650.0, 700.0, 750.0, 800.0, 850.0, 900.0, 950.0, 1000.0]
2025-06-12 12:52:42,529 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-06-12 12:52:42,798 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-06-12 12:52:50,189 [INFO]: io.py(_save_data_metrics_and_plots:85) >> Metrics saved in /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/zonal_mean_2d/MERRA2/MERRA2-T-MAM-global.json
2025-06-12 12:52:50,652 [INFO]: utils.py(_save_plot:92) >> Plot saved in: /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/zonal_mean_2d/MERRA2/MERRA2-T-MAM-global.png
2025-06-12 12:52:50,653 [INFO]: zonal_mean_2d_driver.py(run_diag:42) >> Variable: T
2025-06-12 12:52:51,478 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the test climatology variable using the source variables: ('T',)
2025-06-12 12:52:51,789 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the ref climatology variable using the source variables: ('ta',)
2025-06-12 12:52:51,998 [INFO]: zonal_mean_2d_driver.py(_run_diags_3d:87) >> Selected pressure level: [50.0, 100.0, 150.0, 200.0, 250.0, 300.0, 350.0, 400.0, 450.0, 500.0, 550.0, 600.0, 650.0, 700.0, 750.0, 800.0, 850.0, 900.0, 950.0, 1000.0]
2025-06-12 12:52:52,085 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-06-12 12:52:52,346 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-06-12 12:52:59,732 [INFO]: io.py(_save_data_metrics_and_plots:85) >> Metrics saved in /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/zonal_mean_2d/MERRA2/MERRA2-T-DJF-global.json
2025-06-12 12:53:00,194 [INFO]: utils.py(_save_plot:92) >> Plot saved in: /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/zonal_mean_2d/MERRA2/MERRA2-T-DJF-global.png
2025-06-12 12:53:00,195 [INFO]: zonal_mean_2d_driver.py(run_diag:42) >> Variable: T
2025-06-12 12:53:00,862 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the test climatology variable using the source variables: ('T',)
2025-06-12 12:53:01,239 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the ref climatology variable using the source variables: ('ta',)
2025-06-12 12:53:01,655 [INFO]: zonal_mean_2d_driver.py(_run_diags_3d:87) >> Selected pressure level: [50.0, 100.0, 150.0, 200.0, 250.0, 300.0, 350.0, 400.0, 450.0, 500.0, 550.0, 600.0, 650.0, 700.0, 750.0, 800.0, 850.0, 900.0, 950.0, 1000.0]
2025-06-12 12:53:01,739 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-06-12 12:53:02,007 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/xgcm/grid.py:989: FutureWarning: From version 0.8.0 the Axis computation methods will be removed, in favour of using the Grid computation methods instead. i.e. use `Grid.transform` instead of `Axis.transform`
  warnings.warn(

2025-06-12 12:53:09,569 [INFO]: io.py(_save_data_metrics_and_plots:85) >> Metrics saved in /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/zonal_mean_2d/MERRA2/MERRA2-T-ANN-global.json
2025-06-12 12:53:10,030 [INFO]: utils.py(_save_plot:92) >> Plot saved in: /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/zonal_mean_2d/MERRA2/MERRA2-T-ANN-global.png
2025-06-12 12:53:10,032 [INFO]: lat_lon_driver.py(run_diag:69) >> Variable: PRECT
2025-06-12 12:53:10,576 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the test climatology variable using the source variables: ('PRECC', 'PRECL')
2025-06-12 12:53:10,691 [INFO]: dataset_xr.py(_get_dataset_with_derived_climo_var:714) >> Deriving the ref climatology variable using the source variables: ('sat_gauge_precip',)
2025-06-12 12:53:11,746 [INFO]: regrid.py(subset_and_align_datasets:69) >> Selected region: global
2025-06-12 12:53:20,061 [INFO]: io.py(_save_data_metrics_and_plots:85) >> Metrics saved in /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/lat_lon/GPCP_v3.2/GPCP_v3.2-PRECT-SON-global.json
2025-06-12 12:53:20,092 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/cartopy/mpl/geoaxes.py:856: UserWarning: Attempting to set identical low and high xlims makes transformation singular; automatically expanding.
  self.set_xlim([x1, x2])

2025-06-12 12:53:21,145 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/cartopy/mpl/geoaxes.py:856: UserWarning: Attempting to set identical low and high xlims makes transformation singular; automatically expanding.
  self.set_xlim([x1, x2])

2025-06-12 12:53:25,282 [WARNING]: warnings.py(_showwarnmsg:110) >> /home/ac.tvo/miniforge3/envs/e3sm_diags_dev_985/lib/python3.13/site-packages/cartopy/mpl/geoaxes.py:856: UserWarning: Attempting to set identical low and high xlims makes transformation singular; automatically expanding.
  self.set_xlim([x1, x2])

2025-06-12 12:53:28,662 [INFO]: utils.py(_save_plot:92) >> Plot saved in: /lcrc/group/e3sm/public_html/cdat-migration-fy24/985-perf-degrade-compute-node/lat_lon/GPCP_v3.2/GPCP_v3.2-PRECT-SON-global.png

Log file: https://web.lcrc.anl.gov/public/e3sm/cdat-migration-fy24/985-perf-degrade-compute-node/prov/e3sm_diags_run.log
Core dependencies:

python                    3.13.3          hf636f53_101_cp313    conda-forge
dask                      2025.5.1           pyhd8ed1ab_0    conda-forge
numpy                     2.2.6           py313h17eae1a_0    conda-forge
pandas                    2.3.0           py313ha87cce1_0    conda-forge
xesmf                     0.8.10             pyhd8ed1ab_0    conda-forge
esmf                      8.8.1           mpi_mpich_h10ba997_101    conda-forge
esmpy                     8.8.1              pyhec

3. Compute Node (pinned xESMF/esmpy/ESMF to E3SM Unified versions)

Results -- Same result as 2, stalls with MPI init error

Core dependencies:

python                    3.13.5          hf636f53_100_cp313    conda-forge
dask                      2025.5.1           pyhd8ed1ab_0    conda-forge
numpy                     2.2.6           py313h17eae1a_0    conda-forge
pandas                    2.3.0           py313ha87cce1_0    conda-forge
xesmf                     0.8.8              pyhd8ed1ab_1    conda-forge
esmf                      8.8.0           mpi_mpich_h7cf99a1_100    conda-forge
esmpy                     8.8.0              pyhecae5ae_1    conda-forge

4. Compute Node (latest dependencies, pinned esmf >=8.8.0=nompi_*

Result -- successful

Core dependencies:

python                    3.13.5          hf636f53_100_cp313    conda-forge
dask                      2025.5.1           pyhd8ed1ab_0        conda-forge
numpy                     2.2.6           py313h17eae1a_0       conda-forge
pandas                    2.3.0           py313ha87cce1_0       conda-forge
xesmf                     0.8.10             pyhd8ed1ab_0        conda-forge
esmf                      8.8.1           nompi_hc4cf7a7_1      conda-forge
esmpy                     8.8.1              pyhecae5ae_0        conda-forge

Log file: https://web.lcrc.anl.gov/public/e3sm/cdat-migration-fy24/985-perf-degrade-compute-node-esmf-nompi/prov/e3sm_diags_run.log

Summary of Results

The mpi version of esmf is being installed with e3sm_diags dev/ci environments, which results in the MPI init error on compute nodes.

We need the nompi version of esmf (same as E3SM Unified) in order to run e3sm_diags on compute nodes.

@chengzhuzhang
Copy link
Contributor

chengzhuzhang commented Jun 12, 2025

[UPDATE], I did see error: mpi/pmi2: invalid pmi1 command received: 'init', and hang at the same place as yours: 2025-06-11 21:33:35,408 [INFO]: utils.py(_save_plot:92) >> Plot saved in: /lcrc/group/e3sm/public_html/diagnostic_output/ac.zhang40/tests/985-perf-degrade/lat_lon/GPCP_v3.2/GPCP_v3.2-PRECT-SON-global.png
@tomvothecoder have you used srun ? which helped me to get round the error: mpi/pmi2: invalid pmi1 command received: 'init'

@tomvothecoder
Copy link
Collaborator Author

tomvothecoder commented Jun 12, 2025

[UPDATE], I did see error: mpi/pmi2: invalid pmi1 command received: 'init', and hang at the same place as yours: 2025-06-11 21:33:35,408 [INFO]: utils.py(_save_plot:92) >> Plot saved in: /lcrc/group/e3sm/public_html/diagnostic_output/ac.zhang40/tests/985-perf-degrade/lat_lon/GPCP_v3.2/GPCP_v3.2-PRECT-SON-global.png @tomvothecoder have you used srun ? which helped me to get round the error: mpi/pmi2: invalid pmi1 command received: 'init'

Updated comment above. Pinning xESMF/ESMF/esmpy to E3SM Unified versions still stalled with MPI init init error on compute node.

I'm going to try to pin dask and python <3.13.0 (separately) to see which one it might be.

@chengzhuzhang
Copy link
Contributor

@tomvothecoder could you also list the version and build for these packages in e3sm-unified on compute node?

@tomvothecoder
Copy link
Collaborator Author

tomvothecoder commented Jun 12, 2025

@tomvothecoder could you also list the version and build for these packages in e3sm-unified on compute node?

Ahh, looks like the esmf nompi version is installed in E3SM Unified. I recall @xylar doing some stuff to get specific conda-forge versions of esmf/esmpy versions installed to avoid issues with spack/mpi.

Maybe we need to constrain e3sm_diags conda envs with the same thing? The dev environment currently installs the MPI version: esmf 8.8.1 mpi_mpich_h10ba997_101 conda-forge. Going to try latest dependencies with nompi verison of esmf and see what happens (esmpy=*=nompi_*, from conda-forge/esmpy-feedstock#70 (comment)).

Core dependencies:

python                    3.10.16         habfa6aa_3_cpython    conda-forge
dask                      2024.11.2          pyhff2d567_1        conda-forge
numpy                     2.2.4           py310hefbff90_0       conda-forge
pandas                    2.2.3           py310h5eaa309_1       conda-forge
xesmf                     0.8.8              pyhd8ed1ab_1        conda-forge
esmf                      8.8.0           nompi_h4441c20_0      conda-forge
esmpy                     8.8.0              pyhecae5ae_1        conda-forge

Full dependencies:

(e3sm_diags_dev_985_dask_esmf_pinned) [ac.tvo@chr-0165 2-reproduce]$ source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
(e3sm_unified_1.11.1_chrysalis) [ac.tvo@chr-0165 2-reproduce]$ conda list
# packages in environment at /lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.11.1_chrysalis:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_r-mutex                  1.0.1               anacondar_1    conda-forge
adwaita-icon-theme        48.0                     unix_0    conda-forge
affine                    2.4.0              pyhd8ed1ab_1    conda-forge
algopy                    0.7.2              pyhd8ed1ab_1    conda-forge
alsa-lib                  1.2.13               hb9d3cd8_0    conda-forge
antimeridian              0.4.1              pyhd8ed1ab_0    conda-forge
anyio                     4.9.0              pyh29332c3_0    conda-forge
aom                       3.9.1                hac33072_0    conda-forge
argon2-cffi               23.1.0             pyhd8ed1ab_1    conda-forge
argon2-cffi-bindings      21.2.0          py310ha75aee5_5    conda-forge
arpack                    3.9.1           nompi_hf03ea27_102    conda-forge
arrow                     1.3.0              pyhd8ed1ab_1    conda-forge
asttokens                 3.0.0              pyhd8ed1ab_1    conda-forge
async-lru                 2.0.5              pyh29332c3_0    conda-forge
at-spi2-atk               2.38.0               h0630a04_3    conda-forge
at-spi2-core              2.40.3               h0630a04_0    conda-forge
atk-1.0                   2.38.0               h04ea711_2    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
attrs                     25.3.0             pyh71513ae_0    conda-forge
aws-c-auth                0.8.7                h7743f02_1    conda-forge
aws-c-cal                 0.8.7                h7d555fd_1    conda-forge
aws-c-common              0.12.1               hb9d3cd8_0    conda-forge
aws-c-compression         0.3.1                hcbd9e4e_3    conda-forge
aws-c-event-stream        0.5.4                h286e7e7_3    conda-forge
aws-c-http                0.9.5                hbca0721_0    conda-forge
aws-c-io                  0.17.0               ha855f32_8    conda-forge
aws-c-mqtt                0.12.2               hffac463_3    conda-forge
aws-c-s3                  0.7.13               h4c9fe3b_3    conda-forge
aws-c-sdkutils            0.2.3                hcbd9e4e_3    conda-forge
aws-checksums             0.2.3                hcbd9e4e_3    conda-forge
aws-crt-cpp               0.31.1               h46b750d_1    conda-forge
aws-sdk-cpp               1.11.510             h1fa5cb7_4    conda-forge
azure-core-cpp            1.14.0               h5cfcd09_0    conda-forge
azure-identity-cpp        1.10.0               h113e628_0    conda-forge
azure-storage-blobs-cpp   12.13.0              h3cf044e_1    conda-forge
azure-storage-common-cpp  12.8.0               h736e048_1    conda-forge
azure-storage-files-datalake-cpp 12.12.0              ha633028_1    conda-forge
babel                     2.17.0             pyhd8ed1ab_0    conda-forge
backrefs                  5.8                pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.13.3             pyha770c72_0    conda-forge
binutils_impl_linux-64    2.43                 h4bf12b8_4    conda-forge
blas                      2.131                  openblas    conda-forge
blas-devel                3.9.0           31_h1ea3ea9_openblas    conda-forge
bleach                    6.2.0              pyh29332c3_4    conda-forge
bleach-with-css           6.2.0                h82add2a_4    conda-forge
blosc                     1.21.6               he440d0b_1    conda-forge
bokeh                     3.7.2              pyhd8ed1ab_1    conda-forge
bottleneck                1.4.2           py310hf462985_0    conda-forge
branca                    0.8.1              pyhd8ed1ab_0    conda-forge
brotli                    1.1.0                hb9d3cd8_2    conda-forge
brotli-bin                1.1.0                hb9d3cd8_2    conda-forge
brotli-python             1.1.0           py310hf71b8c6_2    conda-forge
brunsli                   0.1                  h9c3ff4c_0    conda-forge
bwidget                   1.10.1               ha770c72_0    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.34.5               hb9d3cd8_0    conda-forge
c-blosc2                  2.15.2               h3122c55_1    conda-forge
ca-certificates           2025.1.31            hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cairo                     1.18.4               h3394656_0    conda-forge
cartopy                   0.24.0          py310h5eaa309_0    conda-forge
cartopy_offlinedata       0.24.0             pyhd8ed1ab_1    conda-forge
cdat_info                 8.2.1              pyhd8ed1ab_3    conda-forge
cdms2                     3.1.5           py310h4f3777b_25    conda-forge
cdo                       2.4.4                h0e6046d_2    conda-forge
cdp                       1.7.0              pyhd8ed1ab_2    conda-forge
cdtime                    3.1.4           py310h7a088e3_13    conda-forge
cdutil                    8.2.1              pyhd8ed1ab_5    conda-forge
certifi                   2025.1.31          pyhd8ed1ab_0    conda-forge
cf-units                  3.3.0           py310hf462985_0    conda-forge
cf_xarray                 0.10.4             pyhd8ed1ab_0    conda-forge
cffi                      1.17.1          py310h8deb56e_0    conda-forge
cftime                    1.6.4           py310hf462985_1    conda-forge
charls                    2.4.2                h59595ed_0    conda-forge
charset-normalizer        3.4.1              pyhd8ed1ab_0    conda-forge
chemdyg                   1.1.0              pyhd8ed1ab_0    conda-forge
cime_gen_domain           6.1.59               hd383692_0    conda-forge
click                     8.1.8              pyh707e725_0    conda-forge
click-plugins             1.1.1              pyhd8ed1ab_1    conda-forge
cligj                     0.7.2              pyhd8ed1ab_2    conda-forge
cloudpickle               3.1.1              pyhd8ed1ab_0    conda-forge
cmocean                   4.0.3              pyhd8ed1ab_1    conda-forge
cmor                      3.10.0          py310hdd51eb4_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_1    conda-forge
colorcet                  3.1.0              pyhd8ed1ab_1    conda-forge
colorspacious             1.1.2              pyhecae5ae_1    conda-forge
comm                      0.2.2              pyhd8ed1ab_1    conda-forge
configobj                 5.0.9              pyhd8ed1ab_1    conda-forge
contourpy                 1.3.1           py310h3788b33_0    conda-forge
cpython                   3.10.16         py310hd8ed1ab_3    conda-forge
cryptography              44.0.2          py310h6c63255_0    conda-forge
curl                      8.13.0               h332b0f4_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_1    conda-forge
cython                    3.0.12          py310had8cdd9_0    conda-forge
cytoolz                   1.0.1           py310ha75aee5_0    conda-forge
dask                      2024.11.2          pyhff2d567_1    conda-forge
dask-core                 2024.11.2          pyhff2d567_1    conda-forge
dask-expr                 1.1.19             pyhd8ed1ab_0    conda-forge
datashader                0.17.0             pyhd8ed1ab_0    conda-forge
dav1d                     1.2.1                hd590300_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
debugpy                   1.8.13          py310hf71b8c6_0    conda-forge
decorator                 5.2.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
distarray                 2.12.2             pyh3866b66_5    conda-forge
distributed               2024.11.2          pyhff2d567_1    conda-forge
dogpile.cache             1.3.3              pyhd8ed1ab_0    conda-forge
e3sm-tools                3.0.2                h9500345_0    conda-forge
e3sm-unified              1.11.1          hpc_py310_h03c7360_0    conda-forge
e3sm_diags                3.0.0              pyhd8ed1ab_0    conda-forge
e3sm_to_cmip              1.11.3             pyhd8ed1ab_1    conda-forge
eccodes                   2.40.0               h8bb6dbc_0    conda-forge
enso_metrics              1.1.5           py310hff52083_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_1    conda-forge
eofs                      2.0.0              pyhff2d567_0    conda-forge
epoxy                     1.5.10               h166bdaf_1    conda-forge
esmf                      8.8.0           nompi_h4441c20_0    conda-forge
esmpy                     8.8.0              pyhecae5ae_1    conda-forge
exceptiongroup            1.2.2              pyhd8ed1ab_1    conda-forge
executing                 2.1.0              pyhd8ed1ab_1    conda-forge
expat                     2.7.0                h5888daf_0    conda-forge
f90nml                    1.4.4              pyhd8ed1ab_1    conda-forge
fair-research-login       0.2.6              pyhd8ed1ab_0    conda-forge
ffmpeg                    7.1.1           gpl_h0b79d52_704    conda-forge
fftw                      3.3.10          nompi_hf1063bd_110    conda-forge
findlibs                  0.0.5              pyhd8ed1ab_0    conda-forge
flexcache                 0.3                pyhd8ed1ab_1    conda-forge
flexparser                0.4                pyhd8ed1ab_1    conda-forge
folium                    0.19.5             pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_3    conda-forge
fontconfig                2.15.0               h7e30c49_1    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.57.0          py310h89163eb_0    conda-forge
fqdn                      1.5.1              pyhd8ed1ab_1    conda-forge
freeglut                  3.2.2                ha6d2627_3    conda-forge
freetype                  2.13.3               h48d6fc4_0    conda-forge
freexl                    2.0.0                h9dce30a_2    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
fsspec                    2025.3.2           pyhd8ed1ab_0    conda-forge
future                    1.0.0              pyhd8ed1ab_2    conda-forge
g2clib                    2.1.0                ha770c72_0    conda-forge
gcc_impl_linux-64         14.2.0               hdb7739f_2    conda-forge
gdk-pixbuf                2.42.12              hb9ae30d_0    conda-forge
genutil                   8.2.1           py310h21aefe9_5    conda-forge
geometric_features        1.6.1              pyhd8ed1ab_0    conda-forge
geopandas                 1.0.1              pyhd8ed1ab_3    conda-forge
geopandas-base            1.0.1              pyha770c72_3    conda-forge
geos                      3.13.1               h97f6797_0    conda-forge
geotiff                   1.7.4                h3551947_0    conda-forge
geoviews                  1.14.0               hd8ed1ab_0    conda-forge
geoviews-core             1.14.0             pyha770c72_0    conda-forge
gettext                   0.23.1               h5888daf_0    conda-forge
gettext-tools             0.23.1               h5888daf_0    conda-forge
gflags                    2.2.2             h5888daf_1005    conda-forge
gfortran_impl_linux-64    14.2.0               h0ee6e4a_2    conda-forge
ghp-import                2.1.0              pyhd8ed1ab_2    conda-forge
giflib                    5.2.2                hd590300_0    conda-forge
glib                      2.84.1               h07242d1_0    conda-forge
glib-tools                2.84.1               h4833e2c_0    conda-forge
globus-cli                3.33.0             pyhd8ed1ab_0    conda-forge
globus-sdk                3.51.0             pyhd8ed1ab_0    conda-forge
glog                      0.7.1                hbabe93e_0    conda-forge
glpk                      5.0                  h445213a_0    conda-forge
gmp                       6.3.0                hac33072_2    conda-forge
gmpy2                     2.1.5           py310he8512ff_3    conda-forge
graphite2                 1.3.13            h59595ed_1003    conda-forge
gsl                       2.7                  he838d99_0    conda-forge
gst-plugins-base          1.24.7               h0a52356_0    conda-forge
gstreamer                 1.24.7               hf3bb09a_0    conda-forge
gsw                       3.6.19          py310hf462985_1    conda-forge
gtk3                      3.24.43              h0c6a113_5    conda-forge
gxx_impl_linux-64         14.2.0               h2ead766_2    conda-forge
h11                       0.14.0             pyhd8ed1ab_1    conda-forge
h2                        4.2.0              pyhd8ed1ab_0    conda-forge
h5netcdf                  1.6.1              pyhd8ed1ab_0    conda-forge
h5py                      3.13.0          nompi_py310h60e0fe6_100    conda-forge
harfbuzz                  11.0.1               h2c12942_0    conda-forge
hdf4                      4.2.15               h2a13503_7    conda-forge
hdf5                      1.14.3          nompi_h2d575fe_109    conda-forge
healpix                   2024.2          py310hf462985_2    conda-forge
hicolor-icon-theme        0.17                 ha770c72_2    conda-forge
holoviews                 1.20.2             pyhd8ed1ab_0    conda-forge
hpack                     4.1.0              pyhd8ed1ab_0    conda-forge
httpcore                  1.0.7              pyh29332c3_1    conda-forge
httpx                     0.28.1             pyhd8ed1ab_0    conda-forge
hvplot                    0.11.2             pyhd8ed1ab_0    conda-forge
hyperframe                6.1.0              pyhd8ed1ab_0    conda-forge
icu                       75.1                 he02047a_0    conda-forge
idna                      3.10               pyhd8ed1ab_1    conda-forge
ilamb                     2.7.2           py310hff52083_2    conda-forge
imagecodecs               2024.12.30      py310h78a9a29_0    conda-forge
imageio                   2.37.0             pyhfb79c49_0    conda-forge
importlib-metadata        8.6.1              pyha770c72_0    conda-forge
importlib_resources       6.5.2              pyhd8ed1ab_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_1    conda-forge
inpoly                    0.2.0           py310hf462985_8    conda-forge
ipygany                   0.5.0              pyhd8ed1ab_0    conda-forge
ipykernel                 6.29.5             pyh3099207_0    conda-forge
ipython                   8.35.0             pyh907856f_0    conda-forge
ipython_genutils          0.2.0              pyhd8ed1ab_2    conda-forge
ipywidgets                8.1.6              pyhd8ed1ab_0    conda-forge
isoduration               20.11.0            pyhd8ed1ab_1    conda-forge
jack                      1.9.22               h7c63dc7_2    conda-forge
jasper                    4.2.5                h1920b20_0    conda-forge
jedi                      0.19.2             pyhd8ed1ab_1    conda-forge
jigsaw                    0.9.14               hcb278e6_1    conda-forge
jigsawpy                  0.3.3              pyhd8ed1ab_3    conda-forge
jinja2                    3.1.6              pyhd8ed1ab_0    conda-forge
jmespath                  1.0.1              pyhd8ed1ab_1    conda-forge
joblib                    1.4.2              pyhd8ed1ab_1    conda-forge
json-c                    0.18                 h6688a6e_0    conda-forge
json5                     0.12.0             pyhd8ed1ab_0    conda-forge
json_tricks               3.11.0                     py_0    conda-forge
jsonpointer               3.0.0           py310hff52083_1    conda-forge
jsonschema                4.23.0             pyhd8ed1ab_1    conda-forge
jsonschema-specifications 2024.10.1          pyhd8ed1ab_1    conda-forge
jsonschema-with-format-nongpl 4.23.0               hd8ed1ab_1    conda-forge
jupyter                   1.1.1              pyhd8ed1ab_1    conda-forge
jupyter-lsp               2.2.5              pyhd8ed1ab_1    conda-forge
jupyter_client            7.4.9              pyhd8ed1ab_0    conda-forge
jupyter_console           6.6.3              pyhd8ed1ab_1    conda-forge
jupyter_core              5.7.2              pyh31011fe_1    conda-forge
jupyter_events            0.12.0             pyh29332c3_0    conda-forge
jupyter_server            2.15.0             pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.5.3              pyhd8ed1ab_1    conda-forge
jupyterlab                4.4.0              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.3.0              pyhd8ed1ab_2    conda-forge
jupyterlab_server         2.27.3             pyhd8ed1ab_1    conda-forge
jupyterlab_widgets        3.0.14             pyhd8ed1ab_0    conda-forge
jxrlib                    1.1                  hd590300_3    conda-forge
kernel-headers_linux-64   3.10.0              he073ed8_18    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.7           py310h3788b33_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
latexcodec                2.0.1              pyh9f0ad1d_0    conda-forge
lazy-loader               0.4                pyhd8ed1ab_2    conda-forge
lazy-object-proxy         1.10.0          py310ha75aee5_2    conda-forge
lazy_loader               0.4                pyhd8ed1ab_2    conda-forge
lcms2                     2.17                 h717163a_0    conda-forge
ld_impl_linux-64          2.43                 h712a8e2_4    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
level-zero                1.21.8               h84d6215_0    conda-forge
libabseil                 20250127.1      cxx17_hbbce691_0    conda-forge
libaec                    1.1.3                h59595ed_0    conda-forge
libarchive                3.7.7                h4585015_3    conda-forge
libarrow                  19.0.1           hb90904d_7_cpu    conda-forge
libarrow-acero            19.0.1           hcb10f89_7_cpu    conda-forge
libarrow-dataset          19.0.1           hcb10f89_7_cpu    conda-forge
libarrow-substrait        19.0.1           h1bed206_7_cpu    conda-forge
libasprintf               0.23.1               h8e693c7_0    conda-forge
libasprintf-devel         0.23.1               h8e693c7_0    conda-forge
libass                    0.17.3               h52826cd_2    conda-forge
libavif16                 1.2.1                hbb36593_2    conda-forge
libblas                   3.9.0           31_h59b9bed_openblas    conda-forge
libbrotlicommon           1.1.0                hb9d3cd8_2    conda-forge
libbrotlidec              1.1.0                hb9d3cd8_2    conda-forge
libbrotlienc              1.1.0                hb9d3cd8_2    conda-forge
libcap                    2.75                 h39aace5_0    conda-forge
libcblas                  3.9.0           31_he106b2a_openblas    conda-forge
libcdms                   3.1.2              hd841122_131    conda-forge
libcf                     1.0.3           py310h1588dd5_117    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcups                   2.3.3                h4637d8d_4    conda-forge
libcurl                   8.13.0               h332b0f4_0    conda-forge
libdb                     6.2.32               h9c3ff4c_0    conda-forge
libde265                  1.0.15               h00ab1b0_0    conda-forge
libdeflate                1.23                 h4ddbbb0_0    conda-forge
libdrm                    2.4.124              hb9d3cd8_0    conda-forge
libedit                   3.1.20250104    pl5321h7949ede_0    conda-forge
libegl                    1.7.0                ha4b6fd6_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.7.0                h5888daf_0    conda-forge
libffi                    3.4.6                h2dba641_1    conda-forge
libflac                   1.4.3                h59595ed_0    conda-forge
libgcc                    14.2.0               h767d61c_2    conda-forge
libgcc-devel_linux-64     14.2.0             h9c4974d_102    conda-forge
libgcc-ng                 14.2.0               h69a702a_2    conda-forge
libgcrypt-lib             1.11.0               hb9d3cd8_2    conda-forge
libgdal-core              3.10.2               h05269f4_1    conda-forge
libgettextpo              0.23.1               h5888daf_0    conda-forge
libgettextpo-devel        0.23.1               h5888daf_0    conda-forge
libgfortran               14.2.0               h69a702a_2    conda-forge
libgfortran-ng            14.2.0               h69a702a_2    conda-forge
libgfortran5              14.2.0               hf1ad2bd_2    conda-forge
libgl                     1.7.0                ha4b6fd6_2    conda-forge
libglib                   2.84.1               h2ff4ddf_0    conda-forge
libglu                    9.0.3                h03adeef_0    conda-forge
libglvnd                  1.7.0                ha4b6fd6_2    conda-forge
libglx                    1.7.0                ha4b6fd6_2    conda-forge
libgomp                   14.2.0               h767d61c_2    conda-forge
libgoogle-cloud           2.36.0               hc4361e1_1    conda-forge
libgoogle-cloud-storage   2.36.0               h0121fbd_1    conda-forge
libgpg-error              1.51                 hbd13f7d_1    conda-forge
libgrpc                   1.71.0               he753a82_0    conda-forge
libheif                   1.19.7          gpl_hc18d805_100    conda-forge
libhwloc                  2.11.2          default_h0d58e46_1001    conda-forge
libhwy                    1.1.0                h00ab1b0_0    conda-forge
libiconv                  1.18                 h4ce23a2_1    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
libjxl                    0.11.1               hdb8da77_0    conda-forge
libkml                    1.3.0             hf539b9f_1021    conda-forge
liblapack                 3.9.0           31_h7ac8fdf_openblas    conda-forge
liblapacke                3.9.0           31_he2f377e_openblas    conda-forge
libllvm20                 20.1.2               ha7bfdaf_0    conda-forge
liblzma                   5.8.1                hb9d3cd8_0    conda-forge
libnetcdf                 4.9.2           nompi_h00e09a9_116    conda-forge
libnghttp2                1.64.0               h161d5f1_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libogg                    1.3.5                h4ab18f5_0    conda-forge
libopenblas               0.3.29          pthreads_h94d23a6_0    conda-forge
libopentelemetry-cpp      1.20.0               hd1b1c89_0    conda-forge
libopentelemetry-cpp-headers 1.20.0               ha770c72_0    conda-forge
libopenvino               2025.0.0             hdc3f47d_3    conda-forge
libopenvino-auto-batch-plugin 2025.0.0             h4d9b6c2_3    conda-forge
libopenvino-auto-plugin   2025.0.0             h4d9b6c2_3    conda-forge
libopenvino-hetero-plugin 2025.0.0             h981d57b_3    conda-forge
libopenvino-intel-cpu-plugin 2025.0.0             hdc3f47d_3    conda-forge
libopenvino-intel-gpu-plugin 2025.0.0             hdc3f47d_3    conda-forge
libopenvino-intel-npu-plugin 2025.0.0             hdc3f47d_3    conda-forge
libopenvino-ir-frontend   2025.0.0             h981d57b_3    conda-forge
libopenvino-onnx-frontend 2025.0.0             h0e684df_3    conda-forge
libopenvino-paddle-frontend 2025.0.0             h0e684df_3    conda-forge
libopenvino-pytorch-frontend 2025.0.0             h5888daf_3    conda-forge
libopenvino-tensorflow-frontend 2025.0.0             h684f15b_3    conda-forge
libopenvino-tensorflow-lite-frontend 2025.0.0             h5888daf_3    conda-forge
libopus                   1.4                  hd0c01bc_0    conda-forge
libparquet                19.0.1           h081d1f1_7_cpu    conda-forge
libpciaccess              0.18                 hd590300_0    conda-forge
libpng                    1.6.47               h943b412_0    conda-forge
libprotobuf               5.29.3               h501fc15_0    conda-forge
libre2-11                 2024.07.02           hba17884_3    conda-forge
librsvg                   2.58.4               he92a37e_3    conda-forge
librttopo                 1.1.0               hd718a1a_18    conda-forge
libsanitizer              14.2.0               hed042b8_2    conda-forge
libsndfile                1.2.2                hc60ed4a_1    conda-forge
libsodium                 1.0.20               h4ab18f5_0    conda-forge
libspatialite             5.1.0               h366e088_13    conda-forge
libsqlite                 3.49.1               hee588c1_2    conda-forge
libssh2                   1.11.1               hf672d98_0    conda-forge
libstdcxx                 14.2.0               h8f9b012_2    conda-forge
libstdcxx-devel_linux-64  14.2.0             h9c4974d_102    conda-forge
libstdcxx-ng              14.2.0               h4852527_2    conda-forge
libsystemd0               257.4                h4e0b6ca_1    conda-forge
libthrift                 0.21.0               h0e7cc3e_0    conda-forge
libtiff                   4.7.0                hd9ff511_3    conda-forge
libudev1                  257.4                hbe16f8c_1    conda-forge
libudunits2               2.2.28               h40f5838_3    conda-forge
libunwind                 1.6.2                h9c3ff4c_0    conda-forge
liburing                  2.9                  h84d6215_0    conda-forge
libusb                    1.0.28               hb9d3cd8_0    conda-forge
libutf8proc               2.10.0               h4c51ac1_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libuv                     1.50.0               hb9d3cd8_0    conda-forge
libva                     2.22.0               h4f16b4b_2    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libvpx                    1.14.1               hac33072_0    conda-forge
libwebp-base              1.5.0                h851e524_0    conda-forge
libxcb                    1.17.0               h8a09558_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxkbcommon              1.8.1                hc4a0caf_0    conda-forge
libxml2                   2.13.7               h4bc477f_1    conda-forge
libxslt                   1.1.39               h76b75d6_0    conda-forge
libzip                    1.11.2               h6991a6a_0    conda-forge
libzlib                   1.3.1                hb9d3cd8_2    conda-forge
libzopfli                 1.0.3                h9c3ff4c_0    conda-forge
linkify-it-py             2.0.3              pyhd8ed1ab_1    conda-forge
livvkit                   3.1.0              pyhd8ed1ab_1    conda-forge
llvmlite                  0.44.0          py310h1a6248f_1    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lxml                      5.3.2           py310h490dddc_0    conda-forge
lz4                       4.3.3           py310h80b8a69_2    conda-forge
lz4-c                     1.10.0               h5888daf_1    conda-forge
lzo                       2.10              hd590300_1001    conda-forge
mache                     1.28.0             pyhd8ed1ab_0    conda-forge
magics                    4.15.5               hc87abea_0    conda-forge
magics-python             1.5.8              pyhd8ed1ab_1    conda-forge
make                      4.4.1                hb9d3cd8_2    conda-forge
mapclassify               2.8.1              pyhd8ed1ab_1    conda-forge
markdown                  3.6                pyhd8ed1ab_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_1    conda-forge
markdownlint-cli2         0.17.2               h92b4e83_0    conda-forge
markupsafe                3.0.2           py310h89163eb_1    conda-forge
matplotlib-base           3.9.4           py310h68603db_0    conda-forge
matplotlib-inline         0.1.7              pyhd8ed1ab_1    conda-forge
mdit-py-plugins           0.4.2              pyhd8ed1ab_1    conda-forge
mdurl                     0.1.2              pyhd8ed1ab_1    conda-forge
mdutils                   1.6.0              pyhd8ed1ab_1    conda-forge
mergedeep                 1.3.4              pyhd8ed1ab_1    conda-forge
mesalib                   25.0.3               he8ffa69_0    conda-forge
metpy                     1.6.3              pyhd8ed1ab_1    conda-forge
minizip                   4.0.9                h05a5f5f_0    conda-forge
mistune                   3.1.3              pyh29332c3_0    conda-forge
mkdocs                    1.6.1              pyhd8ed1ab_1    conda-forge
mkdocs-bibtex             2.18.0             pyhd8ed1ab_0    conda-forge
mkdocs-get-deps           0.2.0              pyhd8ed1ab_1    conda-forge
mkdocs-material           9.6.11             pyhd8ed1ab_0    conda-forge
mkdocs-material-extensions 1.3.1              pyhd8ed1ab_1    conda-forge
mkdocs-monorepo-plugin    1.1.0              pyhd8ed1ab_1    conda-forge
mkdocs-redirects          1.2.1              pyhd8ed1ab_1    conda-forge
mosaic                    1.1.0              pyhd8ed1ab_0    conda-forge
mpas-analysis             1.13.0             pyhd8ed1ab_0    conda-forge
mpas_tools                0.36.0          py310h2ba5516_2    conda-forge
mpc                       1.3.1                h24ddda3_1    conda-forge
mpfr                      4.2.1                h90cbb55_3    conda-forge
mpg123                    1.32.9               hc50e24c_0    conda-forge
mpi4py                    4.0.1                    pypi_0    pypi
mpmath                    1.3.0              pyhd8ed1ab_1    conda-forge
msgpack-python            1.1.0           py310h3788b33_0    conda-forge
multipledispatch          0.6.0              pyhd8ed1ab_1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
narwhals                  1.34.1             pyhd8ed1ab_0    conda-forge
nb_conda                  2.2.1              pyh707e725_7    conda-forge
nb_conda_kernels          2.5.1              pyh707e725_2    conda-forge
nbclassic                 1.2.0              pyhd8ed1ab_0    conda-forge
nbclient                  0.10.2             pyhd8ed1ab_0    conda-forge
nbconvert-core            7.16.6             pyh29332c3_0    conda-forge
nbformat                  5.10.4             pyhd8ed1ab_1    conda-forge
nc-time-axis              1.4.1              pyhd8ed1ab_1    conda-forge
nceplibs-g2c              2.1.0                hcdb72e9_0    conda-forge
ncurses                   6.5                  h2d0b736_3    conda-forge
ncview                    2.1.8               h31cb22a_11    conda-forge
ncvis-climate             2024.01.26           h6680449_2    conda-forge
nest-asyncio              1.6.0              pyhd8ed1ab_1    conda-forge
netcdf-fortran            4.6.1           nompi_h22f9119_108    conda-forge
netcdf4                   1.7.2           nompi_py310h9f0ad05_101    conda-forge
networkx                  3.4.2              pyh267e887_2    conda-forge
nlohmann_json             3.11.3               he02047a_1    conda-forge
nodejs                    22.13.0              hf235a45_0    conda-forge
notebook                  6.5.7              pyha770c72_0    conda-forge
notebook-shim             0.2.4              pyhd8ed1ab_1    conda-forge
numba                     0.61.2          py310h699fe88_0    conda-forge
numdifftools              0.9.41             pyhd8ed1ab_1    conda-forge
numpy                     2.2.4           py310hefbff90_0    conda-forge
ocl-icd                   2.3.3                hb9d3cd8_0    conda-forge
openblas                  0.3.29          pthreads_h6ec200e_0    conda-forge
opencl-headers            2024.10.24           h5888daf_0    conda-forge
openh264                  2.6.0                hc22cd8d_0    conda-forge
openjpeg                  2.5.3                h5fbd93e_0    conda-forge
openssl                   3.5.0                h7b32b05_0    conda-forge
orc                       2.1.1                h17f744e_1    conda-forge
output_viewer             1.3.3              pyhd8ed1ab_2    conda-forge
overrides                 7.7.0              pyhd8ed1ab_1    conda-forge
packaging                 24.2               pyhd8ed1ab_2    conda-forge
paginate                  0.5.7              pyhd8ed1ab_1    conda-forge
pandas                    2.2.3           py310h5eaa309_1    conda-forge
pandoc                    3.6.4                ha770c72_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
panel                     1.6.2              pyhd8ed1ab_0    conda-forge
pango                     1.56.3               h9ac818e_1    conda-forge
param                     2.2.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.4              pyhd8ed1ab_1    conda-forge
partd                     1.4.2              pyhd8ed1ab_0    conda-forge
pathspec                  0.12.1             pyhd8ed1ab_1    conda-forge
patsy                     1.0.1              pyhd8ed1ab_1    conda-forge
pbr                       6.1.1              pyhd8ed1ab_1    conda-forge
pcmdi_metrics             3.8.2              pyhd8ed1ab_0    conda-forge
pcre2                     10.44                hba22ea6_2    conda-forge
pexpect                   4.9.0              pyhd8ed1ab_1    conda-forge
pickleshare               0.7.5           pyhd8ed1ab_1004    conda-forge
pillow                    10.4.0          py310he228d35_1    conda-forge
pint                      0.24.4             pyhd8ed1ab_1    conda-forge
pip                       25.0.1             pyh8b19718_0    conda-forge
pixman                    0.44.2               h29eaf8c_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_2    conda-forge
platformdirs              4.3.7              pyh29332c3_0    conda-forge
plotly                    6.0.1              pyhd8ed1ab_0    conda-forge
pluggy                    1.5.0              pyhd8ed1ab_1    conda-forge
polars                    1.27.0          py310hc556931_0    conda-forge
pooch                     1.8.2              pyhd8ed1ab_1    conda-forge
popt                      1.16              h0b475e3_2002    conda-forge
progressbar2              4.5.0              pyhd8ed1ab_1    conda-forge
proj                      9.5.1                h0054346_0    conda-forge
prometheus-cpp            1.3.0                ha5d0236_0    conda-forge
prometheus_client         0.21.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.50             pyha770c72_0    conda-forge
prompt_toolkit            3.0.50               hd8ed1ab_0    conda-forge
properscoring             0.1                pyhd8ed1ab_1    conda-forge
psutil                    7.0.0           py310ha75aee5_0    conda-forge
pthread-stubs             0.4               hb9d3cd8_1002    conda-forge
ptyprocess                0.7.0              pyhd8ed1ab_1    conda-forge
pugixml                   1.15                 h3f63f65_0    conda-forge
pulseaudio-client         17.0                 hac146a9_1    conda-forge
pure_eval                 0.2.3              pyhd8ed1ab_1    conda-forge
pyamg                     5.2.1           py310ha2bacc8_1    conda-forge
pyarrow                   19.0.1          py310hff52083_0    conda-forge
pyarrow-core              19.0.1          py310hac404ae_0_cpu    conda-forge
pybtex                    0.24.0             pyhd8ed1ab_3    conda-forge
pycparser                 2.22               pyh29332c3_1    conda-forge
pyct                      0.5.0              pyhd8ed1ab_1    conda-forge
pyevtk                    1.6.0              pyhb0bfe47_1    conda-forge
pygments                  2.19.1             pyhd8ed1ab_0    conda-forge
pyjwt                     2.10.1             pyhd8ed1ab_0    conda-forge
pymdown-extensions        10.14.3            pyhd8ed1ab_0    conda-forge
pyogrio                   0.10.0          py310h0aed7a2_1    conda-forge
pypandoc                  1.15               pyhd8ed1ab_0    conda-forge
pyparsing                 3.2.3              pyhd8ed1ab_1    conda-forge
pyproj                    3.7.0           py310h2e9f774_0    conda-forge
pyremap                   1.4.1              pyhd8ed1ab_0    conda-forge
pyshp                     2.3.1              pyhd8ed1ab_1    conda-forge
pysocks                   1.7.1              pyha55dd90_7    conda-forge
pyspharm                  1.0.9           py310h1ea52e5_1014    conda-forge
pytest                    8.3.5              pyhd8ed1ab_0    conda-forge
python                    3.10.16         habfa6aa_3_cpython    conda-forge
python-dateutil           2.9.0.post0        pyhff2d567_1    conda-forge
python-fastjsonschema     2.21.1             pyhd8ed1ab_0    conda-forge
python-igraph             0.11.8          py310h5996f03_0    conda-forge
python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
python-slugify            8.0.4              pyhd8ed1ab_1    conda-forge
python-tzdata             2025.2             pyhd8ed1ab_0    conda-forge
python-utils              3.9.1              pyhff2d567_1    conda-forge
python_abi                3.10                    6_cp310    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyviz_comms               3.0.4              pyhd8ed1ab_1    conda-forge
pywavelets                1.8.0           py310hf462985_0    conda-forge
pyyaml                    6.0.2           py310h89163eb_2    conda-forge
pyyaml-env-tag            0.1                pyhd8ed1ab_1    conda-forge
pyzmq                     26.4.0          py310h71f11fc_0    conda-forge
qhull                     2020.2               h434a139_5    conda-forge
r                         4.4             r44hd8ed1ab_1008    conda-forge
r-askpass                 1.2.1             r44h2b5f3a1_0    conda-forge
r-assertthat              0.2.1             r44hc72bb7e_5    conda-forge
r-backports               1.5.0             r44hb1dbf0f_1    conda-forge
r-base                    4.4.3                h9c32bc2_0    conda-forge
r-base64enc               0.1_3           r44hb1dbf0f_1007    conda-forge
r-bit                     4.6.0             r44h2b5f3a1_0    conda-forge
r-bit64                   4.6.0_1           r44h2b5f3a1_0    conda-forge
r-blob                    1.2.4             r44hc72bb7e_2    conda-forge
r-boot                    1.3_31            r44hc72bb7e_0    conda-forge
r-broom                   1.0.8             r44hc72bb7e_0    conda-forge
r-bslib                   0.9.0             r44hc72bb7e_0    conda-forge
r-cachem                  1.1.0             r44hb1dbf0f_1    conda-forge
r-callr                   3.7.6             r44hc72bb7e_1    conda-forge
r-cellranger              1.1.0           r44hc72bb7e_1007    conda-forge
r-class                   7.3_23            r44h2b5f3a1_0    conda-forge
r-cli                     3.6.4             r44h93ab643_1    conda-forge
r-clipr                   0.8.0             r44hc72bb7e_3    conda-forge
r-cluster                 2.1.8.1           r44hb67ce94_0    conda-forge
r-codetools               0.2_20            r44hc72bb7e_1    conda-forge
r-colorspace              2.1_1             r44hdb488b9_0    conda-forge
r-conflicted              1.2.0             r44h785f33e_2    conda-forge
r-cpp11                   0.5.2             r44h785f33e_1    conda-forge
r-crayon                  1.5.3             r44hc72bb7e_1    conda-forge
r-curl                    6.2.2             r44h2700575_0    conda-forge
r-data.table              1.17.0            r44he23165d_0    conda-forge
r-dbi                     1.2.3             r44hc72bb7e_1    conda-forge
r-dbplyr                  2.5.0             r44hc72bb7e_1    conda-forge
r-digest                  0.6.37            r44h0d4f4ea_0    conda-forge
r-dplyr                   1.1.4             r44h0d4f4ea_1    conda-forge
r-dtplyr                  1.3.1             r44hc72bb7e_2    conda-forge
r-ellipsis                0.3.2             r44hb1dbf0f_3    conda-forge
r-evaluate                1.0.3             r44hc72bb7e_0    conda-forge
r-fansi                   1.0.6             r44hb1dbf0f_1    conda-forge
r-farver                  2.1.2             r44ha18555a_1    conda-forge
r-fastmap                 1.2.0             r44ha18555a_1    conda-forge
r-fontawesome             0.5.3             r44hc72bb7e_0    conda-forge
r-forcats                 1.0.0             r44hc72bb7e_2    conda-forge
r-foreign                 0.8_90            r44h2b5f3a1_0    conda-forge
r-fs                      1.6.5             r44h93ab643_0    conda-forge
r-gargle                  1.5.2             r44h785f33e_1    conda-forge
r-generics                0.1.3             r44hc72bb7e_3    conda-forge
r-ggplot2                 3.5.2             r44hc72bb7e_0    conda-forge
r-glue                    1.8.0             r44h2b5f3a1_0    conda-forge
r-googledrive             2.1.1             r44hc72bb7e_2    conda-forge
r-googlesheets4           1.1.1             r44h785f33e_2    conda-forge
r-gtable                  0.3.6             r44hc72bb7e_0    conda-forge
r-haven                   2.5.4             r44h0d4f4ea_1    conda-forge
r-highr                   0.11              r44hc72bb7e_1    conda-forge
r-hms                     1.1.3             r44hc72bb7e_2    conda-forge
r-htmltools               0.5.8.1           r44ha18555a_1    conda-forge
r-httr                    1.4.7             r44hc72bb7e_1    conda-forge
r-ids                     1.0.1             r44hc72bb7e_4    conda-forge
r-isoband                 0.2.7             r44ha18555a_3    conda-forge
r-jquerylib               0.1.4             r44hc72bb7e_3    conda-forge
r-jsonlite                2.0.0             r44h2b5f3a1_0    conda-forge
r-kernsmooth              2.23_26           r44h8461fee_0    conda-forge
r-knitr                   1.50              r44hc72bb7e_0    conda-forge
r-labeling                0.4.3             r44hc72bb7e_1    conda-forge
r-lattice                 0.22_7            r44h2b5f3a1_0    conda-forge
r-lifecycle               1.0.4             r44hc72bb7e_1    conda-forge
r-lubridate               1.9.4             r44h2b5f3a1_0    conda-forge
r-magrittr                2.0.3             r44hb1dbf0f_3    conda-forge
r-mass                    7.3_64            r44h2b5f3a1_0    conda-forge
r-matrix                  1.7_3             r44h2ae2be5_0    conda-forge
r-memoise                 2.0.1             r44hc72bb7e_3    conda-forge
r-mgcv                    1.9_3             r44h2ae2be5_0    conda-forge
r-mime                    0.13              r44h2b5f3a1_0    conda-forge
r-modelr                  0.1.11            r44hc72bb7e_2    conda-forge
r-munsell                 0.5.1             r44hc72bb7e_1    conda-forge
r-nlme                    3.1_168           r44hb67ce94_0    conda-forge
r-nnet                    7.3_20            r44h2b5f3a1_0    conda-forge
r-openssl                 2.3.2             r44he8289e2_0    conda-forge
r-pillar                  1.10.2            r44hc72bb7e_0    conda-forge
r-pkgconfig               2.0.3             r44hc72bb7e_4    conda-forge
r-prettyunits             1.2.0             r44hc72bb7e_1    conda-forge
r-processx                3.8.6             r44h2b5f3a1_0    conda-forge
r-progress                1.2.3             r44hc72bb7e_1    conda-forge
r-ps                      1.8.1             r44h2b5f3a1_0    conda-forge
r-purrr                   1.0.4             r44h2b5f3a1_0    conda-forge
r-r6                      2.6.1             r44hc72bb7e_0    conda-forge
r-ragg                    1.4.0             r44hdcc764c_0    conda-forge
r-rappdirs                0.3.3             r44hb1dbf0f_3    conda-forge
r-rcolorbrewer            1.1_3             r44h785f33e_3    conda-forge
r-readr                   2.1.5             r44h0d4f4ea_1    conda-forge
r-readxl                  1.4.5             r44h328fee5_0    conda-forge
r-recommended             4.4             r44hd8ed1ab_1007    conda-forge
r-rematch                 2.0.0             r44hc72bb7e_1    conda-forge
r-rematch2                2.1.2             r44hc72bb7e_4    conda-forge
r-reprex                  2.1.1             r44hc72bb7e_1    conda-forge
r-rlang                   1.1.5             r44h93ab643_0    conda-forge
r-rmarkdown               2.29              r44hc72bb7e_0    conda-forge
r-rpart                   4.1.24            r44h2b5f3a1_0    conda-forge
r-rstudioapi              0.17.1            r44hc72bb7e_0    conda-forge
r-rvest                   1.0.4             r44hc72bb7e_1    conda-forge
r-sass                    0.4.9             r44ha18555a_1    conda-forge
r-scales                  1.3.0             r44hc72bb7e_1    conda-forge
r-selectr                 0.4_2             r44hc72bb7e_4    conda-forge
r-spatial                 7.3_18            r44h2b5f3a1_0    conda-forge
r-stringi                 1.8.7             r44h3c328a7_0    conda-forge
r-stringr                 1.5.1             r44h785f33e_1    conda-forge
r-survival                3.8_3             r44h2b5f3a1_0    conda-forge
r-sys                     3.4.3             r44h2b5f3a1_0    conda-forge
r-systemfonts             1.2.1             r44h5c043a3_0    conda-forge
r-textshaping             1.0.0             r44h0f89ba2_0    conda-forge
r-tibble                  3.2.1             r44hdb488b9_3    conda-forge
r-tidyr                   1.3.1             r44h0d4f4ea_1    conda-forge
r-tidyselect              1.2.1             r44hc72bb7e_1    conda-forge
r-tidyverse               2.0.0             r44h785f33e_2    conda-forge
r-timechange              0.3.0             r44ha18555a_1    conda-forge
r-tinytex                 0.56              r44hc72bb7e_0    conda-forge
r-tzdb                    0.5.0             r44h93ab643_0    conda-forge
r-utf8                    1.2.4             r44hb1dbf0f_1    conda-forge
r-uuid                    1.2_1             r44hdb488b9_0    conda-forge
r-vctrs                   0.6.5             r44h0d4f4ea_1    conda-forge
r-viridislite             0.4.2             r44hc72bb7e_2    conda-forge
r-vroom                   1.6.5             r44h0d4f4ea_1    conda-forge
r-withr                   3.0.2             r44hc72bb7e_0    conda-forge
r-xfun                    0.52              r44h93ab643_0    conda-forge
r-xml2                    1.3.8             r44h1bb2df6_0    conda-forge
r-yaml                    2.3.10            r44hdb488b9_0    conda-forge
rasterio                  1.4.3           py310hbf83865_0    conda-forge
rav1e                     0.6.6                he8a937b_2    conda-forge
re2                       2024.07.02           h9925aae_3    conda-forge
readline                  8.2                  h8c095d6_2    conda-forge
referencing               0.36.2             pyh29332c3_0    conda-forge
regionmask                0.13.0             pyhd8ed1ab_0    conda-forge
requests                  2.32.3             pyhd8ed1ab_1    conda-forge
retrying                  1.3.4              pyhd8ed1ab_0    conda-forge
rfc3339-validator         0.1.4              pyhd8ed1ab_1    conda-forge
rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
rpds-py                   0.24.0          py310hc1293b2_0    conda-forge
rsync                     3.4.1                h81c0278_1    conda-forge
s2n                       1.5.15               hd830067_0    conda-forge
scikit-image              0.25.2          py310h5eaa309_0    conda-forge
scikit-learn              1.6.1           py310h27f47ee_0    conda-forge
scipy                     1.15.2          py310h1d65ade_0    conda-forge
sdl2                      2.32.54              h9b8e6db_0    conda-forge
sdl3                      3.2.10               h3083f51_0    conda-forge
seaborn                   0.13.2               hd8ed1ab_3    conda-forge
seaborn-base              0.13.2             pyhd8ed1ab_3    conda-forge
sed                       4.8                  he412f7d_0    conda-forge
send2trash                1.8.3              pyh0d859eb_1    conda-forge
setuptools                78.1.0             pyhff2d567_0    conda-forge
setuptools-git-versioning 2.1.0              pyhd8ed1ab_0    conda-forge
setuptools-scm            8.2.1              pyhd8ed1ab_0    conda-forge
setuptools_scm            8.2.1                hd8ed1ab_0    conda-forge
shapely                   2.1.0           py310h247727d_0    conda-forge
simplejson                3.20.1          py310ha75aee5_0    conda-forge
six                       1.17.0             pyhd8ed1ab_0    conda-forge
snappy                    1.2.1                h8bd8927_1    conda-forge
sniffio                   1.3.1              pyhd8ed1ab_1    conda-forge
snuggs                    1.4.7              pyhd8ed1ab_2    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_1    conda-forge
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
sparse                    0.16.0             pyh1646c32_0    conda-forge
spatialpandas             0.4.10             pyhd8ed1ab_1    conda-forge
spirv-tools               2024.3               h84d6215_0    conda-forge
sqlite                    3.49.1               h9eae976_2    conda-forge
squadgen                  1.2.2                h262f9a8_0    conda-forge
stack_data                0.6.3              pyhd8ed1ab_1    conda-forge
statsmodels               0.14.4          py310hf462985_0    conda-forge
stevedore                 5.4.1              pyhd8ed1ab_0    conda-forge
svt-av1                   3.0.2                h5888daf_0    conda-forge
sympy                     1.13.3           pyh2585a3b_105    conda-forge
sysroot_linux-64          2.17                h0157908_18    conda-forge
tabulate                  0.9.0              pyhd8ed1ab_2    conda-forge
tbb                       2022.1.0             h4ce085d_0    conda-forge
tblib                     3.1.0              pyhd8ed1ab_0    conda-forge
terminado                 0.18.1             pyh0d859eb_0    conda-forge
text-unidecode            1.3                pyhd8ed1ab_2    conda-forge
texttable                 1.7.0              pyhd8ed1ab_1    conda-forge
threadpoolctl             3.6.0              pyhecae5ae_0    conda-forge
tifffile                  2025.3.30          pyhd8ed1ab_0    conda-forge
tinycss2                  1.4.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tktable                   2.10                 h8bc8fbc_6    conda-forge
tomli                     2.2.1              pyhd8ed1ab_1    conda-forge
toolz                     1.0.0              pyhd8ed1ab_1    conda-forge
tornado                   6.4.2           py310ha75aee5_0    conda-forge
tqdm                      4.67.1             pyhd8ed1ab_1    conda-forge
traitlets                 5.14.3             pyhd8ed1ab_1    conda-forge
traittypes                0.2.1              pyh9f0ad1d_2    conda-forge
types-python-dateutil     2.9.0.20241206     pyhd8ed1ab_0    conda-forge
typing-extensions         4.13.2               h0e9735f_0    conda-forge
typing_extensions         4.13.2             pyh29332c3_0    conda-forge
typing_utils              0.1.0              pyhd8ed1ab_1    conda-forge
tzdata                    2025b                h78e105d_0    conda-forge
uc-micro-py               1.0.3              pyhd8ed1ab_1    conda-forge
udunits2                  2.2.28               h40f5838_3    conda-forge
unicodedata2              16.0.0          py310ha75aee5_0    conda-forge
uri-template              1.3.0              pyhd8ed1ab_1    conda-forge
uriparser                 0.9.8                hac33072_0    conda-forge
urllib3                   2.3.0              pyhd8ed1ab_0    conda-forge
uxarray                   2025.03.0          pyhd8ed1ab_0    conda-forge
validators                0.34.0             pyhd8ed1ab_1    conda-forge
watchdog                  6.0.0           py310hff52083_0    conda-forge
wayland                   1.23.1               h3e06ad9_0    conda-forge
wayland-protocols         1.43                 hd8ed1ab_0    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_1    conda-forge
webcolors                 24.11.1            pyhd8ed1ab_0    conda-forge
webencodings              0.5.1              pyhd8ed1ab_3    conda-forge
websocket-client          1.8.0              pyhd8ed1ab_1    conda-forge
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge
widgetsnbextension        4.0.14             pyhd8ed1ab_0    conda-forge
windspharm                2.0.0              pyhd8ed1ab_1    conda-forge
wxwidgets                 3.2.6                hf87bdbc_3    conda-forge
x264                      1!164.3095           h166bdaf_2    conda-forge
x265                      3.5                  h924138e_3    conda-forge
xarray                    2025.1.1           pyhd8ed1ab_0    conda-forge
xcdat                     0.8.0              pyhd8ed1ab_0    conda-forge
xerces-c                  3.2.5                h988505b_2    conda-forge
xesmf                     0.8.8              pyhd8ed1ab_1    conda-forge
xgcm                      0.8.1              pyhd8ed1ab_1    conda-forge
xhistogram                0.3.2              pyhd8ed1ab_0    conda-forge
xkeyboard-config          2.43                 hb9d3cd8_0    conda-forge
xmltodict                 0.14.2             pyhd8ed1ab_1    conda-forge
xorg-libice               1.1.2                hb9d3cd8_0    conda-forge
xorg-libsm                1.2.6                he73a12e_0    conda-forge
xorg-libx11               1.8.12               h4f16b4b_0    conda-forge
xorg-libxau               1.0.12               hb9d3cd8_0    conda-forge
xorg-libxaw               1.0.16               hb9d3cd8_0    conda-forge
xorg-libxcomposite        0.4.6                hb9d3cd8_2    conda-forge
xorg-libxcursor           1.2.3                hb9d3cd8_0    conda-forge
xorg-libxdamage           1.1.6                hb9d3cd8_0    conda-forge
xorg-libxdmcp             1.1.5                hb9d3cd8_0    conda-forge
xorg-libxext              1.3.6                hb9d3cd8_0    conda-forge
xorg-libxfixes            6.0.1                hb9d3cd8_0    conda-forge
xorg-libxi                1.8.2                hb9d3cd8_0    conda-forge
xorg-libxinerama          1.1.5                h5888daf_1    conda-forge
xorg-libxmu               1.2.1                hb9d3cd8_1    conda-forge
xorg-libxpm               3.5.17               hb9d3cd8_1    conda-forge
xorg-libxrandr            1.5.4                hb9d3cd8_0    conda-forge
xorg-libxrender           0.9.12               hb9d3cd8_0    conda-forge
xorg-libxscrnsaver        1.2.4                hb9d3cd8_0    conda-forge
xorg-libxshmfence         1.3.3                hb9d3cd8_0    conda-forge
xorg-libxt                1.3.1                hb9d3cd8_0    conda-forge
xorg-libxtst              1.2.5                hb9d3cd8_3    conda-forge
xorg-libxxf86vm           1.1.6                hb9d3cd8_0    conda-forge
xskillscore               0.0.26             pyhd8ed1ab_1    conda-forge
xxhash                    0.8.3                hb9d3cd8_0    conda-forge
xyzservices               2025.1.0           pyhd8ed1ab_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zeromq                    4.3.5                h3b0a872_7    conda-forge
zfp                       1.0.1                h5888daf_2    conda-forge
zict                      3.0.0              pyhd8ed1ab_1    conda-forge
zipp                      3.21.0             pyhd8ed1ab_1    conda-forge
zlib                      1.3.1                hb9d3cd8_2    conda-forge
zlib-ng                   2.2.4                h7955e40_0    conda-forge
zppy                      3.0.0              pyhd8ed1ab_0    conda-forge
zppy-interfaces           0.1.2              pyhd8ed1ab_0    conda-forge
zstandard                 0.23.0          py310ha75aee5_1    conda-forge
zstash                    1.4.4              pyhd8ed1ab_1    conda-forge
zstd                      1.5.7                hb8e6e7a_2    conda-forge

E3SM Unified v1.11.1 dependencies (compute node):
source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh

@tomvothecoder
Copy link
Collaborator Author

I can confirm this is related to the nompi vs. mpi version of esmf based on my comment from this thread, which produces the same MPI init error: #933 (comment).

I think we need to constrain our conda environments to use the nompi version`. Do we also need to constrain the conda-forge feedstock @xylar?

@xylar
Copy link
Contributor

xylar commented Jun 12, 2025

@tomvothecoder , @chengzhuzhang , it's up to you but any such constraint you use adds that restriction all packages in the environment, of course. So it would remove MPI parallelism from laptop/desktop users. Probably not a major user base for us.

@xylar
Copy link
Contributor

xylar commented Jun 12, 2025

Is this related to xESFM? If so, I really think you might consider calling ESMF directly. I've found that extra level of control is critical. And there's the option to swap out for MOAB, which is, even more performant with MPI parallelism.

@tomvothecoder
Copy link
Collaborator Author

@tomvothecoder , @chengzhuzhang , it's up to you but any such constraint you use adds that restriction all packages in the environment, of course. So it would remove MPI parallelism from laptop/desktop users. Probably not a major user base for us.

Thanks @xylar.

@chengzhuzhang — I’ve updated my investigation to confirm that e3sm_diags runs correctly with the latest dependencies and the nompi version of esmf.

Since most e3sm_diags users are on HPC systems (I'm guessing), we should constrain esmf to the nompi variant. Otherwise, the default mpi version gets installed, which only works on login nodes (not compute nodes, e.g., on Chrysalis with srun). This change also aligns with E3SM Unified.

I’ve pushed commit e5ebb3f with this update.

Can you confirm the latest dev.yml works for both the minimal test script and a full run? If so, please go ahead and merge. We can open a separate issue to add the esmf constraint to the conda-forge feedstock for v3.1.0.

@tomvothecoder
Copy link
Collaborator Author

tomvothecoder commented Jun 12, 2025

Is this related to xESFM? If so, I really think you might consider calling ESMF directly. I've found that extra level of control is critical. And there's the option to swap out for MOAB, which is, even more performant with MPI parallelism.

I think this is related to how mpi4py initializes MPI. I commented here with a minimum example and Naser's solution.

Not 100% sure.

@xylar
Copy link
Contributor

xylar commented Jun 12, 2025

I think this is related to how mpi4py initializes MPI.

Okay, I see. But it indicates that we should basically seek to keep MPI and python separate from one another, which still speaks to using tools like ESMF and MOAB with subprocess calls, rather than using python wrappers, wouldn't you say?

@tomvothecoder
Copy link
Collaborator Author

I think this is related to how mpi4py initializes MPI.

Okay, I see. But it indicates that we should basically seek to keep MPI and python separate from one another, which still speaks to using tools like ESMF and MOAB with subprocess calls, rather than using python wrappers, wouldn't you say?

I see what you're saying. I recall you suggested using ESMF's regridding weight generator as a separate process call, which could workaround the MPI incompatibilities. Are there any other packages in E3SM Unified that use xESMF for regridding (including weight generation), similar to e3sm_diags right now?


I'll open up a new issue for the team to re-think how we implement regridding in e3sm_diags in order to ensure compatibility with MPI. It might/might not require extensive refactoring. Here's a possible solution:

  1. Use subprocess to generate regridding weights with ESMF (e.g., ESMF_RegridWeightGen) -- This might avoid the MPI init issue
import subprocess

subprocess.run([
    "ESMF_RegridWeightGen",
    "--source", "source_grid.nc",
    "--destination", "target_grid.nc",
    "--weight", "weights_file.nc",
    "--method", "bilinear",
    "--ignore_unmapped"
])
  1. Use xESMF to apply the pre-computed weights.
regridder = xesmf.Regridder(ds, ds_out, weights="weights_file.nc", reuse_weights=True)

As a near-term solution, constraining esmf to nompi makes sense to me.

@xylar
Copy link
Contributor

xylar commented Jun 12, 2025

Are there any other packages in E3SM Unified that use xESMF for regridding (including weight generation), similar to e3sm_diags right now?

No, I think e3sm_diags and MPAS-Analysis are the main codes that do regridding outside of NCO. And NCO definitely doesn't use xESMF. MPAS-Analysis uses pyremap, which makes subprocess calls to ESMF or MOAB.

@xylar
Copy link
Contributor

xylar commented Jun 12, 2025

I am definitely in favor of the long-term solution you proposed -- ESMF (or MOAB) for making mapping files, xESMF if you like for applying the weights (ncremap works fine as well).

@chengzhuzhang
Copy link
Contributor

@chengzhuzhang — I’ve updated my investigation to confirm that e3sm_diags runs correctly with the latest dependencies and the nompi version of esmf.

@tomvothecoder I can confirm that with nompi build of esmf works well, and the test finished about 2mins.
I agree with @tomvothecoder and @xylar that pinning nompi is a reasonable near term approach.

@chengzhuzhang
Copy link
Contributor

I just completed a full zppy run with the new dependencies for code on latest main. The run finished in reasonable time (66mins)

@tomvothecoder tomvothecoder marked this pull request as ready for review June 12, 2025 22:06
@tomvothecoder
Copy link
Collaborator Author

I just completed a full zppy run with the new dependencies for code on latest main. The run finished in reasonable time (66mins)

Nice! I will go ahead and merge this PR. We can tackle #988 later.

@tomvothecoder tomvothecoder merged commit 00839fa into main Jun 12, 2025
6 checks passed
@tomvothecoder tomvothecoder deleted the bug/985-perf branch June 12, 2025 22:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Bug fix (will increment patch version)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Investigate performance degradation on main branch compared to v3.0

4 participants