Skip to content

Update aurora spack templates with latest modules #318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Oct 31, 2025
Merged

Update aurora spack templates with latest modules #318

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
a0cf457
Update config_machines.xml
andrewdnolan Oct 29, 2025
8844647
Update aurora templates with latest modules
andrewdnolan Oct 29, 2025
f44b83b
lint config_machines.xml
andrewdnolan Oct 29, 2025
ab236db
remove fortran version from netcdf-c spec
andrewdnolan Oct 29, 2025
e142506
Update compiler paths
andrewdnolan Oct 29, 2025
cfacabd
Update oneapi compiler version on Aurora
xylar Oct 29, 2025
2f23373
Update all prefix paths
andrewdnolan Oct 30, 2025
6485d9d
Add extra_attributes section to oneapi
andrewdnolan Oct 30, 2025
5ece679
Fix use_e3sm_hdf5_netcdf for Aurora
xylar Oct 30, 2025
5e2cd70
Add gcc to Aurora spack template
xylar Oct 30, 2025
4dcdabf
Fix cmake prefix
xylar Oct 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 15 additions & 21 deletions mache/cime_machine_config/config_machines.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3492,24 +3492,24 @@
<COMPILERS>oneapi-ifxgpu,oneapi-ifx</COMPILERS>
<MPILIBS>mpich,mpich1024</MPILIBS>
<PROJECT>E3SM_Dec</PROJECT>
<SAVE_TIMING_DIR>/lus/flare/projects/E3SM_Dec/performance_archive</SAVE_TIMING_DIR>
<SAVE_TIMING_DIR>/lus/flare/projects/E3SMinput/baselines</SAVE_TIMING_DIR>
<SAVE_TIMING_DIR_PROJECTS>.*</SAVE_TIMING_DIR_PROJECTS>
<CIME_OUTPUT_ROOT>/lus/flare/projects/$PROJECT/$USER/scratch</CIME_OUTPUT_ROOT>
<DIN_LOC_ROOT>/lus/flare/projects/E3SM_Dec/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/lus/flare/projects/E3SM_Dec/inputdata/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DIN_LOC_ROOT>/lus/flare/projects/E3SMinput/data</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/lus/flare/projects/E3SMinput/data/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>$CIME_OUTPUT_ROOT/archive/$CASE</DOUT_S_ROOT>
<BASELINE_ROOT>/lus/flare/projects/E3SM_Dec/baselines/$COMPILER</BASELINE_ROOT>
<CCSM_CPRNC>/lus/flare/projects/E3SM_Dec/tools/cprnc/cprnc</CCSM_CPRNC>
<BASELINE_ROOT>/lus/flare/projects/E3SMinput/baselines/$COMPILER</BASELINE_ROOT>
<CCSM_CPRNC>/lus/flare/projects/E3SMinput/tools/cprnc/cprnc</CCSM_CPRNC>
<GMAKE_J>16</GMAKE_J>
<TESTS>e3sm_developer</TESTS>
<TESTS>e3sm_integration</TESTS>
<NTEST_PARALLEL_JOBS>4</NTEST_PARALLEL_JOBS>
<BATCH_SYSTEM>pbspro</BATCH_SYSTEM>
<SUPPORTED_BY>e3sm</SUPPORTED_BY>
<MAX_TASKS_PER_NODE>102</MAX_TASKS_PER_NODE>
<MAX_TASKS_PER_NODE compiler="oneapi-ifxgpu">96</MAX_TASKS_PER_NODE>
<MAX_MPITASKS_PER_NODE>102</MAX_MPITASKS_PER_NODE>
<MAX_MPITASKS_PER_NODE compiler="oneapi-ifxgpu">12</MAX_MPITASKS_PER_NODE>
<PROJECT_REQUIRED>FALSE</PROJECT_REQUIRED>
<PROJECT_REQUIRED>TRUE</PROJECT_REQUIRED>
<mpirun mpilib="default">
<executable>mpiexec</executable>
<!--executable>numactl -m 2-3 mpiexec</executable--><!--for HBM runs-->
Expand All @@ -3529,9 +3529,13 @@
<cmd_path lang="csh">module</cmd_path>
<cmd_path lang="python">/usr/share/lmod/lmod/libexec/lmod python</cmd_path>
<modules>
<command name="load">cmake/3.30.5</command>
<command name="load">oneapi/release/2025.0.5</command>
</modules>
<command name="use">/lus/flare/projects/E3SMinput/soft/modulefiles</command>
<command name="load">cmake/3.31.8</command>
<command name="load">oneapi/release/2025.2.0</command>
<command name="load">netcdf/4.9.3c-4.6.2f</command>
<command name="load">pnetcdf/1.14.0</command>
<command name="load">adios2/2.10.2</command>
</modules>
<modules mpilib="mpich1024">
<command name="load">mpich-config/collective-tuning/1024</command>
</modules>
Expand All @@ -3540,16 +3544,6 @@
<EXEROOT>$CIME_OUTPUT_ROOT/$CASE/bld</EXEROOT>
<MAX_GB_OLD_TEST_DATA>0</MAX_GB_OLD_TEST_DATA>
<environment_variables>
<env name="NETCDF_PATH">/lus/flare/projects/E3SM_Dec/soft/netcdf/4.9.2c-4.6.1f/oneapi.eng.2024.07.30.002</env>
<env name="PNETCDF_PATH">/lus/flare/projects/E3SM_Dec/soft/pnetcdf/1.14.0/oneapi.eng.2024.07.30.002</env>
<env name="LD_LIBRARY_PATH">/lus/flare/projects/E3SM_Dec/soft/pnetcdf/1.14.0/oneapi.eng.2024.07.30.002/lib:/lus/flare/projects/E3SM_Dec/soft/netcdf/4.9.2c-4.6.1f/oneapi.eng.2024.07.30.002/lib:$ENV{LD_LIBRARY_PATH}</env>
<env name="PATH">/lus/flare/projects/E3SM_Dec/soft/pnetcdf/1.14.0/oneapi.eng.2024.07.30.002/bin:/lus/flare/projects/E3SM_Dec/soft/netcdf/4.9.2c-4.6.1f/oneapi.eng.2024.07.30.002/bin:$ENV{PATH}</env>
<env name="PKG_CONFIG_PATH">/lus/flare/projects/E3SM_Dec/soft/protobuf/21.6/oneapi.release.2025.0.5/lib/pkgconfig:$ENV{PKG_CONFIG_PATH}</env>
<env name="ADIOS2_ROOT">$SHELL{if [ -z "$ADIOS2_ROOT" ]; then echo /lus/flare/projects/E3SM_Dec/soft/adios2/2.10.2/oneapi.release.2025.0.5; else echo "$ADIOS2_ROOT"; fi}</env>
<env name="BLOSC2_ROOT">$SHELL{if [ -z "$BLOSC2_ROOT" ]; then echo /lus/flare/projects/E3SM_Dec/soft/c-blosc2/2.15.2/oneapi.release.2025.0.5; else echo "$BLOSC2_ROOT"; fi}</env>
<env name="MGARD_ROOT">$SHELL{if [ -z "$MGARD_ROOT" ]; then echo /lus/flare/projects/E3SM_Dec/soft/mgard/1.5.2/oneapi.release.2025.0.5; else echo "$MGARD_ROOT"; fi}</env>
<env name="SZ_ROOT">$SHELL{if [ -z "$SZ_ROOT" ]; then echo /lus/flare/projects/E3SM_Dec/soft/sz/2.1.12.5/oneapi.release.2025.0.5; else echo "$SZ_ROOT"; fi}</env>
<env name="ZFP_ROOT">$SHELL{if [ -z "$ZFP_ROOT" ]; then echo /lus/flare/projects/E3SM_Dec/soft/zfp/1.0.1/oneapi.release.2025.0.5; else echo "$ZFP_ROOT"; fi}</env>
<env name="FI_CXI_DEFAULT_CQ_SIZE">131072</env>
<env name="FI_CXI_CQ_FILL_PERCENT">20</env>
<env name="RLIMITS"> </env>
Expand All @@ -3574,7 +3568,7 @@
<env name="romio_cb_write">disable</env>
<env name="GATOR_INITIAL_MB">4000MB</env>
<env name="GATOR_DISABLE">0</env>
<env name="GPU_TILE_COMPACT">/lus/flare/projects/E3SM_Dec/tools/mpi_wrapper_utils/gpu_tile_compact.sh</env>
<env name="GPU_TILE_COMPACT">/lus/flare/projects/E3SMinput/tools/mpi_wrapper_utils/gpu_tile_compact.sh</env>
<env name="RANKS_BIND">list:1-8:9-16:17-24:25-32:33-40:41-48:53-60:61-68:69-76:77-84:85-92:93-100 --gpu-bind list:0.0:0.1:1.0:1.1:2.0:2.1:3.0:3.1:4.0:4.1:5.0:5.1 --mem-bind list:0:0:0:0:0:0:1:1:1:1:1:1</env>
<env name="ZES_ENABLE_SYSMAN">1</env>
<!-- default is ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE: enable this to run 4 MPI/tile or 48 MPI/node
Expand Down
22 changes: 11 additions & 11 deletions mache/spack/templates/aurora_oneapi-ifx_mpich.yaml
View file
Open in desktop
Copy link
Collaborator Author

@andrewdnolan andrewdnolan Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xylar to explain a little of what's going on. First, I loaded the (needed) modules from the config_machines.xml:

source /usr/share/lmod/lmod/init/sh
module load cmake/3.31.8
module load oneapi/release/2025.2.0

Then I ran:

$ which cmake
/opt/aurora/25.190.0/spack/unified/0.10.1/install/linux-sles15-x86_64/gcc-13.3.0/cmake-3.31.8-mvhtj5u/bin/cmake

$ ls -d /opt/aurora/25.190.0/spack/unified/0.10.1/install/linux-sles15-x86_64/gcc-13.3.0/libiconv*
ls: cannot access '/opt/aurora/25.190.0/spack/unified/0.10.1/install/linux-sles15-x86_64/gcc-13.3.0/libiconv*': No such file or directory

$ ls -d /opt/aurora/25.190.0/spack/unified/0.10.1/install/linux-sles15-x86_64/gcc-13.3.0/libxml2*
/opt/aurora/25.190.0/spack/unified/0.10.1/install/linux-sles15-x86_64/gcc-13.3.0/libxml2-2.13.5-dhzcbk4

$ ls -d /opt/aurora/25.190.0/spack/unified/0.10.1/install/linux-sles15-x86_64/gcc-13.3.0/gcc-runtime-*
/opt/aurora/25.190.0/spack/unified/0.10.1/install/linux-sles15-x86_64/gcc-13.3.0/gcc-runtime-13.3.0-ghotoln

$ which mpicc
/opt/aurora/25.190.0/spack/unified/0.10.1/install/linux-sles15-x86_64/oneapi-2025.2.0/mpich-develop-git.6037a7a-cym6jg6/bin/mpicc

which is where I got the new paths from. Also that's what's lead to the decision to delete the libiconv entry (I double checked the oneapi-2025.2.0 directory, but nothing there either.

Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ spack:
buildable: false
cmake:
externals:
- spec: cmake@3.30.5
prefix: /opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/cmake-3.30.5-igar7ts
- spec: cmake@3.31.8
prefix: /opt/aurora/25.190.0/spack/unified/0.10.1/install/linux-sles15-x86_64/gcc-13.3.0/cmake-3.31.8-mvhtj5u/bin/cmake
buildable: false
curl:
externals:
Expand Down Expand Up @@ -89,7 +89,7 @@ spack:
externals:
- spec: {{ compiler }}
modules:
- oneapi/release/2025.0.5
- oneapi/release/2025.2.0
buildable: false
gcc-runtime:
externals:
Expand All @@ -104,23 +104,23 @@ spack:
{%- if e3sm_hdf5_netcdf %}
hdf5:
externals:
- spec: [email protected].5+cxx+fortran+hl+mpi
prefix: /lus/flare/projects/E3SM_Dec/soft/hdf5/1.14.5/oneapi.eng.2024.07.30.002/
- spec: [email protected].6+cxx+fortran+hl+mpi
prefix: /lus/flare/projects/E3SMinput/soft/hdf5/1.14.6/oneapi.2025.2.0
buildable: false
netcdf-c:
externals:
- spec: [email protected].2+mpi~parallel-netcdf
prefix: /lus/flare/projects/E3SM_Dec/soft/netcdf/4.9.2c-4.6.1f/oneapi.eng.2024.07.30.002
- spec: [email protected].3+mpi~parallel-netcdf
prefix: /lus/flare/projects/E3SMinput/soft/netcdf/4.9.3c-4.6.2f/oneapi.2025.2.0
buildable: false
netcdf-fortran:
externals:
- spec: [email protected].1
prefix: /lus/flare/projects/E3SM_Dec/soft/netcdf/4.9.2c-4.6.1f/oneapi.eng.2024.07.30.002
- spec: [email protected].2
prefix: /lus/flare/projects/E3SMinput/soft/netcdf/4.9.3c-4.6.2f/oneapi.2025.2.0
buildable: false
parallel-netcdf:
externals:
- spec: parallel-netcdf@1.11.0+cxx+fortran
prefix: /lus/flare/projects/E3SM_Dec/soft/pnetcdf/1.14.0/oneapi.eng.2024.07.30.002
- spec: parallel-netcdf@1.14.0+cxx+fortran
prefix: /lus/flare/projects/E3SMinput/soft/pnetcdf/1.14.0/oneapi.2025.2.0
buildable: false
{%- endif %}
compilers:
Expand Down
Loading