Skip to content

Commit e0c30c8

Browse files
TidbitSoftwareTidbitSoftware
authored
Merge pull request #17 from TidbitSoftware/main
Added notes on compiling on Pleiades
2 parents 7ea4205 + 2f3ef92 commit e0c30c8

File tree

3 files changed

+279
-46
lines changed

3 files changed

+279
-46
lines changed

‎docs/installation/hpc/index.md‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,7 @@ title: High-Performance Computing (HPC)
33
layout: default
44
parent: Installation
55
nav_order: 4
6-
has_children: false
7-
has_toc: false
6+
has_children: true
87
---
98

109
# High-Performance Computing (HPC)
@@ -16,5 +15,5 @@ Note as well that the 'local' machine in the above case may be one that is physi
1615
## Configuration and Compiling ISSM
1716
Please see the <a href="https://issm.ess.uci.edu/trac/issm/wiki" target="_blank">ISSM Development Wiki</a> for notes on configuring and compiling on various HPC systems.
1817

19-
We will be working soon to migrate the content from the wiki to this documentation.
20-
18+
The following are pages that have already been migrated (also linked in the Main Navigation in the sidebar),
19+
- <a href="./pleiades">Pleiades (NASA NAS HECC)</a>

‎docs/installation/hpc/pleiades.md‎

Lines changed: 236 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,236 @@
1+
---
2+
title: Pleiades (NASA NAS HECC)
3+
layout: default
4+
parent: High-Performance Computing (HPC)
5+
---
6+
7+
# Pleiades (NASA NAS HECC)
8+
{: .no_toc }
9+
For comprehensive information on access to and use of NASA NAS High-End Computing Capability (HECC), please refer to <a href="https://www.nas.nasa.gov/hecc/support/kb" target="_blank">their knowledge base</a>.
10+
11+
## Table of Contents
12+
{: .no_toc }
13+
1. TOC
14+
{:toc}
15+
----
16+
17+
## Getting an Account
18+
New users will have to ask a NASA- or JPL-employed PI to request an account for them on NASA NAS HECC. They will need the group ID of the PI in order to submit this request.
19+
20+
First, the PI should use the following procedure,
21+
- navigate to the <a href="https://www.nas.nasa.gov/hecc/portal/accounts" target="_blank">NASA NAS HECC 'Account Request System' page</a>
22+
- select option '3. I want to request a NASA identity for one of my new users'
23+
- provide the requested information
24+
25+
Then, the user needs to request an account with the following procedure,
26+
- navigate to the <a href="https://www.nas.nasa.gov/hecc/portal/accounts" target="_blank">NASA NAS HECC 'Account Request System' page</a>
27+
- select option '2. I want to request and account for myself'
28+
- provide the group ID for the PI that initiated the request
29+
- the PI will then receive an email to approve the request
30+
31+
NOTE: All users must complete NASA's Basic IT Security Training
32+
33+
### Current Points of Contact
34+
- Dartmouth security officer: <a href="mailto:[email protected]">Sean McNamara</a> (<a href="https://itc.dartmouth.edu/about/who-we-are/itc-leadership" target="_blank">Dartmouth ITC</a>)
35+
- JPL IT security officer: <a href="mailto:[email protected]">Tomas Soderstrom</a>
36+
- UCI OIT security officer: <a href="mailto:[email protected]">Josh Drummond</a>
37+
38+
## Logging In
39+
Please refer to the following documentation pages to log into NAS systems and make subsequent logins more seamless,
40+
- <a href="https://www.nas.nasa.gov/hecc/support/kb/logging-into-nas-systems-for-the-first-time_539.html" target="_blank">Logging Into NAS Systems for the First Time</a>
41+
- <a href="https://www.nas.nasa.gov/hecc/support/kb/setting-up-public-key-authentication_230.html" target="_blank">Setting Up Public Key Authentication</a>
42+
- <a href="https://www.nas.nasa.gov/hecc/support/kb/setting-up-ssh-passthrough_232.html" target="_blank">Setting Up SSH Passthrough</a>
43+
- <a href="https://www.nas.nasa.gov/hecc/support/kb/one-step-connection-using-public-key-and-passthrough_62.html" target="_blank">One-Step Connection Using Public Key and Passthrough</a>
44+
45+
## Checking Out a Copy of the ISSM Code Repository
46+
Users should clone the ISSM code repository to their `/nobackup/` directory where they can save a lot more files than in their home directory.
47+
48+
## Environment
49+
Add the following to `~/.bashrc` in order to set up the environment properly for compiling ISSM,
50+
```sh
51+
# Modules
52+
module load mpi-hpe/mpt
53+
module load comp-intel/2020.4.304
54+
module load petsc/3.17.3_intel_mpt_py
55+
56+
# Variables
57+
export CC=mpicc
58+
export CXX=mpicxx
59+
export F77=mpif77
60+
export MPICC_CC=icx
61+
export MPICXX_CXX=icpx
62+
export MPF90_F90=ifort
63+
64+
export ISSM_DIR="<ISSM_DIR>"
65+
```
66+
replacing `<ISSM_DIR>` with the path to the local copy of the ISSM code repository. Run `source ~/.bashrc` to apply these changes to the current session.
67+
68+
{: .highlight-title }
69+
> NOTE
70+
>
71+
> If `~/.bashrc` is not loaded when on login, add a new file, `~/.bash_login`, with the following content,
72+
> ```sh
73+
> if [ -f ~/.bashrc ]; then . ~/.bashrc; fi
74+
> ```
75+
76+
{: .highlight-title }
77+
> NOTE
78+
>
79+
> The version of the `comp-intel` module as well as the corresponding value of variable `COMP_INTEL_ROOT` may need to be updated as recommended/available modules are updated on HECC. Please update this page or ask a project lead to do so when this occurs.
80+
81+
## Installing ISSM
82+
83+
{: .highlight-title }
84+
> Important
85+
>
86+
> ISSM and external packages should not be compiled on the Pleiades front end. Please refer to the NAS HECC knowledge base article <a href="https://www.nas.nasa.gov/hecc/support/kb/reserving-a-dedicated-compute-node_556.html" target="_blank">'Reserving a Dedicated Compute Node'</a> for instructions on reserving a and logging into a compute node for the purpose of compiling.
87+
88+
{: .highlight-title }
89+
> Important
90+
>
91+
> Do *not* install `mpich` and `petsc`. The MPI implementation (MPT) provided by HECC *must* be used. Pleiades will *only* be used to run solutions and the user's local machine for pre- and post-processing.
92+
93+
There are a number of configurations for ISSM provided below. Users may also refer to the recipes in `${ISSM_DIR}/jenkins/` prefixed with `pleiades-`.
94+
95+
### Installing ISSM with Basic Capabilities
96+
For an installation of ISSM with basic capabilities, the only external packages required are,
97+
```sh
98+
autotools install-linux.sh
99+
triangle install-linux.sh
100+
m1qn3 install-linux.sh
101+
semic install.sh
102+
```
103+
104+
Before configuring ISSM, run,
105+
```sh
106+
cd $ISSM_DIR
107+
autoreconf -ivf
108+
```
109+
110+
Then use the following configuring script (adapting it as needed),
111+
```sh
112+
export CFLAGS="-g -Ofast"
113+
export CXXFLAGS="-g -Ofast -xCORE-AVX512,CORE-AVX2 -xAVX -std=c++11"
114+
115+
./configure \
116+
--prefix="${ISSM_DIR}" \
117+
--enable-development \
118+
--enable-standalone-libraries \
119+
--with-wrappers=no \
120+
--with-fortran-lib="-L${COMP_INTEL_ROOT}/compiler/lib/intel64_lin -lifcore -lifport -lgfortran" \
121+
--with-mkl-libflags="-L${COMP_INTEL_ROOT}/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm" \
122+
--with-mpi-include="${MPI_ROOT}/include" \
123+
--with-mpi-libflags="-L${MPI_ROOT}/lib -lmpi" \
124+
--with-blas-lapack-lib="-L${COMP_INTEL_ROOT}/mkl/lib/intel64 -lmkl_blas95_lp64 -lmkl_lapack95_lp64" \
125+
--with-metis-dir="${PETSC_DIR}" \
126+
--with-parmetis-dir="${PETSC_DIR}" \
127+
--with-scalapack-lib="-L${COMP_INTEL_ROOT}/mkl/lib/intel64/libmkl_scalapack_lp64.so" \
128+
--with-mumps-dir="${PETSC_DIR}" \
129+
--with-petsc-dir="${PETSC_DIR}" \
130+
--with-triangle-dir="${ISSM_DIR}/externalpackages/triangle/install" \
131+
--with-m1qn3-dir="${ISSM_DIR}/externalpackages/m1qn3/install" \
132+
--with-semic-dir="${ISSM_DIR}/externalpackages/semic/install"
133+
```
134+
135+
### Installing ISSM with Dakota
136+
For an installation of ISSM with Dakota, the following external packages are required,
137+
```sh
138+
autotools install-linux.sh
139+
boost install-1.7-linux.sh
140+
dakota install-6.2-pleiades.sh
141+
chaco install-linux.sh
142+
triangle install-linux.shwhic
143+
m1qn3 install-linux.sh
144+
semic install.sh
145+
```
146+
147+
Before configuring ISSM, run,
148+
```sh
149+
cd $ISSM_DIR
150+
autoreconf -ivf
151+
```
152+
153+
Then use the following configuring script (adapting it as needed),
154+
```sh
155+
export CFLAGS="-g -Ofast"
156+
export CXXFLAGS="-g -Ofast -xCORE-AVX512,CORE-AVX2 -xAVX -std=c++11"
157+
158+
./configure \
159+
--prefix="${ISSM_DIR}" \
160+
--enable-development \
161+
--enable-standalone-libraries \
162+
--with-wrappers=no \
163+
--with-fortran-lib="-L${COMP_INTEL_ROOT}/compiler/lib/intel64_lin -lifcore -lifport -lgfortran" \
164+
--with-mkl-libflags="-L${COMP_INTEL_ROOT}/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm" \
165+
--with-mpi-include="${MPI_ROOT}/include" \
166+
--with-mpi-libflags="-L${MPI_ROOT}/lib -lmpi" \
167+
--with-blas-lapack-lib="-L${COMP_INTEL_ROOT}/mkl/lib/intel64 -lmkl_blas95_lp64 -lmkl_lapack95_lp64" \
168+
--with-metis-dir="${PETSC_DIR}" \
169+
--with-parmetis-dir="${PETSC_DIR}" \
170+
--with-scalapack-lib="-L${COMP_INTEL_ROOT}/mkl/lib/intel64/libmkl_scalapack_lp64.so" \
171+
--with-mumps-dir="${PETSC_DIR}" \
172+
--with-petsc-dir="${PETSC_DIR}" \
173+
--with-boost-dir="${ISSM_DIR}/externalpackages/boost/install" \
174+
--with-dakota-dir="${ISSM_DIR}/externalpackages/dakota/install" \
175+
--with-chaco-dir="${ISSM_DIR}/externalpackages/chaco/install" \
176+
--with-triangle-dir="${ISSM_DIR}/externalpackages/triangle/install" \
177+
--with-m1qn3-dir="${ISSM_DIR}/externalpackages/m1qn3/install" \
178+
--with-semic-dir="${ISSM_DIR}/externalpackages/semic/install"
179+
```
180+
181+
### Installing ISSM with Solid Earth Capabilities
182+
For an installation of ISSM with solid Earth capabilities, the following external packages are required,
183+
```sh
184+
autotools install-linux.sh
185+
boost install-1.7-linux.sh
186+
dakota install-6.2-pleiades.sh
187+
chaco install-linux.sh
188+
zlib install-1.sh
189+
hdf5 install-1.sh
190+
netcdf install-4.sh
191+
sqlite install.sh
192+
proj install-6.sh
193+
gdal install-3-linux.sh
194+
gshhg install.sh
195+
gmt install-6-pleiades.sh
196+
gmsh install-4-pleiades.sh
197+
triangle install-linux.sh
198+
m1qn3 install-linux.sh
199+
semic install.sh
200+
```
201+
202+
Before configuring ISSM, run,
203+
```sh
204+
cd $ISSM_DIR
205+
autoreconf -ivf
206+
```
207+
208+
Then use the following configuring script (adapting it as needed),
209+
```sh
210+
export CFLAGS="-g -Ofast"
211+
export CXXFLAGS="-g -Ofast -xCORE-AVX512,CORE-AVX2 -xAVX -std=c++11"
212+
213+
./configure \
214+
--prefix="${ISSM_DIR}" \
215+
--enable-development \
216+
--enable-standalone-libraries \
217+
--with-wrappers=no \
218+
--with-fortran-lib="-L${COMP_INTEL_ROOT}/compiler/lib/intel64_lin -lifcore -lifport -lgfortran" \
219+
--with-mkl-libflags="-L${COMP_INTEL_ROOT}/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm" \
220+
--with-mpi-include="${MPI_ROOT}/include" \
221+
--with-mpi-libflags="-L${MPI_ROOT}/lib -lmpi" \
222+
--with-blas-lapack-lib="-L${COMP_INTEL_ROOT}/mkl/lib/intel64 -lmkl_blas95_lp64 -lmkl_lapack95_lp64" \
223+
--with-metis-dir="${PETSC_DIR}" \
224+
--with-parmetis-dir="${PETSC_DIR}" \
225+
--with-scalapack-lib="-L${COMP_INTEL_ROOT}/mkl/lib/intel64/libmkl_scalapack_lp64.so" \
226+
--with-mumps-dir="${PETSC_DIR}" \
227+
--with-hdf5-dir="${ISSM_DIR}/externalpackages/hdf5/install" \
228+
--with-petsc-dir="${PETSC_DIR}" \
229+
--with-boost-dir="${ISSM_DIR}/externalpackages/boost/install" \
230+
--with-dakota-dir="${ISSM_DIR}/externalpackages/dakota/install" \
231+
--with-chaco-dir="${ISSM_DIR}/externalpackages/chaco/install" \
232+
--with-proj-dir="${ISSM_DIR}/externalpackages/proj/install" \
233+
--with-triangle-dir="${ISSM_DIR}/externalpackages/triangle/install" \
234+
--with-m1qn3-dir="${ISSM_DIR}/externalpackages/m1qn3/install" \
235+
--with-semic-dir="${ISSM_DIR}/externalpackages/semic/install"
236+
```

0 commit comments

Comments
 (0)