Skip to content

Building and Running Tusas

Jump to bottom
Chris Newman edited this page Oct 14, 2019 · 40 revisions

Building and Running Tusas with CMake

Building Tusas

Create and cd to a build directory:

pn1512687:~/work/tusas_git$ mkdir build;cd build

As you enter the CMake command to build the tusas executable, the path to your Trilinos download must be provided as TRILINOS_DIR, shown below. 
pn1512687:~/work/tusas_git/build$ cmake -DTRILINOS_DIR:string=/Users/cnewman/src/trilinos-11.12.1-Source/GCC_4_9_2_MPI_OMP_OPT/ ../

pn1512687:~/work/tusas_git/build$ make

If any mistakes are made with this CMake command, make sure to delete CMakeCache.txt, CMakeFiles, Makefile, and cmake_install.cmake, to be sure that the corrected commands may go through properly.

List of CMake options and macros

There are several build options that can be passed to CMake via the -D option:

  1. TRILINOS_DIR specifies the Trilinos build location. Mandatory. Ex: -DTRILINOS_DIR:string=/Users/cnewman/src/trilinos-11.12.1-Source/GCC_4_9_2_MPI_OMP_OPT/
  2. TUSAS_CXX_FLAGS specifies additional compilation flags. Ex: -DTUSAS_CXX_FLAGS="-DTUSAS_KOKKOS_PRINT_CONFIG -DTUSAS_NEW_MUELU -w"
    Some common options for TUSAS_CXX_FLAGS are:
  • -w turn off warnings for GCC.
  • -DTUSAS_KOKKOS_PRINT_CONFIG specifies to print the Kokkos configuration at the beginning of a Tusas run.
  • -DTUSAS_NEW_MUELU specifies to use the MueLu interface found in the Trilinos master branch. If utilizing a previous release version, do not use this.
  • TUSAS_HAVE_MKL specifies to use MKL blas and lapack.
  • TUSAS_NO_LAPACK do not compile Lapack code; ie disables the error estimator.
  • TUSAS_HAVE_CLAPACK use Clapack interface rather than standard lapack.

Building with Ninja

Ninja can be a little faster than make. Add the -GNinja generator to the cmake line:

pn1931313:~/work/tusas_git/VOTD_KOKKOS$ cmake -DTRILINOS_DIR:string=/Users/cnewman/src/Trilinos/VOTD_KOKKOS/ -DTUSAS_CXX_FLAGS=“-DTUSAS_KOKKOS_PRINT_CONFIG -DTUSAS_NEW_MUELU -w” -GNinja ../

pn1931313:~/work/tusas_git/VOTD_KOKKOS$ ninja

Similar to make, use the following:

pn1931313:~/work/tusas_git/VOTD_KOKKOS$ ninja clean

pn1931313:~/work/tusas_git/VOTD_KOKKOS$ ninja -j 20

See https://ninja-build.org/manual.html

Building with clang/gfortran

To be updated…

We need a section on valgrind and clang address sanitizer. For clang address sanitizer:

$ cmake -DTRILINOS_DIR:string=/Users/cnewman/src/trilinos-12.6.4-Source/LLVM_6_0_0_MPI_OMP_OPT/ -DTUSAS_CXX_FLAGS="-fsanitize=address -O1 -fno-omit-frame-pointer -g" ../../

Running Tusas

$ ./tusas --input-file=tusas.xml

Input files

  1. An example xml input file is tusas/trunk/tusas.xml. Options and default values can be found in the file tusas/trunk/input/include/ParamNames.h.
  2. The xml file points to a meshfile (in exodusII format). There are example meshfiles in tusas/trunk/meshes. Meshfiles can be produced with: http://cubit.sandia.gov Contact cnewman at lanl.gov for a copy of cubit.
  3. Linear solver options can be found at: https://trilinos.org/docs/dev/packages/stratimikos/doc/html/classStratimikos_1_1DefaultLinearSolverBuilder.html
  4. Nonlinear solver options can be found at: https://trilinos.org/docs/dev/packages/nox/doc/html/parameters.html
  5. ML options can be found at: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwisuJPerYzXAhVLj1QKHZ8SA4cQFggoMAA&url=https%3A%2F%2Ftrilinos.org%2Foldsite%2Fpackages%2Fml%2Fmlguide5.pdf&usg=AOvVaw3vV-77r1zAOsGshZtHXbvC

Output files

There are three (optionally four) output files produced by Tusas.

  1. results.e is an ExodusII (binary) file that contains the solution on the mesh. It can be viewed easily with Paraview: http://www.paraview.org
    Two useful tools can be found in the $(TRILINOS_DIR)/bin directory. exodiff can be used to difference two ExodusII files. exotxt can be used to dump the contents of an ExodusII file in text to the terminal or a file.
  2. time.dat is an ASCI file that provides solver timing diagnostics.
  3. jfnk.dat is an ASCI file that provides solver performance diagnostics.
  4. Some test cases produce an ASCI file vtip.dat that contains dendrite tip velocity and position along the x-axis as a function of time.

Additional runtime options

There are three additional runtime options for Tusas.

  1. --writedecomp Particularly with slurm on HPC machines and on some linux machines, the decomposition scripts spawned by Tusas do not get run properly. This option writes the files decompscript and epuscript which can be used to manually decompose and join the mesh files respectively.
  2. --skipdecomp Assumes the decomposed mesh files already exist in decomp/; Tusas does not spawn the decomposition scripts. The workflow to employ these options 1 and 2 is as follows:

    
$ mpirun -np 4 ./tusas —input-file=tusas.xml —writedecomp
    
$ bash decompscript
    
$ mpirun -np 4 ./tusas —input-file=tusas.xml —skipdecomp
    
$ bash epuscript

    Note that to rerun Tusas with the same decomposition, only mpirun -np 4 ./tusas --input-file=tusas.xml --skipdecomp is needed.
  3. --restart Restart from the last output found in decomp/results.*. Assumes the identical simulation parameters as previous run.

Testing Tusas

We utilize CTest, a testing tool that is distributed with CMake. All tests are created and run from within Test, which is a subdirectory of trunk.

Configuring Testing

The build process outlined above configures the ctest testing system.

Running Tests

It is recommended to run tests using 1 openmp thread; from within the build directory, you can run the testing suite as follows:

pn1512687:~/work/tusas_git/build$ OMP_NUM_THREADS=1 ctest

Additionally, you may use these features of ctest to specify certain conditions you may want.

  1. $ OMP_NUM_THREADS=1 ctest —rerun-failed

    This will show the results of only the tests that failed in the last execution of ctest 
  2. $ OMP_NUM_THREADS=1 ctest -R MyTest
    This will show the results of only the tests that contain the phrase “MyTest” in their title.
  3. $ ctest —help
    For more ctest information.

Testing coverage

Issues on LANL HPC Machines (or building with ACML or MKL)

Documentation

Building documentation

Building the Tusas documentation currently requires doxygen 1.8.12:
http://www.stack.nl/~dimitri/doxygen/download.html
and graphviz-2.40:
http://www.graphviz.org/Download..php
To build the doxygen generated documentation, change to the doc directory, execute:

$ cmake .

$ make doc

The doxygen generated files will be located at doc/html/index.html.

DEPRECATED instructions for NEMESIS interface and GPU branch — DEPRECATED

Building Tusas with full OpenMP support for CPU — DEPRECATED

The process described above builds Tusas with limited OpenMP support. The default is loop level parallelization. To build with OpenMP enabled residual and Jacobian fills, configure Tusas as follows:

$ cmake -DTRILINOS_DIR=/Users/cnewman/src/trilinos-12.6.4-Source/GCC_6_2_0_MPI_OMP_OPT/ -DTUSAS_CXX_FLAGS=-DTUSAS_COLOR_CPU .

Note that many tests will diff in this configuration, as the tests were generated with the default configuration.

Building Tusas with full OpenMP offloading support for GPU — DEPRECATED

Currently OpenMP offloading support for GPU is being tested on guido.lanl.gov. To build it is necessary to use the build script on guido. This is due to cmake not knowing about multiple targets at this time.

/usr/bin/mpicxx -v -DTUSAS_COLOR_GPU -flto -fopenmp -foffload=nvptx-none="-lm" \
-I. -I/home/cnewman/src/trilinos-12.6.4-Source/GCC_7_2_0_MPI_OPT//include -I./mesh/include \
-I./basis/include -I./preconditioner/include -I./timestep/include -I./input/include \
-I./error_estimator/include -I./elem_color/include -I./post_process/include -I./periodic_bc/include \
-I./projection/include -I../src/install/include \
tusas.cpp basis/basis.cpp mesh/Mesh.C input/ReadInput.cpp elem_color/elem_color.cpp periodic_bc/periodic_bc.cpp \
post_process/post_process.cpp error_estimator/error_estimator.cpp -o tusas.x \
-L/home/cnewman/src/trilinos-12.6.4-Source/GCC_7_2_0_MPI_OPT//lib \
-lnoxepetra -lnox -lstratimikos -lstratimikosbelos \
-lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 \
-lnemesis -lexodus \
-lbelosepetra -lbelos -lml -lifpack -lamesos -laztecoo -lisorropia \
-lthyraepetra -lthyracore -lepetraext -ltpetraext -ltpetrainout -ltpetra \
-ltpetraclassic -lzoltan -lepetra -lrtop \
-lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore \
-lkokkoscore \
/usr/lib/x86_64-linux-gnu/libX11.so /home/cnewman/src/install/lib/libnetcdf.so /usr/lib/liblapack.so.3 /usr/lib/libblas.so.3
Clone this wiki locally