Grackle Cooling

The dependence of grackle cooling on a couple of libraries (grackle and HDF) make building this module more cumbersome. If everything is installed correctly then ./configure --enable-cooling=grackle should work. If not you will get some hints on what needs to be done. Here are more detailed notes on what to do.

Obtain the libraries

Grackle

The ChaNGa grackle cooling module was updated to grackle v3.1.1. There are likely API changes in the grackle library before or after this version. See the Grackle documentation for details.

HDF5

See the Grackle documentation for HDF requirements, download and build instructions.

Build Grackle

This should be a straightforward configure in the grackle main directory then in src/clib a make machine-XXX where XXX is the appropriate machine name (e.g. "linux-gnu" for a generic linux cluster) and then make. See the Grackle documentation.

Build ChaNGa

The configure may need environment variables set to link to the grackle and HDF libraries. A configure line (in bash) might look like:

CPPFLAGS="-I ../grackle/src/clib" LDFLAGS="-L$HOME/lib -L$HOME/local/lib" ./configure --enable-cooling=grackle

where the CPPFLAGS points to the grackle source directory to find grackle.h and the two LDFLAGS arguments point to the directories where libhdf5.a and libgrackle.a are respectively located.

Run ChaNGa

HDF5 and grackle may build shared libraries, in which case the runtime linker also needs to know where the libraries are located. This is done with the LD_LIBRARY_PATH environment variable. E.g. with the above LDFLAGS arguments, ChaNGa will need to be run with:

LD_LIBRARY_PATH=$HOME/lib:$HOME/local/lib ./ChaNGa

to link in the HDF5 and grackle libraries. Be aware that setting this up in a batch submission script can require some care.

Options

The grackle cooling module adds a few run-time options to ChaNGa that get passed to the grackle library. See the Grackle parameter documentation for more details. They include:

• grackle_data_file (command line -grackle_data_file): file name of the HDF5 data file containing the cooling data. The default value is: CloudyData_UVB=HM2012.h5. The file path should be accessible from the directory in which ChaNGa is being run.
• grackle_verbose (command line: +grackle_verbose: boolean to make the grackle code verbose. Default is 0 (off).
• use_grackle (command line +use_grackle): Turn on the grackle machinery. Default is 1 (on).
• with_radiative_cooling (command line +with_radiative_cooling): See the grackle documentation. Default is 1 (on).
• primordial_chemistry (command line -primordial_chemistry): See the grackle documentation. Default is 0.
• metal_cooling (command line +metal_cooling): See the grackle documentation. Default is 1.
• UVbackground (command line +UVbackground): See the grackle documentation. Default is 1.

Machine Specific Grackle Instructions

PSC Bridges-2

Begin by loading the following modules: phdf5/1.12.1-mvapich2-2.3.5-gcc8.3.1 mvapich2/2.3.5-gcc8.3.1 python/2.7

In the grackle directory, build grackle with the command:

cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=. -DBUILD_SHARED_LIBS=ON -B build

with appropriate changes to the desired install directory (.) and build directory (build). Then run

cmake --build build\
cmake --install build\

again replacing "build" with the chosen build directory.

Next, in the charm++ directory, build charm with:

./buildold ChaNGa verbs-linux-x86_64 smp -j8 --with-production

Finally, in the ChaNGa directory, configure ChaNGa with:

export CPPFLAGS="-I (path to grackle "include" directory)" 
export LDFLAGS="-L(path to the grackle "lib64" directory) -L/opt/packages/phdf5/1.12.1-mvapich2-2.3.5-gcc8.3.1/lib" 
./configure --enable-cooling=grackle

and any other desired ChaNGa options. The path to 1.12.1-mvapich2-2.3.5-gcc8.3.1/lib is very important, as ChaNGa will link to a different version of hdf5 and produce errors at runtime if this is not specified. Finally:

make

In the slurm script, be sure to load phdf5/1.12.1-mvapich2-2.3.5-gcc8.3.1 and mvapich2/2.3.5-gcc8.3.1, and include the line:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"(path to the grackle "lib64" directory)"

before calling ChaNGa.