-
Notifications
You must be signed in to change notification settings - Fork 28
Shannon GPU Cluster
Earlier issues resloved thanks to Si at SNL:
To build and run.
Need -tp=nvidia in Makefile for PGI i.e
FLAGS_PGI = -fastsse -Mipa=fast -Mlist -ta=nvidia
Released compiler and MPI is: module load pgi/13.9.0 openmpi/1.6.4/pgi/13.9.0
Also:
module load pgi/15.0.0 openmpi/1.8.3/pgi/15.0.0
But this is under non-disclosure - DO NOT PUBLISH RESULTS
make COMPILER=PGI MPI_COMPILER=mpif90 C_MPI_COMPILER=mpicc
K80: salloc -N 1 -p stella; mpirun -np 1 ./clover_leaf (Get 1 of dual K80 GPU) K40: salloc -N 1 -p atlas; mpirun -np 1 ./clover_leaf K20x salloc -N 1 -p pbatch; mpirun -np 1 ./clover_leaf
Problems with PGI compiler:
For all compiler builds from 13.1 to 13.7.0 then 13.9.0 to 14.7.0 get:
PGC-F-0206-Can't find include file stddef.h (/home/projects/pgi/13.9.0/linux86-64/13.9/include/stddef.h: 16) PGC/x86-64 Linux 13.9-0: compilation aborted
With any C file I try to compile. Try it with simple hello.c
Tried some of the later builds but getting a different problem. Looks like install isn’t complete for 13.8.0, 14.9.0 and 15.0.0. For the latter two the PATH prepend in the module(s):
/home/projects/x86-64/pgi/2015/15.0.0/linux86-64/15.0/bin and /home/projects/x86-64/pgi/2014/14.9.0/linux86-64/14.9/bin
Doesn’t match reality:
/home/projects/pgi/15.0-install/linux86-64/15.0/bin/ and ?
13.8.0 gives me “please run makelocalrc to complete installation ” – (as does 15.0.0 if I explicitly use the full path)
Something changed recently, as I was using 13.9.0 for a while without any issues (Used for OpenACC paper at SC14)
Get onto a compute node:
Either k40c Atllas:
salloc -N 1 -p atlas
or
k20x
salloc –N 1 –p pbatch
PGI Compiler
module load pgi/13.10.0 module load mvapich/2.0.0-beta/pgi/13.10/cuda-5.5.11
make COMPILER=PGI MPI_COMPILER=mpif90 C_MPI_COMPILER=mpicc NV_ARCH=KEPLER
/home/projects/mvapich/2.0.0/pgi/13.10/lib/libmpich.a(topology-linux.o): In function hwloc_linux_set_area_membind': /tmp/mvapich2-2.0a/src/mpid/ch3/channels/mrail/src/hwloc/src/./topology-linux.c:1173: undefined reference to mbind'
/tmp/mvapich2-2.0a/src/mpid/ch3/channels/mrail/src/hwloc/src/./topology-linux.c:1154: undefined reference to mbind' /home/projects/mvapich/2.0.0/pgi/13.10/lib/libmpich.a(topology-linux.o): In function hwloc_linux_set_thisthread_membind':
/tmp/mvapich2-2.0a/src/mpid/ch3/channels/mrail/src/hwloc/src/./topology-linux.c:1231: undefined reference to migrate_pages' /tmp/mvapich2-2.0a/src/mpid/ch3/channels/mrail/src/hwloc/src/./topology-linux.c:1244: undefined reference to set_mempolicy'
/tmp/mvapich2-2.0a/src/mpid/ch3/channels/mrail/src/hwloc/src/./topology-linux.c:1221: undefined reference to set_mempolicy' /home/projects/mvapich/2.0.0/pgi/13.10/lib/libmpich.a(topology-linux.o): In function hwloc_linux_get_thisthread_membind':
/tmp/mvapich2-2.0a/src/mpid/ch3/channels/mrail/src/hwloc/src/./topology-linux.c:1342: undefined reference to get_mempolicy' /home/projects/mvapich/2.0.0/pgi/13.10/lib/libmpich.a(topology-linux.o): In function hwloc_linux_get_area_membind':
/tmp/mvapich2-2.0a/src/mpid/ch3/channels/mrail/src/hwloc/src/./topology-linux.c:1415: undefined reference to get_mempolicy' /home/projects/mvapich/2.0.0/pgi/13.10/lib/libmpich.a(topology-linux.o): In function hwloc_linux_find_kernel_max_numnodes':
/tmp/mvapich2-2.0a/src/mpid/ch3/channels/mrail/src/hwloc/src/./topology-linux.c:1276: undefined reference to `get_mempolicy'
GNU Compiler
module load gcc/4.7.2 module load mvapich/2.0.0-beta/gnu/4.4.6/cuda-5.5.11
make COMPILER=GNU MPI_COMPILER=mpif90 C_MPI_COMPILER=mpicc NV_ARCH=KEPLER
Compilers ok (as long as on compute node) but at runtime:
Clover Version 1.000 MPI Version OpenMP Version Task Count 1 Thread Count: 1
Clover Version 1.000 MPI Version OpenMP Version Task Count 1 Thread Count: 1
Output file clover.out opened. All output will go there. Step 1 time 0.0000000 control div timestep 4.00E-02 1, 102 x -5.00E-02 y 1.01E+01 Operating system error: Cannot allocate memory Memory allocation failed
Trying with different mvapich with older cuda:
module load mvapich/1.9.0/gnu-4.4.6/cuda-5.5.11
Get same issue at run time
However if compiler for FERMI, at runtime get:
Clover Version 1.000 MPI Version OpenMP Version Task Count 1 Thread Count: 16
Clover Version 1.000 MPI Version OpenMP Version Task Count 1 Thread Count: 16
Output file clover.out opened. All output will go there. terminate called after throwing an instance of 'thrust::system::system_error' what(): invalid device function [k20-0003:mpi_rank_0][error_sighandler] Caught error: Aborted (signal 6) Aborted
Intel Compiler
module load intel/13.2.146 module load mvapich/1.9.0-beta/intel-13.2.146/cuda-5.0.0
make COMPILER=INTEL MPI_COMPILER=mpif90 C_MPI_COMPILER=mpicc NV_ARCH=KEPLER
Fails at link with:
ld: cannot find –lcudart