temporarily disable hypre which seems to be causing crashes with CUDA…

[s-sajid-ali]

… builds

[s-sajid-ali]

Similar error encountered with HYPRE on wc.fnal.gov (previous error was observed on perlmutter.nersc.gov):

(gdb) run
Starting program: /wclustre/accelsim/sajid/packages/synergia2/build_v100_debug/examples/fodo_cxx/fodo_cxx 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: File "/srv/software/el8/x86_64/eb/GCCcore/12.3.0/lib64/libstdc++.so.6.0.30-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path /srv/software/el8/x86_64/eb/GCCcore/12.3.0/lib64/libstdc++.so.6.0.30-gdb.py
line to your configuration file "/nashome/s/sasyed/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/nashome/s/sasyed/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
[New Thread 0x7ffe38002000 (LWP 4109036)]
[New Thread 0x7ffe364dd000 (LWP 4109052)]
[New Thread 0x7ffe34c8e000 (LWP 4109054)]
[Detaching after fork from child process 4109055]
[Detaching after fork from child process 4109056]
[Thread 0x7ffe34c8e000 (LWP 4109054) exited]
[New Thread 0x7ffe34c8e000 (LWP 4109060)]
Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set
  In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads
  For best performance with OpenMP 3.1 set OMP_PROC_BIND=true
  For unit testing set OMP_PROC_BIND=false

MPI detected: For OpenMP binding to work as intended, MPI ranks must be bound to exclusive CPU sets.

[New Thread 0x7ffe28fff000 (LWP 4109061)]
gridx: 32
gridy: 32
gridz: 128
macroparticles: 1048576
real_particles: 2.94e+12
Read lattice, length: 20, 4 elements
[begin,1]  quadrupole fodo_1: at=0, k1=0.07142857142857142461, l=2, yoshida_order=2, propagator_type=yoshida
[1,end]  quadrupole fodo_1: at=0, k1=0.07142857142857142461, l=2, yoshida_order=2, propagator_type=yoshida
[begin,4]  drift fodo_2: at=2, l=8
[4,end]  drift fodo_2: at=2, l=8
[begin,1]  quadrupole fodo_3: at=10, k1=-0.07142857142857142461, l=2, yoshida_order=2, propagator_type=yoshida
[1,end]  quadrupole fodo_3: at=10, k1=-0.07142857142857142461, l=2, yoshida_order=2, propagator_type=yoshida
[begin,4]  drift fodo_4: at=12, l=8
[4,end]  drift fodo_4: at=12, l=8
Statistics before propagation
bunch means
0:  -6.14098886759368e-20
1:  -7.46447784767852e-21
2:  -2.51992301808154e-20
3:  -1.00585162486448e-21
4:  -1.27054942088145e-21
5:  4.10281583826302e-22
bunch stds
0:  0.0055235626163768
1:  0.000437742121860758
2:  0.0027478521442124
3:  0.000437742121860758
4:  0.0128170229171487
5:  0.0001
Statistics before propagation
bunch means
0:  -6.14098886759368e-20
1:  -7.46447784767852e-21
2:  -2.51992301808154e-20
3:  -1.00585162486448e-21
4:  -1.27054942088145e-21
5:  4.10281583826302e-22
bunch stds
0:  0.0055235626163768
1:  0.000437742121860758
2:  0.0027478521442124
3:  0.000437742121860758
4:  0.0128170229171487
5:  0.0001
Propagator: starting turn 1, final turn 4


Thread 1 "fodo_cxx" received signal SIGSEGV, Segmentation fault.
0x00007fffe64f1677 in MatSetValuesCOO_SeqAIJ (A=0x595f750, v=0x7ffdf6000000, imode=INSERT_VALUES)
    at /tmp/sasyed/spack-stage/spack-stage-petsc-3.20.3-acdg5xx7cw2db7i5pgfhhiqpqlu4ou76/spack-src/src/mat/impls/aij/seq/seqhashmat.h:4779
4779	/tmp/sasyed/spack-stage/spack-stage-petsc-3.20.3-acdg5xx7cw2db7i5pgfhhiqpqlu4ou76/spack-src/src/mat/impls/aij/seq/seqhashmat.h: No such file or directory.
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-236.el8.7.x86_64 nvidia-driver-NVML-545.23.08-1.el8.x86_64 nvidia-driver-cuda-libs-545.23.08-1.el8.x86_64
(gdb) bt
#0  0x00007fffe64f1677 in MatSetValuesCOO_SeqAIJ (A=0x595f750, v=0x7ffdf6000000, imode=INSERT_VALUES)
    at /tmp/sasyed/spack-stage/spack-stage-petsc-3.20.3-acdg5xx7cw2db7i5pgfhhiqpqlu4ou76/spack-src/src/mat/impls/aij/seq/seqhashmat.h:4779
#1  0x00007fffe6613b16 in MatSetValuesCOO (A=0x595f750, coo_v=0x7ffdf6000000, imode=imode@entry=INSERT_VALUES)
    at /tmp/sasyed/spack-stage/spack-stage-petsc-3.20.3-acdg5xx7cw2db7i5pgfhhiqpqlu4ou76/spack-src/gcreate.c:781
#2  0x00007fffe638108f in MatConvert_AIJ_HYPRE (A=<optimized out>, type=<optimized out>, reuse=MAT_INITIAL_MATRIX, B=<optimized out>)
    at /tmp/sasyed/spack-stage/spack-stage-petsc-3.20.3-acdg5xx7cw2db7i5pgfhhiqpqlu4ou76/spack-src/mhypre.c:675
#3  0x00007fffe62c4710 in MatConvert (mat=<optimized out>, newtype=<optimized out>, newtype@entry=0x7fffe703ca00 "hypre", 
    reuse=MAT_INITIAL_MATRIX, M=M@entry=0x4789e68)
    at /tmp/sasyed/spack-stage/spack-stage-petsc-3.20.3-acdg5xx7cw2db7i5pgfhhiqpqlu4ou76/spack-src/matrix.c:4419
#4  0x00007fffe6b2caf9 in PCSetUp_HYPRE (pc=0x4788810)
    at /tmp/sasyed/spack-stage/spack-stage-petsc-3.20.3-acdg5xx7cw2db7i5pgfhhiqpqlu4ou76/spack-src/hypre.c:245
#5  0x00007fffe6a94e8d in PCSetUp (pc=0x4788810)
    at /tmp/sasyed/spack-stage/spack-stage-petsc-3.20.3-acdg5xx7cw2db7i5pgfhhiqpqlu4ou76/spack-src/precon.c:1080
#6  0x00007fffe6ccc2de in KSPSetUp (ksp=0x4785980)
    at /tmp/sasyed/spack-stage/spack-stage-petsc-3.20.3-acdg5xx7cw2db7i5pgfhhiqpqlu4ou76/spack-src/itfunc.c:415
#7  0x00007ffff679e700 in compute_mat (lctx=..., sctx=..., gctx=...)
    at /wclustre/accelsim/sajid/packages/synergia2/build_v100_debug/src/synergia/collective/tmpxft_003e4e54_00000000-6_space_charge_3d_fd_utils.cudafe1.cpp:420
#8  0x00007ffff67945da in Space_charge_3d_fd::apply_bunch (this=0x266ff50, bunch=..., time_step=6.9899537156101739e-09, logger=...)
    at /wclustre/accelsim/sajid/packages/synergia2/build_v100_debug/src/synergia/collective/tmpxft_003e4dd4_00000000-6_space_charge_3d_fd.cudafe1.cpp:199
#9  0x00007ffff6793fdb in Space_charge_3d_fd::apply_impl (this=0x266ff50, sim=..., time_step=6.9899537156101739e-09, logger=...)
    at /wclustre/accelsim/sajid/packages/synergia2/build_v100_debug/src/synergia/collective/tmpxft_003e4dd4_00000000-6_space_charge_3d_fd.cudafe1.cpp:150
#10 0x00007ffff716bdd5 in Operator::apply (this=0x266ff50, simulator=..., time_step=6.9899537156101739e-09, logger=...)
    at /tmp/bunch_simulator.h:64
#11 0x00007ffff716814c in Step::apply (this=0x218e3e0, simulator=..., logger=...)
    at /wclustre/accelsim/sajid/packages/synergia2/build_v100_debug/src/synergia/simulation/tmpxft_003e52d2_00000000-6_step.cudafe1.cpp:56
#12 0x00007ffff71e201c in Propagator::do_step (this=0x7fffffffc1c0, simulator=..., step=..., step_count=1, turn_count=0, logger=...)
    at /wclustre/accelsim/sajid/packages/synergia2/build_v100_debug/src/synergia/simulation/tmpxft_003e5cd5_00000000-6_propagator.cudafe1.cpp:50
#13 0x00007ffff71e28ac in Propagator::propagate (this=0x7fffffffc1c0, sim=..., logger=..., max_turns=4)
    at /wclustre/accelsim/sajid/packages/synergia2/build_v100_debug/src/synergia/simulation/tmpxft_003e5cd5_00000000-6_propagator.cudafe1.cpp:181
#14 0x0000000000861ed7 in run (opts=...)
    at /wclustre/accelsim/sajid/packages/synergia2/build_v100_debug/examples/fodo_cxx/tmpxft_003e8776_00000000-6_fodo_cxx.cudafe1.cpp:171
--Type <RET> for more, q to quit, c to continue without paging--
#15 0x00000000008622b6 in main (argc=1, argv=0x7fffffffcba8)
    at /wclustre/accelsim/sajid/packages/synergia2/build_v100_debug/examples/fodo_cxx/tmpxft_003e8776_00000000-6_fodo_cxx.cudafe1.cpp:195
(gdb)