[WIP] OpenMP + load balance debugging

[mrowan137]

A bug is revealed with OpenMP multithreading and load balancing with this commit: d9d9721
This branch is for locating and fixing the bug, and providing a test to ensure the currently broken case is monitored in the future.

The crash does not appear consistently and may take multiple runs to appear (I've had ~15 before, but it is often less), which may be tricky to catch in a regression test. The attached input (similar to the regression test) and can show the bug running on Summit CPUs:

export OMP_NUM_THREADS=7
jsrun -r 2 -a 1 -c 7 -l CPU-CPU -d packed -b rs <executable> <inputs> > output.txt

inputs_bug
Backtrace.1.0

[ax3l]

@mrowan137 thank you - can you please post the backtrace file in debug mode, too?

In CI, this test does not seem to crash - it just lacks the reduced_diags_loadbalancecosts_heuristic_omp.json file.

[ax3l]

Careful in public CI services: We only have two vCores available - so using 8 will likely trigger a watchdog deamon.
Does this also crash locally? I just ran it locally and could not crash it (in Debug mode).

The crash does not appear consistently and may take multiple runs to appear
... maybe need to try more often ...

The PR description bugtrace points to the destructor

38: ./main3d.gnu.DEBUG.TPROF.MTMPI.OMP.ex() [0x1002dd44]
    WarpX::~WarpX() at /ccs/home/mrowan/code/warpx_directory_mrowan137/WarpX/./Source/WarpX.cpp:318

which is problematic but also not 100% the issue we saw during load balancing itself on CPU, I think.

Side note: working with OLCF & NERSC to get some beefier CI machines for such tests integrated...

[ax3l]

@atmyers With the commit found by @mrowan137 above pointing to d9d9721 (#1036), is it possible that the issue is rooted in CopyParticleAttribs that unconditionally accesses all particle positions in 2D/RZ?

https://github.com/ECP-WarpX/WarpX/blob/75931e8a5527f77515657260883f1ad9767210fa/Source/Particles/Pusher/CopyParticleAttribs.H#L60-L62
https://github.com/ECP-WarpX/WarpX/blob/75931e8a5527f77515657260883f1ad9767210fa/Source/Particles/Pusher/CopyParticleAttribs.H#L91-L93

[mrowan137]

@atmyers @ax3l [just to chime in on @ax3l 's suggestion, I confirm from my own testing these are indeed the lines where if uncommenting step by step, we first transition from no hang to hang]

[atmyers]

It looks like TmpParticles always has 3 components for x, y, z, even in 2D / RZ. So I don't think there is an out-of-bounds memory access. However, looking at that code, all the map entries need to already be created before we do xpold = tmp_particle_data[lev][index][TmpIdx::xold ].dataPtr() + a_offset; in a threaded region. If they aren't, and those lines are modifying the structure of the map inside an omp parallel region, then that is definitely a race condition.

If you change it to xpold = tmp_particle_data[lev].at(index)[TmpIdx::xold ].dataPtr() + a_offset; do you get an out-of-range error instead of a hang?

[atmyers]

I think this should fix the issue: #1658

[mrowan137]

I think this should fix the issue: #1658

Awesome, thanks @atmyers ! This fixes the hang in my testing.

As for a test to monitor this in the future, @ax3l how might we do this with the constraint of only two vCores? I think we need more than 1 MPI rank (for load balancing) and at least 2 threads per rank to catch this (for OpenMP). furthermore, the test does not hang consistently, wouldn't we need to rerun several times?

[ax3l]

Discussed today: hm, this might be a good candidate for Cori tests...

We could write a new ini file that runs tests on a Cori node, but we cannot run them yet on a per-PR basis...

[RemiLehe]

@ax3l Should this be merged or closed?