MPI exchange of particles timed out with HT3D

[johodges]

Describe the bug
This is a bug I mentioned in passing a couple of weeks ago that took me awhile to reproduce with a simple case. The attached case fails with ERROR(123): MPI exchange of particles timed out when the obstructions are using HT3D. The reason for the timeout is due to one mesh waiting in INITIALIZE_BACK_WALL_EXCHANGE(2) while another mesh is still working through INITIALIZE_BACK_WALL_EXCHANGE(1) . If I add an MPI_BARRIER between calls 1 and 2 it initializes with no issue. Two questions:

1. Do you see any issue adding an MPI_BARRIER between the two calls to INITIALIZE_BACK_WALL_EXCHANGE?
2. Is there a way we can restructure the error message to identify that the timeout was due to HT3D rather than particles?

bess_simple3_ht3d.txt

Desktop (please complete the following information):

• OS: Windows 11
• Version: Latest master

[mcgratta]

Because the barrier call is made during the initialization phase, it is OK to add it. We try to avoid barriers during the actual time-stepping phase. As for the error message, there are several quantities exchanged initially that use the same buffers for MPI exchange. I'll see what I can do about the message. In the meantime, add the barrier and commit.

[johodges]

Added the barrier in #14901

[mcgratta]

I added more specific error codes for these MPI timeouts.