An scary error! *** Process received signal ***

[mojtabaAmiraslanpour]

Hi,

I am running some simulations on a geometry other than the simple pipe model. While my simulations seem to be fine on smaller scales, the one with a larger scale crashes in the middle of the run with this scary error that I don't know anything about:

[mdc-1057-29-10:259521] *** Process received signal ***
[mdc-1057-29-10:259521] Signal: Segmentation fault (11)
[mdc-1057-29-10:259521] Signal code: Address not mapped (1)
[mdc-1057-29-10:259521] Failing at address: 0x556387a840cc
[mdc-1057-29-10:259521] [ 0] /lib64/libpthread.so.0(+0xf680)[0x2af5e6edd680]
[mdc-1057-29-10:259521] [ 1] ../pipeflow_interior_viscosity(_ZN4hemo21HemoCellParticleField11addParticleERKNS_16HemoCellParticle17serializeValues_tE+0x50c)[0x556345bb899c]
[mdc-1057-29-10:259521] [ 2] ../pipeflow_interior_viscosity(_ZN4hemo28HemoCellParticleDataTransfer7receiveERKN3plb5Box3DEPcjNS1_5modif6ModifTE+0xbb)[0x556345bac8cb]
[mdc-1057-29-10:259521] [ 3] ../pipeflow_interior_viscosity(_ZN4hemo28HemoCellParticleDataTransfer7receiveERKN3plb5Box3DEPcjNS1_5modif6ModifTENS1_5Dot3DE+0x115)[0x556345bad135]
[mdc-1057-29-10:259521] [ 4] ../pipeflow_interior_viscosity(_ZNK3plb27ParallelBlockCommunicator3D11communicateERNS_24CommunicationStructure3DERKNS_12MultiBlock3DERS3_NS_5modif6ModifTE+0xd38)[0x556345ccf688]
[mdc-1057-29-10:259521] [ 5] ../pipeflow_interior_viscosity(_ZNK3plb27ParallelBlockCommunicator3D17duplicateOverlapsERNS_12MultiBlock3DENS_5modif6ModifTE+0x330)[0x556345ccfc20]
[mdc-1057-29-10:259521] [ 6] ../pipeflow_interior_viscosity(_ZN4hemo14HemoCellFields13syncEnvelopesEv+0x176)[0x556345b9cd36]
[mdc-1057-29-10:259521] [ 7] ../pipeflow_interior_viscosity(_ZN4hemo8HemoCell7iterateEv+0x48b)[0x556345b7cc0b]
[mdc-1057-29-10:259521] [ 8] ../pipeflow_interior_viscosity(main+0x1647)[0x556345b3cd17]
[mdc-1057-29-10:259521] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2af5e75503d5]
[mdc-1057-29-10:259521] [10] ../pipeflow_interior_viscosity(+0xaf4d9)[0x556345b404d9]
[mdc-1057-29-10:259521] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 173 with PID 259521 on node mdc-1057-29-10 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
Script Execution Time: 3743.85 seconds
Date: Mon Oct 17 11:22:45 EDT 2022

Any help/hints are much appreciated. I don't have any idea how to proceed.

Mojtaba

[gzavo]

Hi @mojtabaAmiraslanpour,

That is a scary error, and not because of the length of the error message. A segmentation fault broadly means the program you run tries to write to a memory location that is not valid for the program. These are often highly specific, difficult to reproduce problems, arising from race conditions or other rare issues. You can follow up on the function names in the error message to get some hint where it happens in the code. Sometimes changing the decomposition or running the same setup on a different machine can help.

[mojtabaAmiraslanpour]

Thanks @gzavo
I changed blockSize from 25 to 27 and it seems to be working for now. I will see how it goes and report back.

[mojtabaAmiraslanpour]

It crashes again.

@gzavo I see in the HemoCell guide it is mentioned that:

Note These are minimal requirements, avoid OpenMPI 2.0.X as in our experience it introduces memory leaks.

I am using OpenMPI 4.0.2. Is this problematic?