Significant memory consumption increase during merge seismograms after simulation

[dmitry-kabanov]

ASOFI3D uses int types everywhere, which does not allow seismogram files larger than 4(?) GB.
This occurs, at least I believe so, because on Intel Xeon machines ints are 32-bit, hence upper bound for int is 2 billion.

Possible solutions:

• Switch to using 64-bit integers using stdint.h header and type int64_t.

How to reproduce:

• Prepare a config that induces very large seismograms: long time and large number of receivers
• Simulate
• Observe that the simulation crashes at some point.

[dmitry-kabanov]

Investigation together with @vkazei proved tha the problem does not have anything to do with the overflow in 32-bit ints as we successfully ran a simulation with 5 GB seismograms (which, for 32-bit single-precision floats, are the arrays with 10^20 elements).

Studying how seismograms are written to disk, it was found that the procedure is not very memory-efficient, as each MPI process allocates memory for the full seismograms, that leads to ~2x memory requirements during the dumping stage comparing to the simulation stage.

So, probably, the error is due to the very high memory requirements and the seismograms writing should be more memory friendly.