heatmap_mpi_redux: Assertion `heatmap_runtime' failed

[wkliao]

When running a 64-node, 128-MPI-process-per-node job on Perlmutter at NERSC,
I encountered the following error.

darshan-heatmap.c:508: heatmap_mpi_redux: Assertion `heatmap_runtime' failed.
srun: error: nid005055: task 6363: Aborted (core dumped)

Core file trace:

#3  0x00007fa12866dca2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007fa12cb7550c in heatmap_mpi_redux (stdio_buf=<optimized out>, mod_comm=-2080374783, shared_recs=<optimized out>, shared_rec_count=<optimized out>)
    at darshan-heatmap.c:508
#5  0x00007fa12cb4351b in darshan_core_shutdown (write_log=<optimized out>) at darshan-core.c:657
#6  0x00007fa12cb3e674 in MPI_Finalize () at darshan-core-init-finalize.c:89

I am not sure what version of Darshan is loaded, as command "module list" shows

darshan/default                (L,D) 

Maybe we should add a command-line option, for example --version, to all Darshan utility programs.

[carns]

I agree about adding a --version to the utilities. Want to open a separate issue for that?

You can probably see more information about the perlmutter version with module show darshan.

As for the actual bug, there is only one assertion in that routine, and that is the one that checks if the global heatmap_runtime pointer is set. It had to have been at some point to reach that part of the code, and it isn't supposed to be cleared until the heatmap_cleanup() routine, which is called later.

This makes me wonder if the code has somehow triggered multiple (racing) shutdowns of Darshan or something. Otherwise it is possibly an unrelated memory corruption that just happens to exhibit here?

Is this reproducible?

[wkliao]

Instead of module show, I found command pkg-config can show
the Darshan's version on Perlmutter.

%  pkg-config --modversion darshan-runtime
3.4.6

It can be hard to reproduce, as this is the first time I encountered such
Darshan message. I ran 24 srun commands with the same executable,
one after another, in a single batch job and only one of them produce
this message (the srun completed fine). Not sure whether or not this
can "triggered multiple (racing) shutdowns of Darshan".

Maybe we just leave this issue open, in case someone encounter the
same in the future.

[wkliao]

I agree about adding a --version to the utilities. Want to open a separate issue for that?

PR #1051 has been created. It contains changes for darshan-parser only.
If it looks fine, I will add others.