MPIO SMV fails on windows

[johodges]

I have been seeing this issue a few times lately with MPIO creating a lock on the SMV file in windows. When I set SMV_PARALLEL_WRITE=F on the DUMP line it is able to run the model without issues. Have others been running into this as well? What is the logic for when MPIO is used in the SMV file write? Should we disable it by default in windows? I am a bit nervous that if we put out a minor release with it enabled by default without understanding why this is happening that we will get a lot of bug reports related to it.

[mcgratta]

We had this problem on a new linux computer here at NIST that we were helping to test. The problem, if I recall, was the disk, which at the time was not quite set up properly. We left the option to do the write serially, as had been done in the past.

How are you running this case? On a single computer, or multiple, What version of the OS?

[drjfloyd]

I had reported this earlier and it seemed to resolve itself.

#13669

[drjfloyd]

It only ever happened when using more than one process

[johodges]

This example I am running on a single windows machine with multiple processes. The windows machine is using this OS:

OS Version: Windows 11, 10.0.26100 Build 26100

I saw this same issue on my older machine with:

OS Version: Windows 10, 10.0.19045 Build 19045

I can try a few different models to see how reproducible this is once the current model finishes.

[mcgratta]

Marcos added parallel writing of the smv file. Previously, each process would write out the info for its mesh or meshes in serial. For large cases, this was becoming a problem. I assume you are writing directly to the computer's own disk and not some remote server?

[johodges]

That's correct, writing to the local disk.

[mcgratta]

Does this computer have 128 cores? How many does it have and is hyperthreading turned on?

[johodges]

96 physical cores on the machine with hyper threading currently enabled. In this case I was just running to verify a fix in GEOM so I was not concerned with the runtime efficiency on the machine.

I plan to do some benchmarking with and without hyperthreading to get a better understanding of the impacts on runtimes.

[marcosvanella]

We had an issue with the intel libs running on the home directory of a local cluster. This was in Linux and the code was hanging in the MPI-IO collective write for the .smv file. For that reason we added the SMV_PARALLEL_WRITE option, which when false rank 0 gather the information from all processes and writes the file. We were told not to use the home disk in this case and when we switched to the work disk in that machine we did not see the issue.
Jon can you compile the code with gnu/openmpi?

[johodges]

I haven't had this issue with a linux machine. This is on windows which we don't have a build target for gnu.

[drjfloyd]

When I had problems on my Win 10 machine it was using less physical high performance cores than my pc has.

[mcgratta]

I just tried to run a 36 mesh case with the release bundle

Notice that we compiled the release with ifx 25.0.0. We need to use 25.0.4, or later. We'll test again. In any case, my computer has only 6 cores and 1 socket.

[gforney]

latest oneapi version for windows is 25.0.1 https://www.intel.com/content/www/us/en/developer/tools/oneapi/hpc-toolkit-download.html

…

On Tue, Mar 4, 2025 at 10:25 AM Kevin McGrattan ***@***.***> wrote: I just tried to run a 36 mesh case with the release bundle image.png (view on web) <https://github.com/user-attachments/assets/da16f7ab-8ecd-48c1-aa01-3193522d842b> Notice that we compiled the release with ifx 25.0.0. We need to use 25.0.4, or later. We'll test again. In any case, my computer has only 6 cores and 1 socket. — Reply to this email directly, view it on GitHub <#14325 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC6UCRR5LIAMP7EYT5QNIE32SXAXZAVCNFSM6AAAAABYJP5JY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJYGAYTINBWGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> [image: mcgratta]*mcgratta* left a comment (firemodels/fds#14325) <#14325 (comment)> I just tried to run a 36 mesh case with the release bundle image.png (view on web) <https://github.com/user-attachments/assets/da16f7ab-8ecd-48c1-aa01-3193522d842b> Notice that we compiled the release with ifx 25.0.0. We need to use 25.0.4, or later. We'll test again. In any case, my computer has only 6 cores and 1 socket. — Reply to this email directly, view it on GitHub <#14325 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC6UCRR5LIAMP7EYT5QNIE32SXAXZAVCNFSM6AAAAABYJP5JY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJYGAYTINBWGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Glenn Forney

[marcosvanella]

We might as well make the non parallel write the default. We've had fails on three different Windows machines.