Debugging "Warning: Preferred DataPlane "mpi" not found."

[rickybalin]

Hello,

I am working with the SST engine and trying to get the MPI Data Plane to work, but I always see "Warning: Preferred DataPlane "mpi" not found." and then another Data Plane is selected for me. I am trying to stream data between two mpi4py programs, and I can do so with WAN and RDMA planes, but not MPI. I wanted to ask a few questions here to see if I can understand more about the warning message:

• Are there any restrictions with mpi4py programs and the MPI data plane?
• To use the MPI data plane, do I have to launch the applications in MPMD mode? I tried and it did not make a difference.
• When I set SstVerbose to values greater than 1, I don't see any other information as to why the MPI data plane cannot be used. Is there something in the logs I missed?
• I built ADIOS2 from source and in the configure log I see the following. Does it suggest I was able to build the SST engine with MPI support?

    Build Options:
      DataMan            : OFF
      DataSpaces         : OFF
      HDF5               : OFF
      HDF5_VOL           : OFF
      MHS                : ON
      SST                : ON
      Fortran            : ON
      MPI                : ON
      Python             : ON
...
Possible RDMA DataPlanes for SST:  fabric MPI

Note that I am setting MPI4PY_RC_THREADS=TRUE and MPI4PY_RC_THREAD_LEVEL="multiple" to ensure MPI is initialized with multi-thread enabled.

Thank you in advance for any feedback on this issue!

Riccardo

[eisenhauer]

Hi Riccardo! There are restrictions to using the MPI data plane because it requires a 1) an mpi implementation that supports mpi client/server, and 2) mpi to be initialized in thread-safe mode (I.E. with MPI_THREAD_MULTIPLE). We can't know those things until run-time, so we do look for them and then fallback to alternative dataplanes if our requirements are not satisfied. That said, requirement 1 is not something we can actually test trivially, so we have an (probably overly) conservative fallback of testing to see if the MPI implementation is MPICH. If you have a non-MPICH MPI, you should be able to override this by forcing enabling the MPI DP which is passing the flag`-DADIOS2_HAVE_MPI_CLIENT_SERVER=ON' to CMake when building ADIOS. Can you give this a try?

[rickybalin]

Hi @eisenhauer, thanks for the quick reply! I can try again with that flag, maybe on our Polaris machine instead of Sunspot to use plain MPICH, but I did already test with that and with suggestions I found here and here, but saw the same issue. Do you have any pointers on where these runtime checks are taking place so perhaps I can understand better why they are not passing?

[eisenhauer]

Hmm. I'm looking at the code and wondering if something might have broken this. Some things that come into play are MpiGetPriority() in adios/source/adios2/toolkit/sst/cp/mpi_dp.c and the dataplane selection code in dp.c. WIth SSTVerbose=5, you should be getting a trace message like "MPI DP disabled since the following predicate is false:" if SST doesn't think that the MPI dataplane is viable. If you're seeing that, it might give us a clue.

[rickybalin]

Sounds good, I'll take a look at those files and see if I can understand why MPI is being discarded. I'll be able to rebuild and run with SSTVerbose=5 later today and report an update. Thanks again!

[rickybalin]

Hi @eisenhauer , I was able to get the MPI DataPlane to be selected thanks to your pointers. The CMake flag did not work, so I just commented the ifdef statements in source/adios2/toolkit/sst/dp/dp.c in order to force the SelectDP() function to add the MPI "possibility". With this change, I see "Considering DataPlane "mpi" for possible use, priority is 100" and "Selecting DataPlane "mpi" (preferred) for use" in the SST logs.

I am still running into an issue though when running the application with the MPI DataPlane. I open three streams throughout the run (is this bad practice?) -- the first one just sends some "meta data" to synchronize the two components, and this transfer completes with the MPI DataPlane just fine. The second stream produces a seg fault. Core files (below) suggest the issue is in in libmpi.so at internal_Recv(), but this was Sunspot again so will try on Polaris to see if I get the same error.

#0  0x0000154df422ba02 in internal_Recv ()
   from /opt/aurora/24.180.1/CNDA/mpich/20240717/mpich-ofi-all-icc-default-pmix-gpu-drop20240717/lib/libmpi.so.12
[Current thread is 1 (Thread 0x154de7eb4700 (LWP 169591))]
#0  0x0000154df422ba02 in internal_Recv ()
   from /opt/aurora/24.180.1/CNDA/mpich/20240717/mpich-ofi-all-icc-default-pmix-gpu-drop20240717/lib/libmpi.so.12
#1  0x0000154dd8f52264 in MpiReadReplyHandler (cm=0x55daeaad2cd0, conn=<optimized out>, msg_v=0x154dd400c840,
    client_Data=0x154dd907a860 <Svcs>, attrs=<optimized out>)
    at /gila/Aurora_deployment/balin/ALCF-4/build/ADIOS2/source/adios2/toolkit/sst/dp/mpi_dp.c:656

Thanks again for the help!

[eisenhauer]

Opening multiple streams shouldn't be a killer problem, but it does carry extra overheads that might be best avoided. That you're getting this segfault may indicate that starting a new stream after closing the first might be a problem in the MPI dataplane. Maybe try reusing the first stream instead?

@vicentebolea I know you don't have much time on ADIOS at the moment, but any thoughts on the 2nd stream issue with the MPI DP?

[rickybalin]

Reporting more information in case it is useful... Using a single stream vs. two produces the same issue, which I narrowed down specifically to the difference between sending scalar variables vs. arrays.

The following works (myVar = 10):
stream.write("y", myVar) on the producer
stream.read("y") on the consumer

The following does not work (myArray = np.array([0.0, 1.0], dtype=np.float32); nx = len(myArray)):

stream.write("y", myArray, [size * nx], [rank * nx], [nx])

on the producer and

var = stream.inquire_variable("y")
shape = var.shape()
count = int(shape[0] / size)
start = count * rank
if rank == size - 1:
    count += shape[0] % size
train_data = stream.read("y", [start], [count])

on the consumer.

[eisenhauer]

Hmm. I don't use the Python API much, but given the segfault in the receive and that the read must be allocating a destination buffer, I'd guess that something is going wrong in the geometry here and that we're trying to fill a larger memory region than has been allocated. I'm not sure if that would be a problem in the python bindings, if your code is wrong in some subtle way, or if there's something else going on. I wouldn't think it would be your code, because if we're getting a count array, whatever size that specifies should be what we're reading. Ah, one possibility: In the C++ API lots of templating and type rigidity makes sure that we don't get type mismatches. That gets complicated in Python. Years ago I saw one other circumstance where somehow we were reading floats into what Python thought was an array of doubles, giving trash data. If somehow the opposite was happening, I.E. the reader was allocating a buffer for N floats, but somehow the C++ level saw this as a read for N doubles, we might get this buffer overflow and segfault. Playing with the dtype spec might help sort if something like that was an issue.

[rickybalin]

Thanks for the suggestion! Playing with the data type didn't change the outcome, but I guess the definitive test to isolate possible issues with the Python API is to reproduce the example in C++. Another option is to run the Python example on Frontier to isolate possible system-specific issues -- I see there are ADIOS-2 modules I can load which hopefully have the MPI DataPlane enabled. I will try those options this week. Thanks again!

[eisenhauer]

If you have a reasonably small test case, I'm also happy to run it myself to debug...

[rickybalin]

Thanks for the offer, I attached a tarball with a C++ and Python example. On Sunspot, I was able to reproduce the issue with the C++ example, which is a slight modification of one ADIOS-2 examples, making it seem like a system specific issue. But I was also not able to confirm functionality on Frontier because I think the latest module does not have the proper support for the MPI data plane since I was seeing the same set of issues we discussed above. I tried to build from source, but ran into errors (/usr/lib64/gcc/x86_64-suse-linux/12/../../../../x86_64-suse-linux/bin/ld: ../../../../lib64/libadios2_core.so.2.10.0: undefined reference to fi_freeinfo@FABRIC_1.7'`). If you have some instructions for building ADIOS-2 on Frontier I can test the reproducers further.

Thanks!

[eisenhauer]

I don't think i see the attachment...

[rickybalin]

You are right, sorry about that. Here it is.
Uploading adios2_example.tar.gz…

[rickybalin]

You are right, sorry about that. Here it is.
adios2_example.tar.gz

[eisenhauer]

Just wanted to give an update. I spent some time working on Frontier trying to test this. The MPI dataplane should work on Frontier, but doesn't seem to. This is not your problem, but a different problem that we need to look at as well. Hoping to get to both of these, but it's a busy time of year.

[rickybalin]

Thanks for the update. I understand it's a busy time -- any help on this issue is greatly appreciated!

[vicentebolea]

In OLCF Frontier for some reason the client/server vinterface did not setup correctly when we run less than 4 ranks. This was something that showed up after a frontier MPI update. Can you try increase the number of ranks.

Vicente