Issue: FAST.Farm MPI Launch Fails Due to Single Process Spawned

[lorenzoschena]

Hello there,

When running the example_fastfarm.py after installation (and ensuring path_to_simulator set to a valid FAST.Farm executable), the simulation crashes at runtime with:

 This is not a spawned process. Using MPI_COMM_WORLD
Abort(604662278) on node 0 (rank 0 in comm 0): Fatal error in internal_Send: Invalid rank, error stack:
internal_Send(124): MPI_Send(buf=0x64441490f7fc, count=1, MPI_INT, 1, 0, MPI_COMM_WORLD) failed
internal_Send(78).: Invalid rank has value 1 but must be nonnegative and less than 1

It seems that this may happen because FAST.Farm is spawned with only one MPI process (rank 0). However, looking at the source code I understand that in order to run the case successfully we need to ensure a communication between two MPI ranks (0 and 1).

I wonder if this issue could be related to a missing explicit build flag during the FAST.Farm installation phase, or if I am doing something wrong when installing the toolbox.

Environment
• FAST.Farm version: v3.5.3-dirty (MPI active: Yes)
• WFCRL latest master (April 2025)
• Linux machine (no sudo access)
• Conda environment with OpenFAST, FAST.Farm and requirements installed
• Correct LD_LIBRARY_PATH and PATH
• MPI version: OpenMPI 4.1.x

To reproduce the issue, it should be sufficient to follow what written in the README file in a unix environment:

1. pip install -r requirements.txt
2. pip install -e .
3. wfcrl-simulator fastfarm

Thank you for sharing your work.

Cheers,
Lorenzo

[cibeah]

Hello,

Thank you for your detailed report and sorry for the delayed answer ! I can't reproduce your error, but what seems to be happening is that the spawned process running FAST.Farm cannot identify its parent process on your system. So it instead assumes that it has been launched separately and tries to communicate with a rank 1 process that doesn't exist. Launching the 2 processes (FAST.Farm and any controller program sending commands via MPI) separately is indeed an option, but it is not the one used in wfcrl, where the FAST.Farm process should always be a spawned child.

This is the bit that fails:

parent_comm <- get_parent _intercommunicator()
if not parent_comm
          behave as a process launched separately, send info to the only other process running (1)
else
          behave as a spawned child process, send info to your parent (2)
end if

For some reason we are in (1) but should be in (2). Do you still have the same problem when running it with mpiexec (mpiexec -n 1 python examples/example_fastfarm.py) ? Can you spawn a child process and send data to its parent outside of wfcrl, for example with 2 Python processes ?

Regards,