prov/lnx: multi-rail messages delivered out of order

[RaymondMichael]

Describe the bug
When the LNX provider is using multiple NICs, messages can be delivered out of order. The lnx_select_send_endpoints() function round robins between the NICs. If message A is sent on NIC 0, then message B is sent on NIC 1. The problem is that nothing prevents message B from arriving at the destination and being matched before message A.

To Reproduce
export FI_LNX_PROV_LINKS="shm+cxi"
Run the following MPI program on two nodes with one rank each. I ran this with Open MPI 5.0.8, but the version shouldn't matter.

#include <mpi.h>
#include <stdlib.h>
#include <stdio.h>

#define TRANSFERS (1024 * 1024)

static int sdata[TRANSFERS];
static int rdata[TRANSFERS];

int
main(int argc, char *argv[])
{
	int rank, size, bad_count = 0;

	for (int i = 0; i < TRANSFERS; i++) {
		sdata[i] = i;
	}

	MPI_Init(&argc, &argv);
	MPI_Comm_rank(MPI_COMM_WORLD, &rank);
	MPI_Comm_size(MPI_COMM_WORLD, &size);
	MPI_Barrier(MPI_COMM_WORLD);

	for (int i = 0; i < TRANSFERS; i++) {
		if (rank == 0) {
			MPI_Send(&sdata[i], 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
		} else {
			MPI_Recv(&rdata[i], 1, MPI_INT, 0, 0, MPI_COMM_WORLD,
				  MPI_STATUS_IGNORE);
		}
	}

	if (rank == 1) {
		for (int i = 0; i < TRANSFERS; i++) {
			if (rdata[i] != i)
				bad_count++;
		}

		fprintf(stderr, "Bad count %d %f\n",
				bad_count, (float)bad_count / TRANSFERS);
	}

	MPI_Finalize();
	return 0;
}

On my current system I'm seeing a 0.25% error rate, but results will vary.

[aingerson]

Thanks, @RaymondMichael. We are aware of this and are working on a solution.
@amirshehataornl

[amirshehataornl]

One potential solution for this is to round robin per destination. This way all messages heading to the same destination goes over the same NIC, but messages heading to different destination can go over different NICs. That would maintain the message order to the same destination, but not different destinations. Would that satisfy the send-after-send requirement?

[amirshehataornl]

Another thought here is we can have different Multi-Rail policies. If the application doesn't require send-after-send semantics then we can round robin per message. There could exist multiple multi-rail policies and allow the user to select the one that best works for them.

[aingerson]

@amirshehataornl I think both of those sound like good solutions. Do you have the bandwidth to add that or do you want me to?

[amirshehataornl]

I'll take a crack at it. Will attempt to have something January next year