prov/cxi: segfault with FI_MATCH_COMPLETE on small messages

[hzhou]

Describe the bug
We encounter this segfault issue on Aurora using MPICH's pipelining implementation. There is a reproducer in pmodels/mpich#7713. The backtrace from the segfault:

(gdb) bt
#0  0x00001533bdeb9db2 in dlist_insert_after (head=0x3e83680, item=0x3e83680) at ./include/ofi_list.h:81
#1  dlist_insert_before (head=0x3e170b8, item=0x3e83680) at ./include/ofi_list.h:88
#2  cxip_send_req_queue (req=0x3e83590, txc=0x3e14f20) at prov/cxi/src/cxip_msg_hpc.c:4982
#3  cxip_send_common (txc=0x3e14f20, tclass=<optimized out>, buf=0x0, len=<optimized out>, desc=<optimized out>,
    data=9007955168985100, dest_addr=11, tag=26388279066636, context=0x0, flags=33685504, tagged=true,
    triggered=false, trig_thresh=0, trig_cntr=0x0, comp_cntr=<optimized out>) at prov/cxi/src/cxip_msg_hpc.c:5224
#4  0x00001533bdeb13c2 in cxip_tinjectdata (fid_ep=<optimized out>, buf=<optimized out>, len=<optimized out>,
    data=<optimized out>, dest_addr=<optimized out>, tag=<optimized out>) at prov/cxi/src/cxip_msg.c:998
#5  0x00001533bf2b67c6 in fi_tinjectdata (ep=0x3e3a010, buf=0x0, len=0, data=9007955168985100, dest_addr=11,
    tag=26388279066636) at /opt/cray/libfabric/1.22.0/include/rdma/fi_tagged.h:149
#6  0x00001533bf2b5177 in MPIDI_OFI_send_lightweight (buf=0x0, data_sz=0, cq_data=9007955168985100, dst_rank=10,
    tag=12, comm=0x1533c491b670 <MPIR_Comm_builtin>, match_bits=26388279066636, addr=0x3df0138, vci_local=0,
    vci_remote=0, sender_nic=0, receiver_nic=0) at ./src/mpid/ch4/netmod/include/../ofi/ofi_send.h:108
#7  0x00001533bf2b471c in MPIDI_OFI_send (buf=0xff000000016006e0, count=262166, datatype=1283655691, dst_rank=10,
    tag=12, comm=0x1533c491b670 <MPIR_Comm_builtin>, context_offset=0, addr=0x3df0138, vci_src=0, vci_dst=0,
    request=0x7fff210e9860, is_am=false, syncflag=false, is_init=false)
    at ./src/mpid/ch4/netmod/include/../ofi/ofi_send.h:444
#8  0x00001533bf2a0e94 in MPIDI_NM_mpi_isend (buf=0xff000000016006e0, count=262166, datatype=1283655691, rank=10,
    tag=12, comm=0x1533c491b670 <MPIR_Comm_builtin>, attr=0, addr=0x3df0138, request=0x7fff210e9860)
    at ./src/mpid/ch4/netmod/include/../ofi/ofi_send.h:557
#9  0x00001533bf2a036e in MPIDI_isend (buf=0xff000000016006e0, count=262166, datatype=1283655691, rank=10, tag=12,
    comm=0x1533c491b670 <MPIR_Comm_builtin>, attr=0, av=0x3df0138, req=0x7fff210e9860)
    at ./src/mpid/ch4/src/ch4_send.h:30
#10 0x00001533bf2a0015 in MPID_Isend (buf=0xff000000016006e0, count=262166, datatype=1283655691, rank=10, tag=12,
    comm=0x1533c491b670 <MPIR_Comm_builtin>, attr=0, request=0x7fff210e9860) at ./src/mpid/ch4/src/ch4_send.h:58
#11 0x00001533bf29fe61 in MPID_Send (buf=0xff000000016006e0, count=262166, datatype=1283655691, rank=10, tag=12,
    comm=0x1533c491b670 <MPIR_Comm_builtin>, attr=0, request=0x7fff210e9860) at ./src/mpid/ch4/src/ch4_send.h:80
#12 0x00001533bf29ee28 in internal_Send (buf=0xff000000016006e0, count=262166, datatype=1283655691, dest=10, tag=12,
    comm=1140850688) at src/binding/c/pt2pt/send.c:103
#13 0x00001533bf29e277 in PMPI_Send (buf=0xff000000016006e0, count=262166, datatype=1275070475, dest=10, tag=12,
    comm=1140850688) at src/binding/c/pt2pt/send.c:161
#14 0x0000000000402e03 in send_recv_all_to_all (sendbuf_ptr=sendbuf_ptr@entry=0xff00000000200000, sendcnt=...,
    sdispls=..., recvbuf_ptr=recvbuf_ptr@entry=0xff00000003400000, recvcnt=..., rdispls=..., my_rank=12) at c.cpp:37
#15 0x0000000000403829 in main (argc=1, argv=0x7fff210e9c08) at c.cpp:92

In tracing the issue, I found the enqueued message was being dequeued twice, the 2nd time corrupts the dlist. I instrumented the code to dump the entries of head->prev, head, head->next, and we can see the entry 0x4363730 was dequeued twice, the 2nd time the dlist was empty.

Further investigation shows this only happen for small messages (e.g. ~700 bytes) and when we use flag FI_COMPLETION | FI_MATCH_COMPLETE. My guess is, the small message is sent eagerly so the match completion ack arrives later than the local completion, so the match ack triggers a 2nd dequeue.

To Reproduce
pmodels/mpich#7713 (comment)