prov/tcp: Memory leaks detected by LSAN in fabtests with TCP provider

[sydidelot]

fabtest_tcp.log

Description

Memory leaks were detected by LeakSanitizer (LSAN) when running fabtests with the TCP provider. A total of 15 memory leak errors were detected across multiple test programs.

Environment

• Provider: TCP (xnet)
• Compilation: Libfabric and Fabtests compiled with --enable-asan
• Test Command: ./runfabtests.sh tcp 127.0.0.1 -vvv -p /nfs_home/sdidelot/usr/libfabric/bin/ -T 20
• Commit: af8ddbe

Summary of Leaks

1. fi_multi_ep Test Leaks

Fabtests memory allocation issues:

• 768 bytes (3 objects) leaked in alloc_multi_ep_res at functional/multi_ep.c:166
• 768 bytes (3 objects) leaked in alloc_multi_ep_res at functional/multi_ep.c:161
• Total: 38 leak instances in fi_multi_ep tests

TCP/xnet provider leaks:

• 240 bytes (2 objects) leaked via fi_dupinfo called from xnet_handle_conn
• 120 bytes (1 object) leaked via fi_dupinfo called from xnet_handle_conn
• 120 bytes (1 object) leaked in xnet_domain_open
• 88 bytes (1 object) leaked in xnet_mplex_mr_regattr

2. fi_rdm_multi_client Test Leaks

• 144 bytes (9 objects) leaked in getaddr at common/shared.c:2095
• Total: 2 leak instances

Detailed Stack Traces

Example 1: fi_multi_ep buffer allocation leak

Direct leak of 768 byte(s) in 3 object(s) allocated from:
    #0 0xf763e5b576d0 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0xbedb39d3ebe4 in ft_host_alloc include/hmem.h:155
    #2 0xbedb39d20c58 in alloc_multi_ep_res functional/multi_ep.c:166
    #3 0xbedb39d20c58 in run_test functional/multi_ep.c:545
    #4 0xbedb39d20c58 in main functional/multi_ep.c:678

Direct leak of 768 byte(s) in 3 object(s) allocated from:
    #0 0xf763e5b576d0 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0xbedb39d3ebe4 in ft_host_alloc include/hmem.h:155
    #2 0xbedb39d20cb0 in alloc_multi_ep_res functional/multi_ep.c:161
    #3 0xbedb39d20cb0 in run_test functional/multi_ep.c:545
    #4 0xbedb39d20cb0 in main functional/multi_ep.c:678

Example 2: xnet provider fi_dupinfo leak

Direct leak of 240 byte(s) in 2 object(s) allocated from:
    #0 0xf9c7a2ad76d0 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0xf9c7a23825fc in fi_dupinfo@@FABRIC_1.10 (/nfs_home/sdidelot/usr/libfabric/lib/libfabric.so.1+0xa25fc)
    #2 0xf9c7a26199c0 in xnet_handle_conn (/nfs_home/sdidelot/usr/libfabric/lib/libfabric.so.1+0x3399c0)
    #3 0xf9c7a2646910 in xnet_run_progress (/nfs_home/sdidelot/usr/libfabric/lib/libfabric.so.1+0x366910)
    #4 0xf9c7a2647660 in xnet_progress_all (/nfs_home/sdidelot/usr/libfabric/lib/libfabric.so.1+0x367660)
    #5 0xf9c7a263c4b8 in xnet_eq_read (/nfs_home/sdidelot/usr/libfabric/lib/libfabric.so.1+0x35c4b8)
    #6 0xf9c7a23f2d20 in ofi_eq_sread (/nfs_home/sdidelot/usr/libfabric/lib/libfabric.so.1+0x112d20)
    #7 0xb6ad493e45a4 in fi_eq_sread /nfs_home/sdidelot/usr/libfabric//include/rdma/fi_eq.h:393
    #8 0xb6ad493e45a4 in ft_retrieve_conn_req common/shared.c:1237
    #9 0xb6ad493d14b8 in setup_server_ep functional/multi_ep.c:437

Indirect leak of 464 byte(s) in 2 object(s) allocated from:
    #0 0xf9c7a2ad76d0 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0xf9c7a2382a30 in fi_dupinfo@@FABRIC_1.10 (/nfs_home/sdidelot/usr/libfabric/lib/libfabric.so.1+0xa2a30)
    #2 0xf9c7a26199c0 in xnet_handle_conn (/nfs_home/sdidelot/usr/libfabric/lib/libfabric.so.1+0x3399c0)
    #3 0xf9c7a2646910 in xnet_run_progress (/nfs_home/sdidelot/usr/libfabric/lib/libfabric.so.1+0x366910)
    #4 0xf9c7a2647660 in xnet_progress_all (/nfs_home/sdidelot/usr/libfabric/lib/libfabric.so.1+0x367660)
    #5 0xf9c7a263c4b8 in xnet_eq_read (/nfs_home/sdidelot/usr/libfabric/lib/libfabric.so.1+0x35c4b8)

Example 3: fi_rdm_multi_client getaddr leak

Direct leak of 144 byte(s) in 9 object(s) allocated from:
    #0 0xd60a126476d0 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0xbf0a7f1e5f28 in getaddr common/shared.c:2095
    #2 0xbf0a7f1e8d70 in ft_getsrcaddr common/shared.c:2133
    #3 0xbf0a7f1e8d70 in ft_init_fabric common/shared.c:1058
    #4 0xbf0a7f1cebe0 in run functional/rdm_multi_client.c:168
    #5 0xbf0a7f1cebe0 in main functional/rdm_multi_client.c:287

Affected Files

Fabtests:

• fabtests/functional/multi_ep.c (lines 161, 166)
• fabtests/functional/rdm_multi_client.c
• fabtests/common/shared.c (line 2095 - dupaddr/getaddr function)

libfabric (TCP/xnet provider):

• xnet_handle_conn function
• xnet_domain_open function
• xnet_mplex_mr_regattr function
• fi_dupinfo function

Analysis

The leaks appear to be in two categories:

1. Fabtests cleanup issues:

   • Allocated buffers in alloc_multi_ep_res() are not being freed properly before test exit
   • The getaddr/dupaddr function in common/shared.c appears to allocate memory that is not properly freed

2. Provider-side leaks:

   • The xnet provider appears to leak fi_info structures duplicated via fi_dupinfo() during connection handling
   • Additional leaks in xnet_domain_open and xnet_mplex_mr_regattr

Reproduction

Full log available in attached file: fabtest_tcp.log

The issue is reproducible by running:

./runfabtests.sh tcp 127.0.0.1 -vvv -p <libfabric_bin_path> -T 20

Additional Context

• All tests were run with ASAN/LSAN enabled
• Tests were executed on localhost (127.0.0.1) using the TCP provider
• Both successful and failed tests exhibited memory leaks

[sydidelot]

@j-xiong Should we add a test with ASAN-instrumented binaries to the CICD pipeline?

[j-xiong]

AWS CI already has ASAN-instrumented runs with efa provider (also covers core, util and shm that are used by efa). We can consider adding similar test to Intel CI, which will cover more providers. However, this will happen in steps -- starting with internal testing, fixing all issues, adding to the official CI -- to prevent blocking every PR.