[Bug]: Potential QP leak on transfer failure for PD disaggregation scenario

[thincal]

Environment

Components: Mooncake v0.3.7-post2, SGLang v0.5.7
Use Case: SGLang Prefill-Decode (PD) disaggregation for LLM inference

Problem Description

We observed that after running for some time in a PD disaggregation scenario, the Prefill side encounters the error: Failed to create QP: Cannot allocate memory

Analysis

1. When Prefill transfers KV cache to Decode, if the transfer engine returns an error:

• SGLang only marks this session as failed in Python layer
• The transfer engine should have already allocated underlying resources (endpoints, QPs) and established connection with Decode side

2. When transfer fails, we want to confirm whether Mooncake:

• Automatically cleans up QP resources
• Automatically cleans up endpoint resources
• Or leaves these resources allocated (potential leak)

Current Understanding

From code analysis of mooncake-integration/transfer_engine/transfer_engine_py.cpp:

int TransferEnginePy::batchTransferSync(...) {
    // Get or create segment handle
    Transport::SegmentHandle handle;
    {
        std::lock_guard<std::mutex> guard(mutex_);
        if (handle_map_.count(target_hostname)) {
            handle = handle_map_[target_hostname];
        } else {
            handle = engine_->openSegment(target_hostname);
            if (handle == (Transport::SegmentHandle)-1) return -1;
            handle_map_[target_hostname] = handle;  // ← Cached permanently
        }
    }

    // ... submit transfer ...

    // On transfer failure
    else if (status.s == TransferStatusEnum::FAILED) {
        engine_->freeBatchID(batch_id);  // Only frees batch ID
        already_freed = true;
        completed = true;
        // Question: Are QP/endpoint resources cleaned up here?
    }

    return -1;
}

We notice that:

• freeBatchID(batch_id) is called on failure
• handle_map_[target_hostname] is never removed
• closeSegment() appears to be a no-op (return 0)

[alogfans]

If there is no other failures reported, it's probably you don't have enough user prvillege.

[staryxchen]

Hi @thincal, could you try setting the environment variable MC_ENABLE_DEST_DEVICE_AFFINITY=1 and verify if the issue still reproduces?

Thanks!

[thincal]

Hi @thincal, could you try setting the environment variable MC_ENABLE_DEST_DEVICE_AFFINITY=1 and verify if the issue still reproduces?

Thanks!

We have tried this option, but the problem was still happened.

[thincal]

If there is no other failures reported, it's probably you don't have enough user prvillege.

We have already configured the container as privileged: true.

[staryxchen]

Hi @thincal, thanks for the feedback. Could you help us gather more diagnostic info to narrow down the root cause?

1. Check system limits and memory:

   ulimit -a
   free -g

2. Track RDMA resource usage during reproduction:

   # Run this periodically (e.g., every few seconds) while reproducing the issue
   rdma resource

   This will help us see if QP or other RDMA resources are being exhausted.

Please share the output so we can investigate further. Thanks!

[caozhanhao]

Could this be related to #1643? Upgrading to v0.3.10.post1 might resolve the issue.

[thincal]

For quick response, I pick one container with highest QP number currently (but not report failure yet) FYI, and I will provide the necessary info when reproduced again.

• ulimit -a

  • container

    core file size              (blocks, -c) unlimited
    data seg size               (kbytes, -d) unlimited
    scheduling priority                 (-e) 0
    file size                   (blocks, -f) unlimited
    pending signals                     (-i) 8254557
    max locked memory           (kbytes, -l) 64
    max memory size             (kbytes, -m) unlimited
    open files                          (-n) 1048576
    pipe size                (512 bytes, -p) 8
    POSIX message queues         (bytes, -q) 819200
    real-time priority                  (-r) 0
    stack size                  (kbytes, -s) 8192
    cpu time                   (seconds, -t) unlimited
    max user processes                  (-u) unlimited
    virtual memory              (kbytes, -v) unlimited
    file locks                          (-x) unlimited
    

  • host of this container

    core file size              (blocks, -c) 0
    data seg size               (kbytes, -d) unlimited
    scheduling priority                 (-e) 0
    file size                   (blocks, -f) unlimited
    pending signals                     (-i) 8254557
    max locked memory           (kbytes, -l) 524288
    max memory size             (kbytes, -m) unlimited
    open files                          (-n) 524288
    pipe size                (512 bytes, -p) 8
    POSIX message queues         (bytes, -q) 819200
    real-time priority                  (-r) 0
    stack size                  (kbytes, -s) 8192
    cpu time                   (seconds, -t) unlimited
    max user processes                  (-u) 8254557
    virtual memory              (kbytes, -v) unlimited
    file locks                          (-x) unlimited
    

• rdma resource

  0: mlx5_06: pd 28 cq 29 qp 5942 cm_id 0 mr 1407 ctx 32 srq 5 
  3: mlx5_07: pd 28 cq 29 qp 5412 cm_id 0 mr 1407 ctx 32 srq 5 
  4: mlx5_05: pd 28 cq 29 qp 5756 cm_id 0 mr 1407 ctx 32 srq 5 
  5: mlx5_04: pd 28 cq 29 qp 6026 cm_id 0 mr 1407 ctx 32 srq 5 
  6: mlx5_00: pd 28 cq 23 qp 22110 cm_id 0 mr 1375 ctx 32 srq 5 
  7: mlx5_08: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 0 srq 2 
  8: mlx5_09: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 0 srq 2 
  9: mlx5_01: pd 28 cq 23 qp 21852 cm_id 0 mr 1375 ctx 32 srq 5 
  10: mlx5_02: pd 28 cq 23 qp 22040 cm_id 0 mr 1375 ctx 32 srq 5 
  11: mlx5_03: pd 28 cq 23 qp 21976 cm_id 0 mr 1375 ctx 32 srq 5 
  12: mlx5_10: pd 19 cq 4 qp 3 cm_id 0 mr 0 ctx 16 srq 2
  

[thincal]

max locked memory (kbytes, -l) 64

FYI: also have tried set the ulimit -l unlimited , the problem could also be reproduced.

[staryxchen]

Hi @thincal, thanks for the detailed diagnostic info!

From the rdma resource output, the QP counts on mlx5_00 ~ mlx5_03 are all above 20,000, which is very likely hitting the NIC's QP allocation limit. That aligns with the error you're seeing.

The key question now is why so many QPs are being created. To help us investigate further, could you provide:

1. Cluster topology: How many machines (P nodes and D nodes) are involved in your deployment?
2. NIC configuration per machine: How many RDMA NICs per node, and what are their device names (e.g., mlx5_0, mlx5_1, ...)? Specifically, are the NIC names consistent across P nodes and D nodes?

Regarding MC_ENABLE_DEST_DEVICE_AFFINITY — this feature performs device affinity based on NIC device names. If the NIC names differ between P nodes and D nodes (e.g., mlx5_0 on P maps to a different physical port than mlx5_0 on D), the affinity won't take effect as expected, which could explain why it didn't help in your case.

Thanks!

[thincal]

Could this be related to #1643? Upgrading to v0.3.10.post1 might resolve the issue.

I go through the fix it seems most likely be related with my case, also before the problem occur there are many endpoint evicted events happened (1118 times in 9minutes).

[thincal]

How many RDMA NICs per node, and what are their device names (e.g., mlx5_0, mlx5_1, ...)? Specifically, are the NIC names consistent across P nodes and D nodes?

8 compute NIC (mlx5_00 ~ mlx5_07) assigned to mooncake transfer, and P/D nodes' NIC device name are all with same name which is double confirmed.

[rongjunsong]

i have the same problem，how can solve ,thanks @thincal

[ccs1112]

Here's my best guess of the root cause from a quick glance:

1. In rdma_endpoint.cpp a worker posts a slice on QP with submitPostSend incrementing wr_depth_list_[i]++
2. Peer stops responding; NIC retries up to RETRY_CNT in rdma_endpoint.cpp:30
3. Retries exhaust; QP transitions to ERROR state per IB semantics
4. CQ stops producing normal completions for the pending WRs on that QP so wr_depth_list_[i] is never decremented (not sure if this is due to poller thinking this endpoint is inactive or if FLUSH is arriving but decrement is broken)
5. Traffic shifts to other peers, endpoint_map_ fills, triggering SIEVEEndpointStore::evictEndpoint() in endpoint_store.cpp:191
6. Eviction calls set_active(false) and moves the shared_ptr<RdmaEndPoint> into waiting_list_ crucially with no destroyQP() call
7. On the next insertEndpoint, RdmaContext::endpointGetOrCreate invokes reclaimEndpoint() at rdma_context.cpp:356
8. reclaimEndpoint() checks hasOutstandingSlice() at rdma_endpoint.cpp:448, so the stuck wr_depth_list_[i] != 0 makes it return true
9. Endpoint is skipped and shared_ptr stays pinned in waiting_list_. ~RdmaEndPoint never runs so ibv_destroy_qp never called
10. Steps 1–9 repeat; each failure-path eviction leaks num_qp_per_ep QPs
11. After ~1118 such evictions (per @thincal log), NIC QP table is exhausted
12. Next ibv_create_qp() returns ENOMEM: Failed to create QP: Cannot allocate memory

If it's alright with everyone I'd like to claim this. If my hypothesis is correct I should be able to post repro here in ~2-3 days. And if I can repro I'll submit the patch as soon as I can.