CUDA context crash in libfabric backend progress thread when using GPU Direct RDMA

[dmvevents]

Description

When using the libfabric backend with GPU Direct RDMA (FI_EFA_USE_DEVICE_RDMA=1), the progress thread crashes with a CUDA context error because pthrCudaCtx_ is NULL when the thread starts.

Environment

• NIXL 0.8.0
• AWS P5.48xlarge (H100 GPUs + EFA)
• TRT-LLM disaggregated inference
• FI_EFA_USE_DEVICE_RDMA=1

Symptoms

CUDA error: invalid device context
Segmentation fault during fi_read with GPU memory

Root Cause

In src/plugins/libfabric/libfabric_backend.cpp, the progress thread starts in the constructor BEFORE registerMem() is called. When the thread tries to access GPU memory via fi_read(), the CUDA context (pthrCudaCtx_) is NULL.

The UCX backend handles this correctly by restarting the thread when the context changes, but the libfabric backend does not.

Proposed Fix

Apply CUDA context INSIDE the progress loop on every iteration, not just at thread start:

void LibfabricBackend::progressThread() {
    while (!progress_thread_stop_.load()) {
#ifdef HAVE_CUDA
        if (cuda_addr_wa_) {
            vramApplyCtx();  // Apply context on EVERY iteration
        }
#endif
        // ... rest of progress loop
    }
}

Workaround

None - requires source patch.

Impact

This blocks GPU Direct RDMA for all libfabric users with CUDA memory.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[amitrad-aws]

I don't undertsand the issue you are describing.
fi_read is called from the postXfer API, it is not related to the progress thread.
Please share a failing scenario if there is one

[dmvevents]

Summary

@amitrad-aws - Responding to your question about fi_read vs progress thread.

The Issue

progressThread() starts at constructor (line 354) but pthrCudaCtx_ is only set later in registerMem() (line 109). If completions arrive in between, the progress thread accesses GPU memory without a valid CUDA context.

Code Evidence

// Constructor starts thread immediately (line 354)
progress_thread_ = std::thread(&nixlLibfabricEngine::progressThread, this);

// pthrCudaCtx_ only set later in registerMem() path (line 109)
pthrCudaCtx_ = ctx;

// progressThread() never calls vramApplyCtx() inside the loop
while (!progress_thread_stop_.load()) {
    // NO vramApplyCtx() call here
    nixl_status_t status = rail_manager.progressActiveDataRails();
}

Stack Trace

nixlLibfabricRail::progressCompletionQueue(bool) const
nixlLibfabricRailManager::progressActiveDataRails()
nixlLibfabricEngine::progressThread()

Reproduction (P5.48xlarge required)

# Key environment variables
env:
  FI_EFA_USE_DEVICE_RDMA: "1"    # Enable GPU Direct RDMA
  NIXL_BACKEND: "LIBFABRIC"
  NIXL_LIBFABRIC_MAX_RAILS: "32"
  FI_EFA_ENABLE_SHM_TRANSFER: "0"  # Force RDMA path

resources:
  limits:
    nvidia.com/gpu: "8"
    vpc.amazonaws.com/efa: "32"

Steps:

1. Deploy disaggregated TRT-LLM (prefill + decode workers)
2. Send inference request to trigger KV cache transfer
3. Segfault occurs 60-120s after startup during first transfer

Error:

!!!!!!! Segfault encountered !!!!!!!

The Fix (PR #1216)

Add vramApplyCtx() inside the progress loop:

while (!progress_thread_stop_.load()) {
#ifdef HAVE_CUDA
    if (cuda_addr_wa_) {
        vramApplyCtx();  // Safe - checks pthrCudaCtx_ internally
    }
#endif
    nixl_status_t status = rail_manager.progressActiveDataRails();
}

Is there anything else needed to evaluate PR #1216?

[amitrad-aws]

The progress thread only processes completions, it never touches the CUDA context.