NIXL_LIBFABRIC_NUM_RAILS environment variable not enforced during discovery

[dmvevents]

Description

The NIXL_LIBFABRIC_NUM_RAILS environment variable is not enforced during EFA device discovery, causing all available devices to be used regardless of the setting.

Environment

• NIXL 0.8.0
• AWS P5.48xlarge (32 EFA devices)
• libfabric backend

File

src/utils/libfabric/libfabric_rail_manager.cpp

Symptoms

export NIXL_LIBFABRIC_NUM_RAILS=8
# Still initializes all 32 rails

Proposed Fix

Enforce rail limit during discovery loop:

const char* num_rails_env = std::getenv("NIXL_LIBFABRIC_NUM_RAILS");
size_t max_rails = SIZE_MAX;
if (num_rails_env) {
    max_rails = std::stoul(num_rails_env);
}

// In discovery loop:
if (rail_count >= max_rails) break;

Also support NIXL_LIBFABRIC_MAX_RAILS as alternative name for consistency.

Impact

Cannot limit rails for testing or resource management without modifying code.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[fengjica]

Hi! I don't see there is a 'NIXL_LIBFABRIC_NUM_RAILS' env variable in the code today.

[amitrad-aws]

The number of rails being used for VRAM memory depends on the PCI topology and the NICs that should be used with that specific GPU.
For DRAM we currently use all rails but plan to change this to be NUMA aware and limit the number of rails as well.
Please let us know if you need more information or we can close this issue.

[amitrad-aws]

@dmvevents This envirnoment variable does not exists in the code no do we see a reason to have it.
Why do you think it is necessary?

[amitrad-aws]

Why do you need this feature?
The rails used for each GPU are determined according to the PCI topology where each rail is eoungh to maintain the bandwidth the GPU can provide

[amitrad-aws]

The deserialization issue you are referring was resolved with #1142

[dmvevents]

Clarification & Updated Analysis

@amitrad-aws @fengjica - After further investigation, we need to correct our original description.

What We Originally Claimed

We described this as "heterogeneous EFA allocations" where different pods have different numbers of EFA devices.

What Actually Happened

Both workers were on identical P5.48xlarge hardware with access to all 32 EFA devices (uverbs0-uverbs31). The rail count mismatch was caused by something else:

# From our Dec 16 test results:
"Both workers have access to 32 EFA devices (uverbs0-uverbs31)"
"Issue is NOT with EFA device visibility but with LIBFABRIC rail initialization"

Actual Root Cause

HMEM (Heterogeneous Memory) initialization failure during GPU memory registration caused some rails to fail creation:

1. LIBFABRIC rail initialization requires HMEM for GPU Direct RDMA
2. If HMEM init fails for some devices, fewer rails are created
3. Worker A serializes N rails, Worker B only created M rails (M < N)
4. Deserialization fails: dest_data_ep_5 not found

Error We Observed

serdes.cpp:89 - Deserialization of tag dest_data_ep_5 failed
libfabric_rail_manager.cpp:855 - Key dest_data_ep_5 not found or has invalid length: -1

Questions

1. What causes HMEM/rail initialization to fail intermittently?
2. Should NIXL handle partial rail initialization gracefully?
3. Is there a way to diagnose which rails failed to initialize?

The NIXL_LIBFABRIC_NUM_RAILS env var (PR #1226) would work as a workaround to force consistent rail counts, but the real fix may be in HMEM initialization.

If this is expected behavior or there's an existing diagnostic, happy to close this issue.

[amitrad-aws]

1. We'll need to see logs from such a failure to diagnose what can cause this
2. Nixl will handle partial initialization gracefully
3. Yes, with the logs printed when it fails
   The workaround suggested of forcing number of rails with an environment variable is not a viable solution. The GPUs should use the devices according to the PCI topology to allow GPU Direct RDMA and performance. If there is a failure in initialization, that should be addressed. Sharing other device with another GPU is not a valid solution