[None][perf] Gate NCCL NVLS on NVML fabric state, not IMEX availability

[Wanli-Jiang]

The NCCL NVLS gate required getMemHandleType() to resolve to FABRIC, i.e. a fully provisioned IMEX plane. That is stricter than what an NVLS multicast bind actually needs: on single-node systems the bind works over POSIX-FD handles without IMEX, so nodes with a healthy switch fabric but no IMEX (e.g. HGX B200 containers without nvidia-imex) lost NVLS- accelerated NCCL for no reason.

Split the NVML fabric-state check out of getMemHandleType() and gate NCCL NVLS on it directly (ipcNvlsUsable, formerly ipcNvlsFabricUsable): the static multicast capability plus a COMPLETED NVLink fabric registration -- Fabric Manager has configured the switch -- is the actual precondition for a successful single-node bind. Multi-node sessions (detected by comparing the session size with the per-host local session size, both materialized at MPI init) keep the full IMEX requirement, because cross-node multicast memory can only be shared via FABRIC handles. The unprovisioned-fabric nodes that PR #15302 protects against (bind failure poisons the CUDA context and hangs startup) do not reach the COMPLETED state and stay disabled, with the warning split into a fabric-state variant and a multi-node IMEX variant. An explicit NCCL_NVLS_ENABLE setting now skips the check and the warning in getComm() as well, matching NcclCommunicator::createComm.

ipcNvlsSupported() and the ipcNvlsAllocate() path are untouched, so the single-node POSIX-FD allocator restored by PR #15882 is unaffected, and the ipcNvlsMemoryTest regression guards it added still pass.

Verified on 4xB200 (fabric state COMPLETED, no IMEX): a direct cuMulticastCreate/AddDevice/BindMem probe over POSIX-FD succeeds, and with NVLS enabled NCCL reports "NVLS multicast support is available" with 24 NVLS channels while the BART tp2 end-to-end test passes. On 4xH200 NVL (no NVSwitch, MULTICAST_SUPPORTED=0): NVLS stays disabled with no warning, BART tp2 passes, and ipcNvlsMemoryTest passes 2/2. Explicitly setting NCCL_NVLS_ENABLE=0 or =1 skips the probe and warning entirely.

Summary by CodeRabbit

• Bug Fixes
  • Improved NVLS capability detection across single-node and multi-node environments.
  • Automatically disables NCCL NVLS when required fabric or multicast capabilities are unavailable, preventing unreliable communicator initialization.
  • Selects compatible GPU memory sharing mechanisms based on the detected NVLink fabric configuration.
• Tests
  • Updated regression coverage for NVLS usability and static capability checks.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[Wanli-Jiang]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #60296 [ run ] triggered by Bot. Commit: 490c563 Link to invocation

[coderabbitai]

📝 Walkthrough

Walkthrough

NVLS runtime checks now distinguish static multicast support from fabric registration and session topology. Fabric-aware memory handle selection is cached, while NCCL communicator setup centrally disables NVLS when the session cannot use it. Tests now validate the updated usability contract.

Changes

NVLS usability and NCCL gating

Layer / File(s)	Summary
Fabric probe and handle selection cpp/tensorrt_llm/runtime/ipcNvlsMemory.cu	A cached NVML fabric-registration probe selects FABRIC handles only when configured, otherwise falling back to POSIX file descriptors.
NVLS usability contract and state evaluation cpp/include/tensorrt_llm/runtime/ipcNvlsMemory.h, cpp/tensorrt_llm/runtime/ipcNvlsMemory.cu	ipcNvlsUsable() replaces the fabric-only API and evaluates static capability, fabric state, and single-node or multi-node IMEX requirements.
NCCL gating and validation cpp/include/tensorrt_llm/runtime/ipcNvlsMemory.h, cpp/tensorrt_llm/common/opUtils.cpp, cpp/tensorrt_llm/runtime/ncclCommunicator.cpp, cpp/tests/unit_tests/runtime/ipcNvlsMemoryTest.cpp	maybeDisableNcclNvls() centralizes environment handling and warnings before communicator initialization; tests assert that usability implies static support.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant getComm
  participant NcclCommunicator
  participant maybeDisableNcclNvls
  participant nvlsUsability
  participant NCCL

  getComm->>maybeDisableNcclNvls: prepare NCCL NVLS
  NcclCommunicator->>maybeDisableNcclNvls: prepare NCCL NVLS
  maybeDisableNcclNvls->>nvlsUsability: query cached session usability
  nvlsUsability-->>maybeDisableNcclNvls: usable or unusable state
  maybeDisableNcclNvls->>NCCL: set NCCL_NVLS_ENABLE=0 when unusable
  getComm->>NCCL: initialize communicator
  NcclCommunicator->>NCCL: initialize communicator

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise, specific, and accurately summarizes the main NVLS gating change.
Description check	✅ Passed	The description explains the problem, solution, tests, and checklist items, though it doesn't use the template headings verbatim.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #60296 [ run ] completed with state FAILURE. Commit: 490c563
/LLM/main/L0_MergeRequest_PR pipeline #48648 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation