[Question] How does nixl use MNNVL communication in the GB200 series environment?

[kebe7jun]

When I use NIXL as the connector for Prefill/Decode (P/D) separation in vLLM, and enable UCX_PROTO_INFO=y, I observe that the traffic never negotiates the MNNVL path.

Details

[1769083945.891234] [local-cluster-10:1768445:0]   +--------------------------------+---------------------+-----------------------------------------------------+
[1769083945.891317] [local-cluster-10:1768445:0]   +--------------------------------+--------------------------------------------------------------------+
[1769083945.891317] [local-cluster-10:1768445:0]   | ucp_context_0 intra-node cfg#1 | remote memory write by ucp_put* from host memory to cuda/dev[1]    |
[1769083945.891317] [local-cluster-10:1768445:0]   +--------------------------------+--------------+-----------------------------------------------------+
[1769083945.891318] [local-cluster-10:1768445:0]   |                         0..220 | short        | rc_mlx5/mlx5_2:1                                    |
[1769083945.891318] [local-cluster-10:1768445:0]   |                221..3371787874 | copy-in      | rc_mlx5/mlx5_2:1                                    |
[1769083945.891318] [local-cluster-10:1768445:0]   |                3371787875..inf | zero-copy    | 72% on rc_mlx5/mlx5_2:1 and 28% on dc_mlx5/mlx5_3:1 |
[1769083945.891318] [local-cluster-10:1768445:0]   +--------------------------------+--------------+-----------------------------------------------------+
[1769083945.891379] [local-cluster-10:1768445:0]   +--------------------------------+-------------------------------------------------------------------------------------+
[1769083945.891379] [local-cluster-10:1768445:0]   | ucp_context_0 intra-node cfg#1 | remote memory write by ucp_put*(fast-completion) from host memory to cuda/dev[1]    |
[1769083945.891379] [local-cluster-10:1768445:0]   +--------------------------------+-------------------------------+-----------------------------------------------------+
[1769083945.891380] [local-cluster-10:1768445:0]   |                         0..220 | short                         | rc_mlx5/mlx5_2:1                                    |
[1769083945.891380] [local-cluster-10:1768445:0]   |                221..3371891700 | copy-in                       | rc_mlx5/mlx5_2:1                                    |
[1769083945.891380] [local-cluster-10:1768445:0]   |                3371891701..inf | zero-copy                     | 72% on rc_mlx5/mlx5_2:1 and 28% on dc_mlx5/mlx5_3:1 |
[1769083945.891380] [local-cluster-10:1768445:0]   +--------------------------------+-------------------------------+-----------------------------------------------------+
[1769083945.891443] [local-cluster-10:1768445:0]   +--------------------------------+---------------------------------------------------------------------------+
[1769083945.891443] [local-cluster-10:1768445:0]   | ucp_context_0 intra-node cfg#1 | remote memory write by ucp_put*(multi) from host memory to cuda/dev[1]    |
[1769083945.891443] [local-cluster-10:1768445:0]   +--------------------------------+---------------------+-----------------------------------------------------+
[1769083945.891443] [local-cluster-10:1768445:0]   |                         0..220 | short               | rc_mlx5/mlx5_2:1                                    |
[1769083945.891443] [local-cluster-10:1768445:0]   |                   221..1157826 | copy-in             | rc_mlx5/mlx5_2:1                                    |
[1769083945.891444] [local-cluster-10:1768445:0]   |                   1157827..inf | zero-copy           | 72% on rc_mlx5/mlx5_2:1 and 28% on dc_mlx5/mlx5_3:1 |
[1769083945.891444] [local-cluster-10:1768445:0]   +--------------------------------+---------------------+-----------------------------------------------------+
[1769084049.338751] [local-cluster-10:1768445:a]   +--------------------------------+-------------------------------------------------------------+
[1769084049.338755] [local-cluster-10:1768445:a]   | ucp_context_0 inter-node cfg#2 | remote memory write by ucp_put* from host memory to host    |
[1769084049.338756] [local-cluster-10:1768445:a]   +--------------------------------+------------------------------------------+------------------+
[1769084049.338756] [local-cluster-10:1768445:a]   |                         0..220 | short                                    | rc_mlx5/mlx5_3:1 |
[1769084049.338757] [local-cluster-10:1768445:a]   |                       221..inf | copy-in                                  | rc_mlx5/mlx5_3:1 |
[1769084049.338757] [local-cluster-10:1768445:a]   +------------------------------------------------------------+
[1769083942.580488] [local-cluster-10:1768444:0]   |                         0..220 | short               | rc_mlx5/mlx5_2:1                                    |
[1769083942.580488] [local-cluster-10:1768444:0]   |                   221..1157826 | copy-in             | rc_mlx5/mlx5_2:1                                    |
[1769083942.580489] [local-cluster-10:1768444:0]   |                   1157827..inf | zero-copy           | 72% on rc_mlx5/mlx5_2:1 and 28% on dc_mlx5/mlx5_3:1 |
[1769083942.580489] [local-cluster-10:1768444:0]   +--------------------------------+---------------------+-----------------------------------------------------+
[1769083942.580573] [local-cluster-10:1768444:0]   +--------------------------------+--------------------------------------------------------------------+
[1769083942.580573] [local-cluster-10:1768444:0]   | ucp_context_0 intra-node cfg#1 | remote memory write by ucp_put* from host memory to cuda/dev[0]    |
[1769083942.580573] [local-cluster-10:1768444:0]   +--------------------------------+--------------+-----------------------------------------------------+
[1769083942.580573] [local-cluster-10:1768444:0]   |                         0..220 | short        | rc_mlx5/mlx5_2:1                                    |
[1769083942.580574] [local-cluster-10:1768444:0]   |                221..3371787874 | copy-in      | rc_mlx5/mlx5_2:1                                    |
[1769083942.580574] [local-cluster-10:1768444:0]   |                3371787875..inf | zero-copy    | 72% on rc_mlx5/mlx5_2:1 and 28% on dc_mlx5/mlx5_3:1 |
[1769083942.580574] [local-cluster-10:1768444:0]   +--------------------------------+--------------+-----------------------------------------------------+
[1769083942.580632] [local-cluster-10:1768444:0]   +--------------------------------+-------------------------------------------------------------------------------------+
[1769083942.580632] [local-cluster-10:1768444:0]   | ucp_context_0 intra-node cfg#1 | remote memory write by ucp_put*(fast-completion) from host memory to cuda/dev[0]    |
[1769083942.580633] [local-cluster-10:1768444:0]   +--------------------------------+-------------------------------+-----------------------------------------------------+
[1769083942.580633] [local-cluster-10:1768444:0]   |                         0..220 | short                         | rc_mlx5/mlx5_2:1                                    |
[1769083942.580633] [local-cluster-10:1768444:0]   |                221..3371891700 | copy-in                       | rc_mlx5/mlx5_2:1                                    |
[1769083942.580633] [local-cluster-10:1768444:0]   |                3371891701..inf | zero-copy                     | 72% on rc_mlx5/mlx5_2:1 and 28% on dc_mlx5/mlx5_3:1 |
[1769083942.580634] [local-cluster-10:1768444:0]   +--------------------------------+-------------------------------+-----------------------------------------------------+

The KV transfer efficiency also seems suboptimal:

KV Transfer metrics: Num successful transfers=2, Avg xfer time (ms)=1.49, P90 xfer time (ms)=1.627, Avg post time (ms)=1.15, P90 post time (ms)=1.28, Avg MB per transfer=4.781, Throughput (MB/s)=3209.478.

This behavior is consistent across both nixl[cu13]==0.9.0 and nixl[cu13]==0.8.0.

To isolate the issue, I tested with ucx_perftest (both my self-built version and the system-provided one; the system binary also does not enable MNNVL by default in my environment). My test setup is as follows:

Server: UCX_PROTO_INFO=y UCX_NET_DEVICES=mlx5_2:1 UCX_TLS=ib,sm,cuda_copy,cuda_ipc,self UCX_CUDA_IPC_ENABLE_MNNVL=y ucx_perftest -t tag_bw -m cuda -n 1000 -s 4194304

Client: UCX_NET_DEVICES=mlx5_2:1 UCX_TLS=ib,sm,cuda_copy,cuda_ipc,self UCX_CUDA_IPC_ENABLE_MNNVL=y ucx_perftest -t tag_bw -m cuda -n 1000 -s 41943040 <serverip>

In this case, MNNVL is successfully used：

[1769084848.088832] [local-cluster-10:1776775:0]   +---------------------------+----------------------------------------------------------+
[1769084848.088841] [local-cluster-10:1776775:0]   | perftest inter-node cfg#1 | rendezvous data fetch(multi) into cuda/GPU0 from cuda    |
[1769084848.088843] [local-cluster-10:1776775:0]   +---------------------------+------------------------------------------+---------------+
[1769084848.088844] [local-cluster-10:1776775:0]   |                         0 | no data fetch                            |               |
[1769084848.088848] [local-cluster-10:1776775:0]   |                    1..inf | zero-copy read from remote               | cuda_ipc/cuda |
[1769084848.088850] [local-cluster-10:1776775:0]   +---------------------------+------------------------------------------+---------------+
[1769084848.091072] [local-cluster-10:1776775:0]   +---------------------------+--------------------------------------------------------------+
[1769084848.091082] [local-cluster-10:1776775:0]   | perftest inter-node cfg#1 | tagged message by ucp_tag_send*(multi) from cuda/GPU0        |
[1769084848.091083] [local-cluster-10:1776775:0]   +---------------------------+-------------------------------------------+------------------+
[1769084848.091085] [local-cluster-10:1776775:0]   |                         0 | eager short                               | rc_mlx5/mlx5_2:1 |
[1769084848.091088] [local-cluster-10:1776775:0]   |                    1..inf | (?) rendezvous zero-copy read from remote | cuda_ipc/cuda    |
[1769084848.091090] [local-cluster-10:1776775:0]   +---------------------------+-------------------------------------------+------------------+

And the performance is excellent：

+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
Final:                  1000     55.456    55.555    55.555   720005.84  720005.84       18000       18000

I have reviewed many documents and references, and most of them indicate that setting UCX_CUDA_IPC_ENABLE_MNNVL=y should be sufficient. However, in my case, that does not appear to enable MNNVL for vLLM/NIXL.

I would like to understand what additional configuration or constraints are required to make vLLM/NIXL use the MNNVL path, and why it works with ucx_perftest but not in the vLLM + NIXL connector workflow.

[linear]

from mikhailb:

In the NIXL logs, I only see host-to-GPU memory transfers, whereas MNNVL can be used only for GPU-to-GPU communication. Also GPU memory must be allocated as VMM with a fabric type to be eligible for MNNVL.

Since NIXL is using the RMA UCX API, it would be better to run ucx_perftest with the ucp_put_bw test to mimic NIXL traffic patterns.

[kebe7jun]

GPU <-> GPU also use rc.

[tvegas1]

Can you please run the ucx_perftest command with ucp_put_bw option, and provide log output with environment UCX_LOG_LEVEL=debug? Also you can try to exclude IB with UCX_TLS=^ib which should fallback on MNNVL.

[kebe7jun]

Thanks for you reply! @tvegas1

ucp_put_bw is normal:

ucx_perftest outputs

# UCX_PROTO_INFO=y UCX_NET_DEVICES=mlx5_2:1 UCX_TLS=ib,sm,cuda_copy,cuda_ipc,self UCX_CUDA_IPC_ENABLE_MNNVL=y ucx_perftest -t ucp_put_bw -m cuda -n 1000 -s 41943040 10.0.8.10
[1769515123.373141] [XN-GB-Tray-15:1813812:0]        perftest.c:828  UCX  WARN  CPU affinity is not set (bound to 144 cpus). Performance may be impacted.
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[1769515141.328901] [XN-GB-Tray-15:1813812:0]   +---------------------------+--------------------------------------------------------------------+
[1769515141.328923] [XN-GB-Tray-15:1813812:0]   | perftest inter-node cfg#1 | remote memory write by ucp_put* from host memory to cuda/dev[0]    |
[1769515141.328927] [XN-GB-Tray-15:1813812:0]   +---------------------------+-------------------------------------------------+------------------+
[1769515141.328929] [XN-GB-Tray-15:1813812:0]   |                     0..2K | short                                           | rc_mlx5/mlx5_2:1 |
[1769515141.328930] [XN-GB-Tray-15:1813812:0]   |                 2049..inf | zero-copy                                       | rc_mlx5/mlx5_2:1 |
[1769515141.328932] [XN-GB-Tray-15:1813812:0]   +---------------------------+-------------------------------------------------+------------------+
[1769515141.328991] [XN-GB-Tray-15:1813812:0]   +---------------------------+-------------------------------------------------------------------------------------+
[1769515141.328995] [XN-GB-Tray-15:1813812:0]   | perftest inter-node cfg#1 | remote memory write by ucp_put*(fast-completion) from host memory to cuda/dev[0]    |
[1769515141.328997] [XN-GB-Tray-15:1813812:0]   +---------------------------+------------------------------------------------------------------+------------------+
[1769515141.328999] [XN-GB-Tray-15:1813812:0]   |                     0..2K | short                                                            | rc_mlx5/mlx5_2:1 |
[1769515141.329001] [XN-GB-Tray-15:1813812:0]   |               2049..24146 | copy-in                                                          | rc_mlx5/mlx5_2:1 |
[1769515141.329003] [XN-GB-Tray-15:1813812:0]   |                24147..inf | zero-copy                                                        | rc_mlx5/mlx5_2:1 |
[1769515141.329006] [XN-GB-Tray-15:1813812:0]   +---------------------------+------------------------------------------------------------------+------------------+
[1769515141.329060] [XN-GB-Tray-15:1813812:0]   +---------------------------+---------------------------------------------------------------------------+
[1769515141.329064] [XN-GB-Tray-15:1813812:0]   | perftest inter-node cfg#1 | remote memory write by ucp_put*(multi) from host memory to cuda/dev[0]    |
[1769515141.329065] [XN-GB-Tray-15:1813812:0]   +---------------------------+--------------------------------------------------------+------------------+
[1769515141.329068] [XN-GB-Tray-15:1813812:0]   |                    0..587 | short                                                  | rc_mlx5/mlx5_2:1 |
[1769515141.329069] [XN-GB-Tray-15:1813812:0]   |                  588..inf | zero-copy                                              | rc_mlx5/mlx5_2:1 |
[1769515141.329070] [XN-GB-Tray-15:1813812:0]   +---------------------------+--------------------------------------------------------+------------------+
[1769515141.436530] [XN-GB-Tray-15:1813812:0]   +---------------------+--------------------------------------------------------------------+
[1769515141.436539] [XN-GB-Tray-15:1813812:0]   | perftest self cfg#2 | remote memory write by ucp_put* from host memory to cuda/dev[1]    |
[1769515141.436542] [XN-GB-Tray-15:1813812:0]   +---------------------+-------------------------------------------------+------------------+
[1769515141.436546] [XN-GB-Tray-15:1813812:0]   |               0..2K | short                                           | rc_mlx5/mlx5_2:1 |
[1769515141.436549] [XN-GB-Tray-15:1813812:0]   |           2049..inf | zero-copy                                       | cuda_copy/cuda   |
[1769515141.436553] [XN-GB-Tray-15:1813812:0]   +---------------------+-------------------------------------------------+------------------+
[1769515141.436591] [XN-GB-Tray-15:1813812:0]   +---------------------+-------------------------------------------------------------------------------------+
[1769515141.436595] [XN-GB-Tray-15:1813812:0]   | perftest self cfg#2 | remote memory write by ucp_put*(fast-completion) from host memory to cuda/dev[1]    |
[1769515141.436597] [XN-GB-Tray-15:1813812:0]   +---------------------+------------------------------------------------------------------+------------------+
[1769515141.436600] [XN-GB-Tray-15:1813812:0]   |               0..2K | short                                                            | rc_mlx5/mlx5_2:1 |
[1769515141.436601] [XN-GB-Tray-15:1813812:0]   |           2049..inf | zero-copy                                                        | cuda_copy/cuda   |
[1769515141.436603] [XN-GB-Tray-15:1813812:0]   +---------------------+------------------------------------------------------------------+------------------+
[1769515141.436646] [XN-GB-Tray-15:1813812:0]   +---------------------+---------------------------------------------------------------------------+
[1769515141.436649] [XN-GB-Tray-15:1813812:0]   | perftest self cfg#2 | remote memory write by ucp_put*(multi) from host memory to cuda/dev[1]    |
[1769515141.436650] [XN-GB-Tray-15:1813812:0]   +---------------------+--------------------------------------------------------+------------------+
[1769515141.436653] [XN-GB-Tray-15:1813812:0]   |               0..2K | short                                                  | rc_mlx5/mlx5_2:1 |
[1769515141.436654] [XN-GB-Tray-15:1813812:0]   |           2049..inf | zero-copy                                              | cuda_copy/cuda   |
[1769515141.436655] [XN-GB-Tray-15:1813812:0]   +---------------------+--------------------------------------------------------+------------------+
[1769515141.438856] [XN-GB-Tray-15:1813812:0]   +---------------------------+-------------------------------------------------------------------------+
[1769515141.438862] [XN-GB-Tray-15:1813812:0]   | perftest inter-node cfg#1 | remote memory write by ucp_put*(multi) from cuda/GPU1 to cuda/dev[0]    |
[1769515141.438864] [XN-GB-Tray-15:1813812:0]   +---------------------------+------------------------------------------------------+------------------+
[1769515141.438866] [XN-GB-Tray-15:1813812:0]   |                         0 | short                                                | rc_mlx5/mlx5_2:1 |
[1769515141.438868] [XN-GB-Tray-15:1813812:0]   |                    1..inf | zero-copy                                            | cuda_ipc/cuda    |
[1769515141.438870] [XN-GB-Tray-15:1813812:0]   +---------------------------+------------------------------------------------------+------------------+
Final:                  1000     61.216    61.312    61.312   652401.26  652401.26       16310       16310

But when I set UCX_TLS=^ib for nixl, all connections fallback to tcp:

UCX Proto info in vllm logs

[1769515316.411565] [XN-GB-Tray-17:1281331:1]   +--------------------------------+-------------------------------------------------------------+
[1769515316.411565] [XN-GB-Tray-17:1281331:1]   | ucp_context_0 inter-node cfg#2 | remote memory write by ucp_put* from host memory to cuda    |
[1769515316.411566] [XN-GB-Tray-17:1281331:1]   +--------------------------------+-----------------------------------------+-------------------+
[1769515316.411566] [XN-GB-Tray-17:1281331:1]   |                         0..inf | software emulation                      | tcp/enP6p3s0f0np0 |
[1769515316.411566] [XN-GB-Tray-17:1281331:1]   +--------------------------------+-----------------------------------------+-------------------+
[1769515316.413445] [XN-GB-Tray-17:1281331:1]   +--------------------------------+-------------------------------------------------------------+
[1769515316.413445] [XN-GB-Tray-17:1281331:1]   | ucp_context_0 inter-node cfg#2 | remote memory write by ucp_put* from host memory to cuda    |
[1769515316.413446] [XN-GB-Tray-17:1281331:1]   +--------------------------------+-----------------------------------------+-------------------+
[1769515316.413446] [XN-GB-Tray-17:1281331:1]   |                         0..inf | software emulation                      | tcp/enP6p3s0f0np0 |
[1769515316.413446] [XN-GB-Tray-17:1281331:1]   +--------------------------------+-----------------------------------------+-------------------+
[1769515316.415307] [XN-GB-Tray-17:1281331:1]   +--------------------------------+-------------------------------------------------------------+
[1769515316.415308] [XN-GB-Tray-17:1281331:1]   | ucp_context_0 inter-node cfg#2 | remote memory write by ucp_put* from host memory to cuda    |
[1769515316.415308] [XN-GB-Tray-17:1281331:1]   +--------------------------------+-----------------------------------------+-------------------+
[1769515316.415308] [XN-GB-Tray-17:1281331:1]   |                         0..inf | software emulation                      | tcp/enP6p3s0f0np0 |
[1769515316.415308] [XN-GB-Tray-17:1281331:1]   +--------------------------------+-----------------------------------------+-------------------+
[1769515317.343658] [XN-GB-Tray-17:1281331:0]   +--------------------------------+-------------------------------------------------------------------+
[1769515317.343660] [XN-GB-Tray-17:1281331:0]   | ucp_context_0 inter-node cfg#2 | remote memory read by ucp_get*(multi) into cuda/GPU0 from cuda    |
[1769515317.343661] [XN-GB-Tray-17:1281331:0]   +--------------------------------+-----------------------------------------------+-------------------+
[1769515317.343661] [XN-GB-Tray-17:1281331:0]   |                         0..inf | software emulation                            | tcp/enP6p3s0f0np0 |
[1769515317.343662] [XN-GB-Tray-17:1281331:0]   +--------------------------------+-----------------------------------------------+-------------------+
[1769515317.358664] [XN-GB-Tray-17:1281331:1]   +--------------------------------+-------------------------------------------------------------+
[1769515317.358667] [XN-GB-Tray-17:1281331:1]   | ucp_context_0 inter-node cfg#3 | remote memory write by ucp_put* from host memory to host    |
[1769515317.358668] [XN-GB-Tray-17:1281331:1]   +--------------------------------+-----------------------------------------+-------------------+
[1769515317.358668] [XN-GB-Tray-17:1281331:1]   |                         0..inf | zero-copy                               | tcp/enP6p3s0f0np0 |
[1769515317.358669] [XN-GB-Tray-17:1281331:1]   +--------------------------------+-----------------------------------------+-------------------+
[1769515317.367611] [XN-GB-Tray-17:1281331:1]   +--------------------------------+--------------------------------------------------------------------+
[1769515317.367612] [XN-GB-Tray-17:1281331:1]   | ucp_context_0 inter-node cfg#3 | remote memory write by ucp_put* from host memory to cuda/dev[0]    |
[1769515317.367613] [XN-GB-Tray-17:1281331:1]   +--------------------------------+------------------------------------------------+-------------------+
[1769515317.367613] [XN-GB-Tray-17:1281331:1]   |                         0..inf | software emulation                             | tcp/enP6p3s0f0np0 |
[1769515317.367613] [XN-GB-Tray-17:1281331:1]   +--------------------------------+------------------------------------------------+-------------------+
[1769515317.375925] [XN-GB-Tray-17:1281331:1]   +--------------------------------+----------------------------------------------------+
[1769515317.375927] [XN-GB-Tray-17:1281331:1]   | ucp_context_0 inter-node cfg#3 | active message by ucp_am_send* from host memory    |
[1769515317.375927] [XN-GB-Tray-17:1281331:1]   +--------------------------------+--------------------------------+-------------------+
[1769515317.375928] [XN-GB-Tray-17:1281331:1]   |                              0 | short                          | tcp/enP6p3s0f0np0 |
[1769515317.375929] [XN-GB-Tray-17:1281331:1]   |                        1..8184 | zero-copy                      | tcp/enP6p3s0f0np0 |
[1769515317.375929] [XN-GB-Tray-17:1281331:1]   |     8185..18446744073709551590 | multi-frag zero-copy           | tcp/enP6p3s0f0np0 |
[1769515317.375929] [XN-GB-Tray-17:1281331:1]   |      18446744073709551591..inf | multi-frag copy-in             | tcp/enP6p3s0f0np0 |
[1769515317.375929] [XN-GB-Tray-17:1281331:1]   +--------------------------------+--------------------------------+-------------------+

Performance is also very poor (compared to IB).