Skip to content

[P/D][Serve LLM] Fix NIXL side channel host to use Ray node IP for cross-node P/D disaggregation #60817

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kouroshHakha wants to merge 1 commit into
base: master
Choose a base branch
from
+5 −2
Open

[P/D][Serve LLM] Fix NIXL side channel host to use Ray node IP for cross-node P/D disaggregation #60817

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions python/ray/llm/_internal/serve/engines/vllm/kv_transfer/nixl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import os

import ray
from ray.llm._internal.serve.engines.vllm.kv_transfer.base import (
BaseConnectorBackend,
)
Expand All @@ -22,10 +23,12 @@ def _set_side_channel_port(self):

def _set_side_channel_host(self):
from vllm import envs as vllm_envs
from vllm.utils.network_utils import get_ip

if not vllm_envs.is_set("VLLM_NIXL_SIDE_CHANNEL_HOST"):
os.environ["VLLM_NIXL_SIDE_CHANNEL_HOST"] = get_ip()
# Use Ray's node IP (internal/cluster IP) instead of vLLM's
# get_ip() which can return external/public IPs on hostNetwork
# pods, causing cross-node NIXL handshakes to fail.
os.environ["VLLM_NIXL_SIDE_CHANNEL_HOST"] = ray.util.get_node_ip_address()
Comment on lines 24 to +31
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For consistency with the local import pattern used for vllm dependencies in this file, it's preferable to import get_node_ip_address within this method. This also allows for the removal of the top-level import ray, keeping the module's namespace cleaner as it's only used here.

Suggested change
def _set_side_channel_host(self):
from vllm import envs as vllm_envs
from vllm.utils.network_utils import get_ip
if not vllm_envs.is_set("VLLM_NIXL_SIDE_CHANNEL_HOST"):
os.environ["VLLM_NIXL_SIDE_CHANNEL_HOST"] = get_ip()
# Use Ray's node IP (internal/cluster IP) instead of vLLM's
# get_ip() which can return external/public IPs on hostNetwork
# pods, causing cross-node NIXL handshakes to fail.
os.environ["VLLM_NIXL_SIDE_CHANNEL_HOST"] = ray.util.get_node_ip_address()
def _set_side_channel_host(self):
from vllm import envs as vllm_envs
from ray.util import get_node_ip_address
if not vllm_envs.is_set("VLLM_NIXL_SIDE_CHANNEL_HOST"):
# Use Ray's node IP (internal/cluster IP) instead of vLLM's
# get_ip() which can return external/public IPs on hostNetwork
# pods, causing cross-node NIXL handshakes to fail.
os.environ["VLLM_NIXL_SIDE_CHANNEL_HOST"] = get_node_ip_address()


def setup(self) -> None:
"""Initialize the NIXL connector backend.
Expand Down