Llmd leader worker clean 2p2d enable#2094
Draft
shikamd123 wants to merge 2 commits into
Draft
Conversation
Enable MoRI-IO WRITE-mode and Wide-EP features for production use: 1. Set MoRIIOFeatureEnabled = true to activate the feature gate 2. Add DNS hostname resolution for --moriio-remote-hosts and --moriio-decode-hosts flags (resolves DNS to IPs at startup) 3. Support BOTH raw IPs (static deployments) and DNS names (LWS) 4. Update MORIIO_README.md with enabled status and usage examples Supported topologies: - 1P1D (DP=EP=8, TP=1): Intra-node Wide-EP - 2P2D (DP=EP=16, TP=1): Inter-node Wide-EP with LeaderWorkerSet Host address options: - Raw IPs: Pass through unchanged (e.g., 10.0.0.1) - for static deployments - DNS names: Resolved to IPs at startup (e.g., pod.ns.svc) - for LWS Signed-off-by: Shiksha Patel <shiksha.patel@amd.com>
github-actions
Bot
added
size/XL
shikamd123
marked this pull request as draft
July 20, 2026 23:57
github-actions
Bot
added
kind/feature
Contributor
|
@shikamd123 Please remove the file PR_DESCRIPTION.md and add any missing text to the PR description. |
In serial WRITE dispatch the prefill leg runs first and returns the DP rank it actually ran on (remote_dp_rank in kv_transfer_params). Pin the decode leg to that returned rank instead of independently re-hashing the request id, so both legs agree without each side hashing. This is the router-applies-the-connector-returned-rank model from the vLLM PR #45043 review: the connector returns the rank, the router sets the x-data-parallel-rank header on decode. Falls back to the blake2s hash only when the prefill response omits remote_dp_rank (older vLLM without the returnable rank). Signed-off-by: Shiksha Patel <shiksha.patel@amd.com>
shikamd123
commented
Jul 21, 2026
| require.Empty(t, result) | ||
| }) | ||
|
|
||
| t.Run("raw IPv4 addresses are passed through", func(t *testing.T) { |
Contributor
Author
There was a problem hiding this comment.
remove the IP addresses, there should be DNS resolution, test it and update it.
shikamd123
commented
Jul 21, 2026
| opts.MoRIIODecodePodIP = "10.0.1.1" | ||
| opts.MoRIIODPSize = 16 | ||
| opts.MoRIIODPSizeLocal = 8 | ||
| opts.MoRIIORemoteHosts = []string{"10.0.0.1", "10.0.0.2"} |
Contributor
Author
There was a problem hiding this comment.
do we need IP addresses?
shikamd123
force-pushed
the
llmd_leader_worker_clean_2p2d_enable
branch
from
July 21, 2026 23:52
1e1c0e6 to
1c0c816
Compare
Contributor
|
🚨 Unsigned commits detected! Please sign your commits. For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation. |
github-actions
Bot
added
size/L
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
feature
What this PR does / why we need it:
Enables AMD MoRI-IO WRITE-mode and Wide-EP disaggregation in the llm-d
sidecar, so a single DP world can be split across pods for prefill/decode
serving on ROCm (MI300X).
(
--moriio-write-mode,--moriio-parallel-dispatch,--moriio-dp-size,--moriio-dp-size-local,--moriio-remote-hosts,--moriio-decode-hosts).parallel WRITE (concurrent prefill/decode) dispatch.
is pinned to the
remote_dp_rankthe prefill leg actually returned (fallingback to a stable hash only if omitted); in parallel mode the rank is pinned up
front with
remote_dp_rank_override=true. Both legs agree without each sideindependently hashing the request id.
at startup, LWS-friendly); raw IPs remain accepted for backward compatibility.
MORIIO_README.mdand adds option tests.Companion to vLLM PR vllm-project/vllm#vllm-project/vllm#45043