feat: Variable DP rank

[agolajko]

Overview:

Encodes prefill dp_rank to bootstrap_room so SGLang can use to find the rank to be used for KV cache retrieval during decode

Details:

Problem

SGLang decode workers use bootstrap_room % prefill_dp_size (SGLang source) to identify which prefill worker holds their KV cache. Currently, Dynamo assigns bootstrap_room randomly, which doesn't encode the prefill worker information needed for this calculation.

This forces decode workers to be routed to the same rank as their corresponding prefill workers to ensure correct KV cache retrieval. As a result, prefill and decode must have matching DP sizes, preventing configurations like 4 prefill workers with 8 decode workers and limiting routing flexibility.

Related Issues:

• Relates to GitHub issue: [CONTRIBUTION]: Support variable DP rank + P/D routing in Slgnag #5638

Summary by CodeRabbit

• Refactor
  • Improved prefill router decision-making with enhanced determinism and observability for internal routing logic.
  • Refined router configuration handling to support more consistent distribution of requests across available resources.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi agolajko! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

Walkthrough

The prefill router's bootstrap_room generation was refactored from random generation to a deterministic encoding scheme. A static AtomicU64 counter now accumulates base values combined with dp_rank to compute bootstrap_room. Prefill decision pool size is now derived from the active router and added to logging context.

Changes

Cohort / File(s)	Summary
Prefill Router Bootstrap Logic lib/llm/src/kv_router/prefill_router.rs	Replaced random bootstrap_room generation with deterministic calculation using static AtomicU64 counter. Added AtomicU64 import and BOOTSTRAP_ROOM_COUNTER static. Compute prefill_dp_size from KV or non-KV active router and inject into bootstrap_info logging. Extended logic to handle both router types when deriving pool size.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Poem

🐰 A counter now ticks, atomic and true,
Bootstrap rooms born from logic, not brew,
Determinism blooms where chaos once grew,
Each decision point nestled, precise in its queue! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Title check	❓ Inconclusive	The title 'feat: Variable DP rank' is vague and does not clearly convey the main objective of encoding prefill dp_rank into bootstrap_room for SGLang decode routing.	Clarify the title to specifically mention bootstrap_room encoding or decode routing, e.g., 'feat: Encode prefill dp_rank in bootstrap_room for decode routing'.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description check	✅ Passed	The PR description covers all required template sections with clear explanation of the problem, solution, and related issue, meeting the standard for merge readiness.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}