[serve][llm][doc] Document direct streaming and ingress request router#63860

Open

eicherseiji wants to merge 22 commits into

ray-project:masterfrom

eicherseiji:seiji/docs-llm-direct-streaming

Commits on Jun 4, 2026

[serve][llm][doc] Document direct streaming and ingress request router
eicherseiji
committed
[serve][llm][doc] Frame ingress request router as internal; add body-forwarding and TCP_NODELAY notes
eicherseiji
committed
[serve][llm][doc] Keep routing-policies note to the public surface
eicherseiji
committed
[serve][llm][doc] Say 'implementation detail' instead of 'internal' in routing-policies note
eicherseiji
committed
[serve][llm][doc] Make direct streaming build/deploy snippet a tested doc_code example
eicherseiji
committed
[serve][llm][doc] Address review: tested router snippet, dashboard check, truncation scope
eicherseiji
committed
[serve][llm][doc] Drop hardcoded bufsize default to avoid staleness
eicherseiji
committed
[serve][llm][doc] Frame truncation around TTFT, add metrics + multiplex limits
eicherseiji
committed
[serve][llm][doc] Fold workarounds into limitation bullets, trim truncation note
eicherseiji
committed
[serve][llm][doc] Trim non-limitation bullets from direct streaming limitations
eicherseiji
committed
[serve][llm][doc] Keep limitations to genuine functional constraints
eicherseiji
committed
[serve][llm][doc] Reframe 'When to use' as the low-latency path, not a limitations restatement
eicherseiji
committed

Commits on Jun 8, 2026

Commits on Jun 9, 2026

[serve][llm][doc] Add serve status and dashboard screenshot to direct streaming verification
eicherseiji
committed

Commits on Jun 10, 2026