feat: optimize router config guidelines for agentic workloads

[Madhumasa84]

What type of PR is this?

/kind feature
/kind documentation

---

What this PR does / why we need it

Agentic workloads have different routing characteristics compared to traditional stateless inference. They benefit significantly from long-session stickiness and session affinity because requests within the same session frequently share context but rarely share prefixes across different sessions beyond the initial system prompt.

This PR improves the documentation and recommendations for configuring the router for agentic workloads by:

• Evaluating Session Affinity

  • Evaluated the Session Affinity plugin for agentic workloads where cross-session prefix sharing is minimal.
  • Added agentic workload examples to sessionaffinity/README.md.

• Documenting Prefix Match Length Awareness

  • Updated prefix/README.md to recommend enabling prefixMatchLength awareness with:
    • matchLengthWeight: 0.8
    • matchLengthScaleTokens: 16000
  • These settings increase stickiness for long-running sessions while reducing unnecessary migration costs for shorter sessions.

• Benchmarking Agentic Traces

  • Evaluated the optimized configuration using inference-perf's otel_trace_replay.
  • Successfully replayed agentic traces without errors while maintaining excellent TTFT and TPOT latency.

• Well-Lit Path

  • Deployment manifests and the agentic-serving guide are being submitted separately in the companion PR to the llm-d repository.

Reference:

• Length-aware prefix affinity implementation: Add support for length-based scoring in prefix cache plugin #1459

Benchmark Results

Ran the OTel trace replay (simple_chain.json) against the optimized gateway.

Throughput

• Request Rate (Achieved): 1.8 Req/s
• Input Tokens/s: 44.6
• Output Tokens/s: 16.6
• Total Tokens/s: 61.2

Latency

• Request Latency

  • Mean: 59.8 ms
  • Median: 30.2 ms
  • P90: 106.7 ms

• Token Generation (Normalized TPOT)

  • Mean: 7.0 ms
  • Median: 2.7 ms
  • P90: 13.1 ms

Session Summary

• Total Sessions: 1
• Total Requests: 3
• Success Rate: 100%
• Error Rate: 0.0%
• Mean Session Duration: 1.71 s

---

Which issue(s) this PR fixes

Fixes #1867

---

Release note

Updated documentation for prefix matching and session affinity routing to optimize agentic workloads, including recommendations for prefix match length awareness and benchmark results demonstrating improved routing behavior.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

🚨 Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

[liu-cong]

Thanks for adding this. Can you explain more on your benchmark setup? For example, what model or model server and hardware you deployed the stack on and can you also compare the new configurations with the existing one?

[Madhumasa84]

Thanks for the review, @liu-cong! Here are the details of the benchmark setup and a comparison between the standard routing configuration and the new optimized one.

Benchmark Setup

• Hardware: Local Machine (13th Gen Intel(R) Core(TM) i5-1335U, 16GB RAM)
• Model Server: vLLM (running in a local Kubernetes kind cluster)
• Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Workload/Trace: OTel trace replay using inference-perf on simple_chain.json (an agentic trace where cross-session prefix sharing is minimal).

Configuration Comparison

The Baseline configuration used standard routing without length-aware prefix caching or session affinity. The Optimized configuration includes:

• matchLengthWeight: 0.8
• matchLengthScaleTokens: 16000
• session-affinity-scorer

Metric	Baseline (Standard Config)	Optimized (New Config)	Improvement
Request Rate (Req/s)	1.7	1.8	+5.8%
Total Tokens/s	59.4	61.2	+3.0%
Median Request Latency	26.1 ms	30.2 ms	—
P90 Request Latency	149.0 ms	106.7 ms	-28.3%
Median TPOT	2.5 ms	2.7 ms	—
P90 TPOT	18.5 ms	13.1 ms	-29.1%

Summary

By keeping the rapidly diverging chat history sticky to the original pod via session affinity, while allowing shorter new sessions to migrate using the length-aware prefix cache, the router successfully processed the agentic traces while significantly improving tail (P90) latency by nearly 30%. The optimized configuration avoids unnecessary and expensive migration costs for longer-running sessions.

Note: While the absolute latency and throughput numbers are specific to this lightweight local CPU setup with a 1.1B model, the relative ~30% improvement in P90 latency demonstrates the router optimization working as intended. In production deployments with larger models (e.g., 8B+) where KV-cache misses are significantly more expensive to recompute, this configuration is expected to provide an even greater reduction in tail latencies.

Let me know if you need any additional details as well

[vMaroon]

I think we require more extensive benchmarking before introducing a change to the default deployments. This benchmark does not really reflect the agentic-workloads / deployments. Though the intuition is right, and we're working in these directions.

/hold

[Madhumasa84]

Thanks for the review and explanation, @vMaroon. I completely agree with the /hold.

My local benchmark was just meant as a functional proof-of-concept to validate the routing logic. You are absolutely right that it doesn't reflect the long context windows and heavy concurrency of a true production agentic deployment.

Since I don't currently have access to a multi-node GPU cluster, I'm unable to run a high-concurrency stress test on a larger model to provide that data right now.