High TTFT and Throughput Drop in gRPC Mode Under High Concurrency (vs HTTP Mode)

[yansiyu550]

Under the same benchmarking conditions, I tested the Router in both HTTP and gRPC modes using our benchmark tool.

What I observed is:

• HTTP Load Balancer vs. HTTP Router: the performance is almost identical.
• gRPC Router: the cache hit rate is significantly higher than HTTP Router.
• However, once the load increases, the overall performance of the gRPC Router drops sharply.

My question is:
Is this behavior expected?
Could it be because the current gRPC implementation has not been fully optimized for high-concurrency scenarios (e.g., connection management, flow control, or thread scheduling)?

I’m using four 1P1D (1 Prefill + 1 Decode) pairs. The router command is as follows:

python3 -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill grpc://localhost:3880 9441 \
  --prefill grpc://localhost:3882 9442 \
  --prefill grpc://localhost:3884 9443 \
  --prefill grpc://localhost:3886 9444 \
  --decode grpc://localhost:3881 \
  --decode grpc://localhost:3883 \
  --decode grpc://localhost:3885 \
  --decode grpc://localhost:3887 \
  --policy cache_aware \
  --model-path /work/models/Qwen3-4B \
  --tokenizer-path /work/models/Qwen3-4B \
  --reasoning-parser qwen3 \
  --host 0.0.0.0 --port 39998

python3 -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://localhost:3880 9441 \
  --prefill http://localhost:3882 9442 \
  --prefill http://localhost:3884 9443 \
  --prefill http://localhost:3886 9444 \
  --decode http://localhost:3881 \
  --decode http://localhost:3883 \
  --decode http://localhost:3885 \
  --decode http://localhost:3887 \
  --policy cache_aware \
  --model-path /work/models/Qwen3-4B \
  --tokenizer-path /work/models/Qwen3-4B \
  --reasoning-parser qwen3 \
  --host 0.0.0.0 --port 39998