Skip to content

Commit a550284

Browse files
jtechappsjtechapps
authored
Add more performance test configurations and analysis (#2069)
* [Performance testing] Add new router configurations for performance tests - add light weight router config with random routing and passthrough-parser to simulate the most barebones router configuration. - add precise prefix cache router configuration Signed-off-by: Jacob Murry <jacobmurry@google.com> * Add performance analysis reports Signed-off-by: Jacob Murry <jacobmurry@google.com> --------- Signed-off-by: Jacob Murry <jacobmurry@google.com>
1 parent 493cec1 commit a550284

16 files changed

Lines changed: 1006 additions & 38 deletions

‎test/perf/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@ This directory contains the performance testing pipeline for the Endpoint Picker
66

77
- **`config/`**: Configuration manifests for test runs.
88
- **`router-configs/`**: Router Helm configuration values recipes (e.g. `optimized-baseline.yaml`).
9+
- **`studyanalysis/`**: Detailed analysis reports covering prefix-caching scaling performance, ITL streaming duration impact, input/output token scaling, and high QPS comparisons.
910
- `llm-d-sim-deployment.yaml` / `llm-d-sim-service.yaml`: Kubernetes manifests for deploying the vLLM simulator.
1011
- `shared_prefix_job1.yaml`: Performance test workload specification defining load stages, API target, and request distributions.
11-
- **`results/`**: Execution results logged as Markdown tables, grouped by test recipe name (e.g., `results/optimized-baseline/`). If profiling is enabled, the `.pprof` and `.svg` files are saved here as well.
1212
- **`run_nightly_perf.py`**: The Python orchestrator script responsible for test namespace setup, EPP and simulator deployments, pprof profiling collection, metrics scraping, and markdown generation. It assumes that `kubectl` is already configured to target an active Kubernetes cluster.
1313

1414
---

‎test/perf/config/llm-d-sim-deployment-1m-context.yaml‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
labels:
5+
app: llm-d-sim
6+
name: llm-d-sim
7+
namespace: llm-d-sim
8+
spec:
9+
replicas: 10
10+
selector:
11+
matchLabels:
12+
app: llm-d-sim
13+
template:
14+
metadata:
15+
labels:
16+
app: llm-d-sim
17+
spec:
18+
topologySpreadConstraints:
19+
- maxSkew: 1
20+
topologyKey: topology.kubernetes.io/zone
21+
whenUnsatisfiable: DoNotSchedule
22+
labelSelector:
23+
matchLabels:
24+
app: llm-d-sim
25+
containers:
26+
- args:
27+
- --model
28+
- deepseek-ai/DeepSeek-V4-Flash
29+
- --port
30+
- '8000'
31+
- --max-model-len
32+
- '1000000'
33+
- --max-num-seqs
34+
- '256'
35+
- --max-waiting-queue-length
36+
- '2048'
37+
- --inter-token-latency
38+
- 0.1ms
39+
- --prefill-time-per-token
40+
- 0.1us
41+
- --prefill-overhead
42+
- 5ms
43+
- --enable-kvcache
44+
- --enable-prefix-caching
45+
- --kv-cache-size
46+
- '1000000'
47+
- --block-size
48+
- '16'
49+
# - --render-timeout
50+
# - 120s
51+
# - --mm-render-timeout
52+
# - 120s
53+
- --force-dummy-tokenizer
54+
- --max-request-body-size-mb
55+
- '10'
56+
- '--v'
57+
- '4'
58+
env:
59+
- name: POD_NAME
60+
valueFrom:
61+
fieldRef:
62+
fieldPath: metadata.name
63+
- name: POD_NAMESPACE
64+
valueFrom:
65+
fieldRef:
66+
fieldPath: metadata.namespace
67+
- name: POD_IP
68+
valueFrom:
69+
fieldRef:
70+
fieldPath: status.podIP
71+
- name: GOMEMLIMIT
72+
value: 12GiB
73+
- name: GOGC
74+
value: "50"
75+
image: ghcr.io/llm-d/llm-d-inference-sim:v0.10.2
76+
imagePullPolicy: IfNotPresent
77+
livenessProbe:
78+
httpGet:
79+
path: /health/ready
80+
port: 8000
81+
initialDelaySeconds: 15
82+
periodSeconds: 10
83+
name: vllm-sim
84+
ports:
85+
- containerPort: 8000
86+
name: http
87+
protocol: TCP
88+
- containerPort: 5557
89+
name: zmq-kv-events
90+
protocol: TCP
91+
readinessProbe:
92+
httpGet:
93+
path: /health/ready
94+
port: 8000
95+
initialDelaySeconds: 10
96+
periodSeconds: 5
97+
resources:
98+
requests:
99+
cpu: '16'
100+
memory: 16Gi

‎test/perf/config/llm-d-sim-deployment-zmq.yaml‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
labels:
5+
app: llm-d-sim
6+
name: llm-d-sim
7+
namespace: llm-d-sim
8+
spec:
9+
replicas: 10
10+
selector:
11+
matchLabels:
12+
app: llm-d-sim
13+
template:
14+
metadata:
15+
labels:
16+
app: llm-d-sim
17+
llm-d.ai/inference-serving: "true"
18+
spec:
19+
topologySpreadConstraints:
20+
- maxSkew: 1
21+
topologyKey: topology.kubernetes.io/zone
22+
whenUnsatisfiable: DoNotSchedule
23+
labelSelector:
24+
matchLabels:
25+
app: llm-d-sim
26+
containers:
27+
- args:
28+
- --model
29+
- Qwen/Qwen3-8B
30+
- --port
31+
- '8000'
32+
- --max-model-len
33+
- '131072'
34+
- --max-num-seqs
35+
- '256'
36+
- --max-waiting-queue-length
37+
- '2048'
38+
- --inter-token-latency
39+
- 1ms
40+
- --prefill-time-per-token
41+
- 4us
42+
- --prefill-overhead
43+
- 5ms
44+
- --enable-kvcache
45+
- --kv-cache-size
46+
- '10000000'
47+
- --block-size
48+
- '16'
49+
- --render-timeout
50+
- 120s
51+
- --mm-render-timeout
52+
- 120s
53+
- --zmq-endpoint
54+
- tcp://127.0.0.1:5558
55+
env:
56+
- name: POD_NAME
57+
valueFrom:
58+
fieldRef:
59+
fieldPath: metadata.name
60+
- name: POD_NAMESPACE
61+
valueFrom:
62+
fieldRef:
63+
fieldPath: metadata.namespace
64+
- name: POD_IP
65+
valueFrom:
66+
fieldRef:
67+
fieldPath: status.podIP
68+
image: ghcr.io/llm-d/llm-d-inference-sim:v0.10.2
69+
imagePullPolicy: IfNotPresent
70+
livenessProbe:
71+
httpGet:
72+
path: /health/ready
73+
port: 8000
74+
initialDelaySeconds: 15
75+
periodSeconds: 10
76+
name: vllm-sim
77+
ports:
78+
- containerPort: 8000
79+
name: http
80+
protocol: TCP
81+
readinessProbe:
82+
httpGet:
83+
path: /health/ready
84+
port: 8000
85+
initialDelaySeconds: 10
86+
periodSeconds: 5
87+
resources:
88+
requests:
89+
cpu: '1'
90+
memory: 8Gi
91+
- name: zmq-proxy
92+
image: quay.io/mayab/zmq-listener:latest
93+
imagePullPolicy: IfNotPresent
94+
command: ["python", "-c"]
95+
args:
96+
- |
97+
import zmq
98+
ctx = zmq.Context()
99+
frontend = ctx.socket(zmq.SUB)
100+
frontend.bind("tcp://127.0.0.1:5558")
101+
frontend.setsockopt_string(zmq.SUBSCRIBE, "")
102+
backend = ctx.socket(zmq.PUB)
103+
backend.bind("tcp://0.0.0.0:5557")
104+
while True:
105+
msg = frontend.recv_multipart()
106+
if msg:
107+
parts = msg[0].split(b'@')
108+
if len(parts) == 3:
109+
parts[1] = parts[1] + b':8000'
110+
msg[0] = b'@'.join(parts)
111+
backend.send_multipart(msg)
112+
ports:
113+
- containerPort: 5557
114+
name: zmq-kv-events
115+
protocol: TCP
116+
initContainers:
117+
- args:
118+
- launch
119+
- render
120+
- Qwen/Qwen3-8B
121+
- --port=8082
122+
- --max-model-len=131072
123+
command:
124+
- vllm
125+
env:
126+
- name: VLLM_ALLOW_LONG_MAX_MODEL_LEN
127+
value: '1'
128+
- name: TOKENIZERS_PARALLELISM
129+
value: 'true'
130+
- name: OMP_NUM_THREADS
131+
value: '4'
132+
- name: HF_TOKEN
133+
valueFrom:
134+
secretKeyRef:
135+
key: token
136+
name: hf-secret
137+
image: vllm/vllm-openai-cpu:v0.19.1
138+
imagePullPolicy: IfNotPresent
139+
livenessProbe:
140+
failureThreshold: 10
141+
httpGet:
142+
path: /health
143+
port: 8082
144+
initialDelaySeconds: 15
145+
periodSeconds: 10
146+
name: vllm-render
147+
ports:
148+
- containerPort: 8082
149+
name: http-render
150+
protocol: TCP
151+
readinessProbe:
152+
failureThreshold: 10
153+
httpGet:
154+
path: /health
155+
port: 8082
156+
initialDelaySeconds: 10
157+
periodSeconds: 5
158+
resources:
159+
requests:
160+
cpu: '1'
161+
memory: 8Gi
162+
restartPolicy: Always

‎test/perf/config/router-configs/optimized-baseline.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,9 @@ router:
2626
plugins:
2727
- type: queue-scorer
2828
- type: kv-cache-utilization-scorer
29+
- type: approx-prefix-cache-producer
30+
parameters:
31+
maxPrefixTokensToMatch: 100000
2932
- type: prefix-cache-scorer
3033
- type: no-hit-lru-scorer
3134
schedulingProfiles:

0 commit comments

Comments
 (0)