Skip to content

Commit 451239f

Browse files
BenjaminBraunDevBenjaminBraunDev
authored
Add EPP model server protocol to docs (llm-d#1267)
* Add EPP model server protocol to docs Signed-off-by: BenjaminBraunDev <benjaminbraun@google.com> * Rename model-server-protocol to plugin-metric-protocol, adjust to focus on plugins adding a producer and consumer section to table Signed-off-by: BenjaminBraunDev <benjaminbraun@google.com> * Break metrics out of table, individual sections for each metric Signed-off-by: BenjaminBraunDev <benjaminbraun@google.com> * Remove doc link to fix lint Signed-off-by: BenjaminBraunDev <benjaminbraun@google.com> --------- Signed-off-by: BenjaminBraunDev <benjaminbraun@google.com>
1 parent 508f15a commit 451239f

1 file changed

Lines changed: 121 additions & 0 deletions

File tree

‎docs/plugin-metric-protocol.md‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# Plugin Metric Protocol
2+
3+
This document describes the contract the EPP expects from model servers
4+
it routes traffic to. Because the EPP uses a pluggable architecture, the
5+
requirements below describe what is needed to use the built-in plugins;
6+
individual plugins may relax or extend these requirements.
7+
8+
## Metrics Reporting
9+
10+
The inference extension scrapes metrics from the model servers to make optimal request scheduling
11+
decisions. The model servers MUST provide the following metrics via a Prometheus endpoint. The exact
12+
metric names don't necessarily need to be the same as the recommended names here, however the
13+
metric types and semantics MUST follow this doc.
14+
15+
Each metric below lists the plugins that need it under **Required by**. You only need to expose metrics for plugins you are using.
16+
17+
### TotalQueuedRequests
18+
19+
The current total number of requests in the queue.
20+
21+
- **Type:** Gauge
22+
- **Required by:** `queue-scorer`, `load-aware-scorer`, `latency-scorer` (via `predicted-latency`)
23+
24+
| Model server | Metric |
25+
| --- | --- |
26+
| vLLM | `vllm:num_requests_waiting` |
27+
| SGLang | `sglang:num_queue_reqs` |
28+
| Triton TensorRT-LLM | `nv_trt_llm_request_metrics{request_type=waiting}` |
29+
| trtllm-serve | `trtllm_num_requests_waiting` |
30+
31+
### TotalRunningRequests
32+
33+
The current total number of requests actively being served on the model server.
34+
35+
- **Type:** Gauge
36+
- **Required by:** `running-requests-size-scorer`, `latency-scorer` (via `predicted-latency`)
37+
38+
| Model server | Metric |
39+
| --- | --- |
40+
| vLLM | `vllm:num_requests_running` |
41+
| SGLang | `sglang:num_running_reqs` |
42+
| Triton TensorRT-LLM | `nv_trt_llm_request_metrics{request_type=scheduled}` |
43+
| trtllm-serve | `trtllm_num_requests_running` |
44+
45+
### KVCacheUtilization
46+
47+
The current KV cache utilization in percentage.
48+
49+
- **Type:** Gauge
50+
- **Required by:** `kv-cache-utilization-scorer`, `latency-scorer` (via `predicted-latency`)
51+
52+
| Model server | Metric |
53+
| --- | --- |
54+
| vLLM | `vllm:kv_cache_usage_perc` |
55+
| SGLang | `sglang:token_usage` |
56+
| Triton TensorRT-LLM | `nv_trt_llm_kv_cache_block_metrics{kv_cache_block_type=fraction}` |
57+
| trtllm-serve | `trtllm_kv_cache_utilization` |
58+
59+
### BlockSize (optional)
60+
61+
The block size in tokens to allocate memory. Used to auto-tune the approximate prefix cache.
62+
If absent, the value is taken from the `approximate-prefix` plugin's `BlockSizeTokens` config.
63+
64+
- **Type:** Labeled/Gauge
65+
- **Required by:** `prefix-cache-scorer`, `prefix-cache-affinity-filter` (via `approximate-prefix` when `AutoTune` is enabled)
66+
67+
| Model server | Metric | Label |
68+
| --- | --- | --- |
69+
| vLLM | `vllm:cache_config_info` | `block_size` |
70+
| SGLang | `sglang:cache_config_info` | `page_size` |
71+
| Triton TensorRT-LLM | `nv_trt_llm_kv_cache_block_metrics{kv_cache_block_type=tokens_per}` | — |
72+
| trtllm-serve | `trtllm_kv_cache_tokens_per_block` | — |
73+
74+
### NumGPUBlocks (optional)
75+
76+
The total number of blocks in the HBM KV cache. Used to auto-tune the approximate prefix cache.
77+
If absent, the value is taken from the `approximate-prefix` plugin's `LRUCapacityPerServer` config.
78+
79+
- **Type:** Labeled/Gauge
80+
- **Required by:** `prefix-cache-scorer`, `prefix-cache-affinity-filter` (via `approximate-prefix` when `AutoTune` is enabled)
81+
82+
| Model server | Metric | Label |
83+
| --- | --- | --- |
84+
| vLLM | `vllm:cache_config_info` | `num_gpu_blocks` |
85+
| SGLang | `sglang:cache_config_info` | `num_pages` |
86+
| Triton TensorRT-LLM | `nv_trt_llm_kv_cache_block_metrics{kv_cache_block_type=max}` | — |
87+
| trtllm-serve | `trtllm_kv_cache_max_blocks` | — |
88+
89+
## LoRA Adapter Serving
90+
91+
**Required by:** `lora-affinity-scorer`
92+
93+
Model servers that support dynamic LoRA serving can benefit from the LoRA affinity algorithm. Note
94+
the current LoRA affinity algorithm in this EPP is highly biased towards vLLM's current
95+
dynamic LoRA implementation.
96+
97+
The model servers MUST support serving a LoRA adapter specified in the `model` argument of the
98+
request, provided the requested adapter is valid.
99+
100+
The model server MUST expose the following LoRA adapter metrics via the same Prometheus endpoint:
101+
102+
* Metric name implemented in vLLM: `vllm:lora_requests_info`
103+
* Metric type: Gauge
104+
* Metric value: The last updated timestamp (so the EPP can find the latest).
105+
* Metric labels:
106+
* `max_lora`: The maximum number of adapters that can be loaded to GPU memory to serve a batch.
107+
Requests will be queued if the model server has reached MaxActiveAdapter and cannot load the
108+
requested adapter. Example: `"max_lora": "8"`.
109+
* `running_lora_adapters`: A comma separated list of adapters that are currently loaded in GPU
110+
memory and ready to serve requests. Example: `"running_lora_adapters": "adapter1, adapter2"`
111+
* `waiting_lora_adapters`: A comma separated list of adapters that are waiting to be served.
112+
Example: `"waiting_lora_adapters": "adapter1, adapter2"`
113+
114+
## Prefix Cache Reuse
115+
116+
**Required by:** `precise-prefix-cache-scorer`, `prefix-cache-scorer`, `prefix-cache-affinity-filter`
117+
118+
The EPP supports prefix cache optimized request scheduling via the
119+
[precise prefix cache plugin](../pkg/epp/framework/plugins/scheduling/scorer/preciseprefixcache/README.md).
120+
To benefit from optimal prefix-aware request scheduling, model servers SHOULD support prefix
121+
cache reuse, such as the [vllm automatic prefix caching](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html) feature.

0 commit comments

Comments
 (0)