You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- We have introduced experimental **Encode/Prefill/Decode (E/P/D and all its permutations)** execution. For a detailed explanation, see [Disaggregated Inference Serving](./disaggregation.md)
34
-
- Pluggable **filters**, **scorers**, and **scrapers** for extensible scheduling
34
+
- Pluggable **filters** and **scorers**, backed by a pluggable **data layer**, for extensible scheduling
35
35
36
36
---
37
37
@@ -42,12 +42,12 @@ The design enables:
42
42
- KV cache reuse
43
43
- Load balancing
44
44
- Support multi-model deployments on heterogeneous hardware
45
-
- Enable runtime extensibility with pluggable logic (filters, scorers, scrapers)
45
+
- Enable runtime extensibility with pluggable logic (filters, scorers, data layer)
46
46
- Community-aligned implementation using GIE and Envoy + External Processing (EPP)
47
47
48
48
---
49
49
50
-
## Filters, Scorers, and Scrapers
50
+
## Filters, Scorers, and the Data Layer
51
51
52
52
### Core Design Principles
53
53
@@ -58,14 +58,16 @@ The design enables:
58
58
59
59
### Routing Flow
60
60
61
+
See the upstream [Request Scheduler](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/scheduling.md) doc for the canonical scheduling model.
62
+
61
63
1.**Filtering**
62
64
- Pods in an `InferencePool` go through a sequential chain of filters
63
65
- Pods may be excluded based on criteria like model compatibility, resource usage, or custom logic
64
66
65
67
2.**Scoring**
66
68
- Filtered pods are scored using a weighted set of scorers
67
69
- Scorers currently run sequentially (future: parallel execution)
68
-
- Scorers access a shared datastore populated by scrapers
70
+
- Scorers access a shared datastore populated by the data layer
69
71
70
72
3.**Pod Selection**
71
73
- The highest-scored pod is selected
@@ -77,6 +79,8 @@ The design enables:
77
79
78
80
The llm-d Endpoint Picker relies on a YAML-based configuration—provided either as a file or an in-line parameter—to determine which lifecycle hooks (plugins) are active.
79
81
82
+
See the upstream [Configuration](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/configuration.md) doc for the canonical schema.
83
+
80
84
Specifically, this configuration establishes the following components:
81
85
82
86
-`Plugins`: The specific plugins to instantiate, along with their parameters. Because each instantiated plugin is assigned a unique name, you can configure the same plugin type multiple times if necessary.
@@ -210,12 +214,18 @@ To learn more about the available plugins, check the plugins [README.md](../pkg/
210
214
211
215
---
212
216
213
-
## Metric Scraping
217
+
## Metric Scraping and the Data Layer
218
+
219
+
The data layer follows a Source -> Extract -> Attribute lifecycle:
214
220
215
-
- Scrapers collect metrics (e.g., memory usage, active adapters)
216
-
- Data is injected into the shared datastore for scorers
221
+
- Data sources collect per-endpoint data. Some poll pods periodically, for metrics (e.g., memory
222
+
usage, active adapters) or served models and LoRA adapters (via `/v1/models`); others react to
223
+
endpoint or Kubernetes object change notifications
224
+
- Extractors populate per-endpoint attributes in the shared datastore for scorers
217
225
- Scoring can rely on numerical metrics or metadata (model ID, adapter tags)
218
226
227
+
See the upstream [Data Layer](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/datalayer.md) doc for the canonical model.
228
+
219
229
---
220
230
221
231
## Disaggregated Encode/Prefill/Decode (E/P/D)
@@ -307,3 +317,14 @@ Enable chunked decode via the pd-sidecar flag:
307
317
- [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension)
|`estimate.video.frames.sampled.temporalPatchSize`| – | Sampled-mode: merge every N sampled frames into one token group (qwen3-vl = 2; <2 = no merge). |
91
+
|`estimate.video.frames.strided.defaultSourceFPS`|`24`| Strided-mode source frame rate; fallback for the `x-llm-d-video-fps` header. |
92
+
|`estimate.video.frames.strided.frameStride`|`1`| Strided-mode divisor: keep every Nth source frame. |
93
+
|`estimate.video.defaultResolution`| 640×360 | Per-frame resolution for dynamic tokens-per-frame; fallback for the `x-llm-d-video-resolution` header. |
94
+
|`estimate.video.defaultDuration`|`10`| Video length in seconds for frame counting; fallback for the `x-llm-d-video-duration-seconds` header. |
95
+
|`estimate.video.maxVideoTokens`| – | Overall placeholder cap for a video (0 = uncapped). |
96
+
66
97
## Failure mode
67
98
68
99
Per-request errors are returned to the Director, which currently logs and
0 commit comments