You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- We have introduced experimental **Encode/Prefill/Decode (E/P/D and all its permutations)** execution. For a detailed explanation, see [Disaggregated Inference Serving](./disaggregation.md)
34
-
- Pluggable **filters**, **scorers**, and **scrapers** for extensible scheduling
34
+
- Pluggable **filters** and **scorers**, backed by a pluggable **data layer**, for extensible scheduling
35
35
36
36
---
37
37
@@ -42,12 +42,12 @@ The design enables:
42
42
- KV cache reuse
43
43
- Load balancing
44
44
- Support multi-model deployments on heterogeneous hardware
45
-
- Enable runtime extensibility with pluggable logic (filters, scorers, scrapers)
45
+
- Enable runtime extensibility with pluggable logic (filters, scorers, data layer)
46
46
- Community-aligned implementation using GIE and Envoy + External Processing (EPP)
47
47
48
48
---
49
49
50
-
## Filters, Scorers, and Scrapers
50
+
## Filters, Scorers, and the Data Layer
51
51
52
52
### Core Design Principles
53
53
@@ -58,14 +58,16 @@ The design enables:
58
58
59
59
### Routing Flow
60
60
61
+
See the upstream [Request Scheduler](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/scheduling.md) doc for the canonical scheduling model.
62
+
61
63
1.**Filtering**
62
64
- Pods in an `InferencePool` go through a sequential chain of filters
63
65
- Pods may be excluded based on criteria like model compatibility, resource usage, or custom logic
64
66
65
67
2.**Scoring**
66
68
- Filtered pods are scored using a weighted set of scorers
67
69
- Scorers currently run sequentially (future: parallel execution)
68
-
- Scorers access a shared datastore populated by scrapers
70
+
- Scorers access a shared datastore populated by the data layer
69
71
70
72
3.**Pod Selection**
71
73
- The highest-scored pod is selected
@@ -77,6 +79,8 @@ The design enables:
77
79
78
80
The llm-d Endpoint Picker relies on a YAML-based configuration—provided either as a file or an in-line parameter—to determine which lifecycle hooks (plugins) are active.
79
81
82
+
See the upstream [Configuration](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/configuration.md) doc for the canonical schema.
83
+
80
84
Specifically, this configuration establishes the following components:
81
85
82
86
-`Plugins`: The specific plugins to instantiate, along with their parameters. Because each instantiated plugin is assigned a unique name, you can configure the same plugin type multiple times if necessary.
@@ -210,12 +214,18 @@ To learn more about the available plugins, check the plugins [README.md](../pkg/
210
214
211
215
---
212
216
213
-
## Metric Scraping
217
+
## Metric Scraping and the Data Layer
218
+
219
+
The data layer follows a Source -> Extract -> Attribute lifecycle:
214
220
215
-
- Scrapers collect metrics (e.g., memory usage, active adapters)
216
-
- Data is injected into the shared datastore for scorers
221
+
- Data sources collect per-endpoint data. Some poll pods periodically, for metrics (e.g., memory
222
+
usage, active adapters) or served models and LoRA adapters (via `/v1/models`); others react to
223
+
endpoint or Kubernetes object change notifications
224
+
- Extractors populate per-endpoint attributes in the shared datastore for scorers
217
225
- Scoring can rely on numerical metrics or metadata (model ID, adapter tags)
218
226
227
+
See the upstream [Data Layer](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/datalayer.md) doc for the canonical model.
228
+
219
229
---
220
230
221
231
## Disaggregated Encode/Prefill/Decode (E/P/D)
@@ -307,3 +317,14 @@ Enable chunked decode via the pd-sidecar flag:
307
317
- [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension)
The fairness_id metric label is now capped at 1000 distinct values; additional fairness IDs are reported under the `other` label value. Flow control metric series for a fairness ID are removed when its flow is garbage collected. This bounds EPP memory and metrics-pipeline load, which were previously unbounded because the label derives from a client-controlled header.
0 commit comments