Skip to content

docs: align architecture.md terminology with the data layer #2084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
elevran merged 3 commits into from
Jul 22, 2026
+30 −9
Merged

docs: align architecture.md terminology with the data layer #2084

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
eac571a
docs: align architecture.md terminology with the data layer
chethanuk Jul 19, 2026
e96ad4e
docs: rename Metric Scraping heading to include the data layer
chethanuk Jul 19, 2026
b1ce1ae
docs: data sources collect more than metrics
chethanuk Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 28 additions & 9 deletions docs/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@

- [Overview](#overview)
- [Core Goals](#core-goals)
- [Filters, Scorers, and Scrapers](#filters-scorers-and-scrapers)
- [Filters, Scorers, and the Data Layer](#filters-scorers-and-the-data-layer)
- [Core Design Principles](#core-design-principles)
- [Routing Flow](#routing-flow)
- [Configuration](#configuration)
- [`Plugins` Configuration](#plugins-configuration)
- [`SchedulingProfiles` Configuration](#schedulingprofiles-configuration)
- [Available plugins](#available-plugins)
- [Metric Scraping](#metric-scraping)
- [Metric Scraping and the Data Layer](#metric-scraping-and-the-data-layer)
- [Disaggregated Encode/Prefill/Decode (E/P/D)](#disaggregated-encodeprefilldecode-epd)
- [InferencePool & InferenceModel Design](#inferencepool--inferencemodel-design)
- [Current Assumptions](#current-assumptions)
Expand All @@ -31,7 +31,7 @@ The design enables:
**model metadata**
- Disaggregated **Prefill/Decode (P/D)** execution
- We have introduced experimental **Encode/Prefill/Decode (E/P/D and all its permutations)** execution. For a detailed explanation, see [Disaggregated Inference Serving](./disaggregation.md)
- Pluggable **filters**, **scorers**, and **scrapers** for extensible scheduling
- Pluggable **filters** and **scorers**, backed by a pluggable **data layer**, for extensible scheduling

---

Expand All @@ -42,12 +42,12 @@ The design enables:
- KV cache reuse
- Load balancing
- Support multi-model deployments on heterogeneous hardware
- Enable runtime extensibility with pluggable logic (filters, scorers, scrapers)
- Enable runtime extensibility with pluggable logic (filters, scorers, data layer)
- Community-aligned implementation using GIE and Envoy + External Processing (EPP)

---

## Filters, Scorers, and Scrapers
## Filters, Scorers, and the Data Layer

### Core Design Principles

Expand All @@ -58,14 +58,16 @@ The design enables:

### Routing Flow

See the upstream [Request Scheduler](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/scheduling.md) doc for the canonical scheduling model.

1. **Filtering**
- Pods in an `InferencePool` go through a sequential chain of filters
- Pods may be excluded based on criteria like model compatibility, resource usage, or custom logic

2. **Scoring**
- Filtered pods are scored using a weighted set of scorers
- Scorers currently run sequentially (future: parallel execution)
- Scorers access a shared datastore populated by scrapers
- Scorers access a shared datastore populated by the data layer

3. **Pod Selection**
- The highest-scored pod is selected
Expand All @@ -77,6 +79,8 @@ The design enables:

The llm-d Endpoint Picker relies on a YAML-based configuration—provided either as a file or an in-line parameter—to determine which lifecycle hooks (plugins) are active.

See the upstream [Configuration](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/configuration.md) doc for the canonical schema.

Specifically, this configuration establishes the following components:

- `Plugins`: The specific plugins to instantiate, along with their parameters. Because each instantiated plugin is assigned a unique name, you can configure the same plugin type multiple times if necessary.
Expand Down Expand Up @@ -210,12 +214,16 @@ To learn more about the available plugins, check the plugins [README.md](../pkg/

---

## Metric Scraping
## Metric Scraping and the Data Layer

The data layer follows a Source -> Extract -> Attribute lifecycle:

- Scrapers collect metrics (e.g., memory usage, active adapters)
- Data is injected into the shared datastore for scorers
- Data sources collect metrics (e.g., memory usage, active adapters) from pods

@elevran elevran Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nits:
From an architecture perspective, the data sources periodicallt collect different kinds of data, not only metrics.
For example a models data source periodically lists /v1/models to determine the model/LoRA hosted by the engine.
The data layer defines several different types of source, not just "scarpers". There is an endpoint change notification source, a generic k8s change notification source, an endpoint discovery source, ...
Not sure how detailed this doc needs to be.

- Extractors populate per-endpoint attributes in the shared datastore for scorers
- Scoring can rely on numerical metrics or metadata (model ID, adapter tags)

See the upstream [Data Layer](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/datalayer.md) doc for the canonical model.

---

## Disaggregated Encode/Prefill/Decode (E/P/D)
Expand Down Expand Up @@ -307,3 +315,14 @@ Enable chunked decode via the pd-sidecar flag:
- [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension)
- [Envoy External Processing](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter)
- [EPP Container Sizing Guide](./operations.md)

### Canonical llm-d architecture (upstream)

- [Router overview](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/README.md)
- [Proxy](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/proxy.md)
- [Endpoint Picker (EPP)](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/README.md)
- [Configuration](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/configuration.md)
- [Request Scheduler](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/scheduling.md)
- [Request Handler](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/request-handling.md)
- [Flow Control](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/flow-control.md)
- [Data Layer](https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/datalayer.md)
Loading