Skip to content

Commit bb6e51b

Browse files
nicole-lihuinicole-lihui
committed
docs: add experimental shared scheduling state guide (RFC llm-d#1593)
Documents the stateful/stateless EPP mode introduced across the P0-P2 commits: architecture, degradation strategy, the full --epp-mode/ --state-access-mode-* flag reference, and a local kind walkthrough (including the EndpointPickerConfig and Deployment changes needed to actually exercise the remote path, not just flip the flag). Marked experimental and cross-linked from RFC llm-d#1593 so reviewers evaluating the spike have one place to start instead of reverse-engineering it from code comments. Signed-off-by: nicole-lihui <nicole.li@daocloud.io>
1 parent 28f626d commit bb6e51b

1 file changed

Lines changed: 239 additions & 0 deletions

File tree

‎docs/shared-scheduling-state.md‎

Lines changed: 239 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,239 @@
1+
# Shared Scheduling State for Horizontally Scalable EPP (Experimental)
2+
3+
> [!WARNING]
4+
> Feasibility spike for RFC #1593, not a finished feature. Default behavior (`--epp-mode`
5+
> unset) is unchanged. See [Known Limitations](#known-limitations) before relying on this
6+
> beyond local evaluation.
7+
8+
## Overview
9+
10+
Today's EPP deployment is a single active-passive replica set: leader election picks one
11+
Pod to serve `ext-proc` traffic, and every scheduling signal (inflight load, prefix cache,
12+
flow control) lives only in that leader's memory. A leader restart is a real traffic
13+
outage, and throughput is capped at what one process can do.
14+
15+
This introduces `--epp-mode=stateful|stateless`, separating **who serves traffic** from
16+
**who holds shared scheduling state**: N `stateless` replicas serve traffic concurrently
17+
(no leader election), each falling back to a local, classic-equivalent view; one
18+
`stateful` replica holds the fleet-wide view over an internal gRPC State API. See RFC #1593
19+
for the full design discussion.
20+
21+
---
22+
23+
## Key Components
24+
25+
| Component | Role |
26+
|---|---|
27+
| **Stateless EPP** | Serves `ext-proc` traffic, no leader election. Reads/writes shared state through a `StateStore`. |
28+
| **Stateful EPP** | Holds the fleet-wide view. Never serves `ext-proc` traffic; only exposes the State API and health. Single active-passive instance, no sharding path. |
29+
| **State API** (`pkg/epp/statestore/stateapi`) | The gRPC service the stateful EPP exposes, backed by an in-memory `Store` (no eviction). |
30+
| **StateStore** (`pkg/epp/statestore`) | Client-side abstraction: `Local` (in-process, classic-equivalent), `Remote` (forwards to the State API), `FailOpen`/`LocalFallback` (prefers Remote within a timeout, falls back to Local). |
31+
| **ConcurrencyState** | New primitive (`concurrency_lease.go`) for a fleet-wide concurrency cap — the existing `FlowRegistry` counts queue occupancy, not concurrent execution, so it can't serve this role. |
32+
33+
---
34+
35+
## Architecture
36+
37+
```mermaid
38+
flowchart LR
39+
subgraph SL["N stateless EPP replicas"]
40+
DR["Director"] --> AD["FlowControlAdmissionController"]
41+
DR --> DP["DataProducer DAG"]
42+
DP --> INF["InFlightLoadProducer"]
43+
DP --> APX["approximateprefix"]
44+
end
45+
subgraph SF["1 stateful EPP replica"]
46+
SRV["stateapi.Server (gRPC :9004)"]
47+
end
48+
INF -. "batched read + async write" .-> SRV
49+
APX -. "batched read + async write" .-> SRV
50+
AD -. "concurrency lease, after local admit" .-> SRV
51+
```
52+
53+
Reads stay on the request's critical path (bounded by `--state-api-remote-timeout`,
54+
default 50ms) since a scorer consumes the result immediately. Writes are fire-and-forget:
55+
the local mutation the request depends on already happened synchronously, so a remote
56+
push only affects future requests and never blocks the current one.
57+
58+
Two degradation strategies, chosen per capability: **FailOpen** (inflight, prefix) prefers
59+
the remote read and falls back to local on error/timeout. **LocalFallback** (flow control)
60+
always admits locally first, then requests a fleet-wide lease; on remote failure it falls
61+
back to a local concurrency cap (`--flow-control-local-max-concurrency`) and reports
62+
`Degraded` rather than silently `Admitted`.
63+
64+
---
65+
66+
## Configuration
67+
68+
| Flag | Default | Description |
69+
|---|---|---|
70+
| `--epp-mode` | `classic` | `classic`, `stateful`, or `stateless`. |
71+
| `--state-api-port` | `9004` | State API gRPC port. Stateful mode only. |
72+
| `--stateful-epp-address` | — | `host:port` of the stateful replica. Required in stateless mode. |
73+
| `--state-api-remote-timeout` | `50ms` | Per-call timeout for State API reads/writes. |
74+
| `--flow-control-global-max-concurrency` | `0` (unlimited) | Fleet-wide concurrency cap per flow key. |
75+
| `--flow-control-local-max-concurrency` | `0` (unlimited) | Local fallback cap when the State API is unreachable. |
76+
| `--state-access-mode-inflight` / `-prefix` | `FailOpen` | Or `Local` — never call remote for that capability. |
77+
| `--state-access-mode-flowcontrol` | `LocalFallback` | Or `Local`. |
78+
| `--state-api-artificial-delay` | `0` | Fixed per-call delay, for modeling a real network hop on loopback test clusters. Not a production flag. |
79+
80+
```bash
81+
epp --epp-mode=stateful --state-api-port=9004 --config-file=/etc/epp/epp-config.yaml
82+
83+
epp --epp-mode=stateless --stateful-epp-address=stateful-epp:9004 \
84+
--config-file=/etc/epp/epp-config.yaml
85+
```
86+
87+
---
88+
89+
## Try It Locally
90+
91+
Not yet wired into `kind-dev-env.sh`'s scenario flags (unlike `DISAGG_E`/`DISAGG_P`). To
92+
try it manually:
93+
94+
1. Add `inflight-load-producer` to your `EndpointPickerConfig`'s plugin list and set
95+
`featureGates: [flowControl]`. Without both, `--epp-mode` alone won't exercise the
96+
inflight or flow-control capabilities — only prefix works with the default
97+
`deploy/config/sim-epp-config.yaml`. This config becomes the `epp-config` ConfigMap
98+
both the stateless and stateful replicas mount at `/etc/epp/epp-config.yaml`, so one
99+
copy covers both:
100+
101+
```yaml
102+
# sim-epp-config-stateful.yaml
103+
apiVersion: llm-d.ai/v1alpha1
104+
kind: EndpointPickerConfig
105+
featureGates: [flowControl]
106+
plugins:
107+
- type: approx-prefix-cache-producer
108+
parameters:
109+
maxPrefixBlocksToMatch: 256
110+
lruCapacityPerServer: 31250
111+
- type: inflight-load-producer
112+
- type: prefix-cache-scorer
113+
- type: decode-filter
114+
- type: max-score-picker
115+
- type: single-profile-handler
116+
schedulingProfiles:
117+
- name: default
118+
plugins:
119+
- pluginRef: decode-filter
120+
- pluginRef: max-score-picker
121+
- pluginRef: prefix-cache-scorer
122+
weight: 2
123+
```
124+
125+
2. `EPP_CONFIG=sim-epp-config-stateful.yaml ./scripts/kind-dev-env.sh` for the classic
126+
baseline — this is what creates the `epp-config` ConfigMap the next step reuses.
127+
3. Deploy a second EPP Deployment in stateful mode, reusing the image, `ServiceAccount`,
128+
and `epp-config` ConfigMap the step above already created (no new RBAC needed). Save
129+
this as `stateful-epp.yaml` and `envsubst < stateful-epp.yaml | kubectl apply -f -`
130+
(`$EPP_IMAGE`, `$NAMESPACE`, `$POOL_NAME`, `$EPP_NAME` are already exported by
131+
`kind-dev-env.sh`):
132+
133+
```yaml
134+
apiVersion: apps/v1
135+
kind: Deployment
136+
metadata:
137+
name: stateful-epp
138+
namespace: ${NAMESPACE}
139+
spec:
140+
replicas: 1
141+
# A single leader-elected replica: RollingUpdate would surge a second pod
142+
# while the old one is still up, and the new pod can never pass its
143+
# leader-gated readiness check until the old one is killed, deadlocking
144+
# the rollout.
145+
strategy:
146+
type: Recreate
147+
selector:
148+
matchLabels:
149+
app: stateful-epp
150+
template:
151+
metadata:
152+
labels:
153+
app: stateful-epp
154+
spec:
155+
serviceAccountName: ${EPP_NAME}
156+
containers:
157+
- name: epp
158+
image: ${EPP_IMAGE}
159+
args:
160+
- --pool-name
161+
- "${POOL_NAME}"
162+
- --pool-namespace
163+
- "${NAMESPACE}"
164+
- --config-file
165+
- "/etc/epp/epp-config.yaml"
166+
- --epp-mode=stateful
167+
- --state-api-port=9004
168+
ports:
169+
- containerPort: 9003
170+
name: grpc-health
171+
- containerPort: 9004
172+
name: state-api
173+
# service must be set: an empty/overall-health service name is
174+
# gated on isLeader and deadlocks a rolling update, since the new
175+
# pod never turns Ready while the old leader is still up.
176+
livenessProbe:
177+
grpc:
178+
port: 9003
179+
service: liveness
180+
readinessProbe:
181+
grpc:
182+
port: 9003
183+
service: readiness
184+
volumeMounts:
185+
- name: epp-config
186+
mountPath: /etc/epp
187+
volumes:
188+
- name: epp-config
189+
configMap:
190+
name: epp-config
191+
---
192+
apiVersion: v1
193+
kind: Service
194+
metadata:
195+
name: stateful-epp
196+
namespace: ${NAMESPACE}
197+
spec:
198+
selector:
199+
app: stateful-epp
200+
ports:
201+
- name: state-api
202+
port: 9004
203+
- name: grpc-health
204+
port: 9003
205+
```
206+
207+
4. Patch the existing Deployment's args to add `--epp-mode=stateless
208+
--stateful-epp-address=stateful-epp:9004`, then scale it to N replicas:
209+
210+
```bash
211+
kubectl patch deployment "${EPP_NAME}" --type=json -p='[
212+
{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--epp-mode=stateless"},
213+
{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--stateful-epp-address=stateful-epp:9004"}
214+
]'
215+
kubectl scale deployment "${EPP_NAME}" --replicas=3
216+
```
217+
218+
5. Before trusting any comparison, confirm `statestore_remote_call_total` is nonzero on
219+
`:9090/metrics` — an absent metric means the remote path was never exercised, not that
220+
it never failed (`statestore_remote_fallback_total` gives the failure rate).
221+
222+
---
223+
224+
## Known Limitations
225+
226+
- Single active-passive stateful replica, no sharding path.
227+
- In-memory prefix/inflight stores have no eviction — memory grows unbounded. A Redis- or
228+
PVC-backed `Store` is future work; `stateapi.Store` is already factored for a drop-in
229+
replacement.
230+
- Internal State API is plaintext gRPC (no TLS/mTLS).
231+
- The stateful replica's own crash-loop behavior hasn't been fault-injected (only clean
232+
pod delete-and-replace).
233+
234+
---
235+
236+
## References
237+
238+
- RFC #1593, "Shared Scheduling State for Horizontally Scalable EPP Deployments"
239+
- [Architecture Documentation](./architecture.md)

0 commit comments

Comments
 (0)