You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pkg/epp/framework/plugins/scheduling/profilehandler/disagg/README.md
+41-9Lines changed: 41 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,6 +2,8 @@
2
2
3
3
Plugins for disaggregated inference scheduling: a profile handler that selects the active stages: EPD (no disaggregation), P/D (Prefill/Decode), E/P/D (Encode/Prefill/Decode), or E/PD (Encode/Prefill-Decode), legacy headers handlers (deprecated) kept for backward compatibility, and decider plugins that control whether each disaggregation stage runs per request.
4
4
5
+
`PrefixBasedPDDecider` additionally drives the EPP director's conditional-decode 412 gate (RFC 7240 `Prefer: if-available`) via the `ConditionalDecodeDecider` extension point — see its section for details.
6
+
5
7
## Contents
6
8
7
9
-[Profile Handlers](#profile-handlers)
@@ -170,20 +172,35 @@ plugins:
170
172
171
173
**Type:** `prefix-based-pd-decider`
172
174
173
-
Decides per-request whether P/D disaggregation should run, based on how much of the prompt is already cached on the selected decode pod.
175
+
Drives two decisions per request, both based on how much of the prompt is already cached on the selected decode pod:
176
+
177
+
1. **P/D disaggregation** — whether to offload prefill to a remote prefill pod (consumed by `disagg-profile-handler` via `deciders.prefill`).
178
+
2. **Conditional-decode 412 gate** — whether a request carrying RFC 7240 `Prefer: if-available` should be rejected with HTTP 412 instead of forwarded (consumed by the EPP director through the `ConditionalDecodeDecider` extension point).
179
+
180
+
Both decisions share the same `nonCachedTokens` threshold and the same uncached-suffix computation; they differ only in their failure semantics (see [How It Works](#how-it-works)).
174
181
175
182
#### What it does
176
183
177
-
Compares the uncached portion of the request prompt against a configurable threshold, triggering P/D disaggregation only when the uncached suffix is long enough to justify the overhead.
184
+
Compares the uncached portion of the request prompt against a configurable threshold:
178
185
179
186
1. Read the prompt token count as `len(request.Body.TokenizedPrompt.TokenIDs)`.
180
187
2. Read `PrefixCacheMatchInfo` from the decode endpoint attributes.
181
188
3. Compute uncached suffix length.
182
-
4. Return true (disaggregate) if uncached tokens ≥ `nonCachedTokens`.
189
+
4. **Disaggregation:** return true (disaggregate) if uncached tokens ≥ `nonCachedTokens`.
190
+
5. **Conditional-decode gate:** return true (reject with 412) if uncached tokens ≥ `nonCachedTokens`.
183
191
184
192
#### How It Works
185
193
186
-
The prompt token count is `len(request.Body.TokenizedPrompt.TokenIDs)`, populated by a `token-producer` — auto-created with the tokenizer-free `estimate` backend when none is configured. Prefix cache state is read from the `PrefixCacheMatchInfo` attribute on the decode endpoint, populated by `approx-prefix-cache-producer`. If the attribute is absent or malformed, disaggregation is skipped. Setting `nonCachedTokens: 0` disables the decider entirely (always returns false).
194
+
The prompt token count is `len(request.Body.TokenizedPrompt.TokenIDs)`, populated by a `token-producer` — auto-created with the tokenizer-free `estimate` backend when none is configured. Prefix cache state is read from the `PrefixCacheMatchInfo` attribute on the decode endpoint, populated by `approx-prefix-cache-producer`.
195
+
196
+
Setting `nonCachedTokens: 0` disables both decisions entirely (disaggregation never runs, the conditional-decode gate always forwards).
197
+
198
+
**Failure semantics** when prefix-cache state is unreadable (attribute missing, malformed, or producer not configured) differ between the two decisions:
199
+
200
+
| Decision | Behavior on unreadable prefix info | Rationale |
201
+
|---|---|---|
202
+
| Disaggregation | fail open — return false (no disaggregation) | A misconfigured prefix-cache producer should not silently route every request to remote prefill. |
203
+
| Conditional-decode 412 gate | fail closed — return true (reject with 412) | The header is a "fast-fail" hint from the coordinator: if the EPP cannot prove the cache covers the prompt, the safe answer is to bounce so the coordinator falls back to the full pipeline. |
187
204
188
205
#### Inputs consumed
189
206
@@ -195,9 +212,9 @@ The prompt token count is `len(request.Body.TokenizedPrompt.TokenIDs)`, populate
195
212
##### Parameters
196
213
| Name | Type | Required | Default | Description |
| `nonCachedTokens` | `int` | No | `0` | Uncached token threshold above which P/D disaggregation is triggered. `0` disables the decider. |
215
+
| `nonCachedTokens` | `int` | No | `0` | Uncached token threshold above which P/D disaggregation is triggered, and at or above which the conditional-decode 412 gate rejects. `0` disables both behaviors. |
199
216
200
-
##### Example
217
+
##### Example — P/D disaggregation
201
218
```yaml
202
219
plugins:
203
220
- type: prefix-based-pd-decider
@@ -209,11 +226,26 @@ plugins:
209
226
prefill: prefix-based-pd-decider
210
227
```
211
228
229
+
##### Example — conditional-decode 412 gate only (no P/D)
230
+
```yaml
231
+
plugins:
232
+
- type: prefix-based-pd-decider
233
+
parameters:
234
+
nonCachedTokens: 512
235
+
# No disagg-profile-handler / no `prefill` profile.
236
+
# The plugin is auto-registered as the ConditionalDecodeDecider via
237
+
# AddPlugins; the director consults it whenever a request carries the
238
+
# RFC 7240 `Prefer: if-available` header.
239
+
```
240
+
241
+
When the plugin is *not* declared in the EPP config, the conditional-decode gate is disabled and the director forwards every `Prefer: if-available` request unconditionally.
- A `token-producer` populates `TokenizedPrompt`; when none is configured the framework auto-creates one with the `estimate` backend, so disaggregation works without extra setup.
216
-
- Requires `PrefixCacheMatchInfo` on the decode endpoint; if absent, disaggregation is skipped with an error log.
245
+
-`nonCachedTokens: 0` disables both the disaggregation decision and the conditional-decode gate (the decider returns false for disaggregation and false for "should reject").
246
+
- A `token-producer` populates `TokenizedPrompt`; when none is configured the framework auto-creates one with the `estimate` backend, so both behaviors work without extra setup.
247
+
- Requires `PrefixCacheMatchInfo` on the decode endpoint. If absent: disaggregation is skipped with an error log (fail open); the conditional-decode gate rejects with 412 (fail closed).
248
+
- Only one `ConditionalDecodeDecider` is consulted per director. If multiple plugins implement the interface, the first one registered through `Config.AddPlugins` wins; later instances are logged and ignored.
0 commit comments