Skip to content

Commit cd229c3

Browse files
committed
docs: more clarification about conditional-decode 412 gate behavior change
Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>
1 parent 541981e commit cd229c3

1 file changed

Lines changed: 41 additions & 9 deletions

File tree

  • pkg/epp/framework/plugins/scheduling/profilehandler/disagg

pkg/epp/framework/plugins/scheduling/profilehandler/disagg/README.md

Lines changed: 41 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
Plugins for disaggregated inference scheduling: a profile handler that selects the active stages: EPD (no disaggregation), P/D (Prefill/Decode), E/P/D (Encode/Prefill/Decode), or E/PD (Encode/Prefill-Decode), legacy headers handlers (deprecated) kept for backward compatibility, and decider plugins that control whether each disaggregation stage runs per request.
44

5+
`PrefixBasedPDDecider` additionally drives the EPP director's conditional-decode 412 gate (RFC 7240 `Prefer: if-available`) via the `ConditionalDecodeDecider` extension point — see its section for details.
6+
57
## Contents
68

79
- [Profile Handlers](#profile-handlers)
@@ -170,20 +172,35 @@ plugins:
170172

171173
**Type:** `prefix-based-pd-decider`
172174

173-
Decides per-request whether P/D disaggregation should run, based on how much of the prompt is already cached on the selected decode pod.
175+
Drives two decisions per request, both based on how much of the prompt is already cached on the selected decode pod:
176+
177+
1. **P/D disaggregation** — whether to offload prefill to a remote prefill pod (consumed by `disagg-profile-handler` via `deciders.prefill`).
178+
2. **Conditional-decode 412 gate** — whether a request carrying RFC 7240 `Prefer: if-available` should be rejected with HTTP 412 instead of forwarded (consumed by the EPP director through the `ConditionalDecodeDecider` extension point).
179+
180+
Both decisions share the same `nonCachedTokens` threshold and the same uncached-suffix computation; they differ only in their failure semantics (see [How It Works](#how-it-works)).
174181

175182
#### What it does
176183

177-
Compares the uncached portion of the request prompt against a configurable threshold, triggering P/D disaggregation only when the uncached suffix is long enough to justify the overhead.
184+
Compares the uncached portion of the request prompt against a configurable threshold:
178185

179186
1. Read the prompt token count as `len(request.Body.TokenizedPrompt.TokenIDs)`.
180187
2. Read `PrefixCacheMatchInfo` from the decode endpoint attributes.
181188
3. Compute uncached suffix length.
182-
4. Return true (disaggregate) if uncached tokens ≥ `nonCachedTokens`.
189+
4. **Disaggregation:** return true (disaggregate) if uncached tokens ≥ `nonCachedTokens`.
190+
5. **Conditional-decode gate:** return true (reject with 412) if uncached tokens ≥ `nonCachedTokens`.
183191

184192
#### How It Works
185193

186-
The prompt token count is `len(request.Body.TokenizedPrompt.TokenIDs)`, populated by a `token-producer` — auto-created with the tokenizer-free `estimate` backend when none is configured. Prefix cache state is read from the `PrefixCacheMatchInfo` attribute on the decode endpoint, populated by `approx-prefix-cache-producer`. If the attribute is absent or malformed, disaggregation is skipped. Setting `nonCachedTokens: 0` disables the decider entirely (always returns false).
194+
The prompt token count is `len(request.Body.TokenizedPrompt.TokenIDs)`, populated by a `token-producer` — auto-created with the tokenizer-free `estimate` backend when none is configured. Prefix cache state is read from the `PrefixCacheMatchInfo` attribute on the decode endpoint, populated by `approx-prefix-cache-producer`.
195+
196+
Setting `nonCachedTokens: 0` disables both decisions entirely (disaggregation never runs, the conditional-decode gate always forwards).
197+
198+
**Failure semantics** when prefix-cache state is unreadable (attribute missing, malformed, or producer not configured) differ between the two decisions:
199+
200+
| Decision | Behavior on unreadable prefix info | Rationale |
201+
|---|---|---|
202+
| Disaggregation | fail open — return false (no disaggregation) | A misconfigured prefix-cache producer should not silently route every request to remote prefill. |
203+
| Conditional-decode 412 gate | fail closed — return true (reject with 412) | The header is a "fast-fail" hint from the coordinator: if the EPP cannot prove the cache covers the prompt, the safe answer is to bounce so the coordinator falls back to the full pipeline. |
187204

188205
#### Inputs consumed
189206

@@ -195,9 +212,9 @@ The prompt token count is `len(request.Body.TokenizedPrompt.TokenIDs)`, populate
195212
##### Parameters
196213
| Name | Type | Required | Default | Description |
197214
|------|------|----------|---------|-------------|
198-
| `nonCachedTokens` | `int` | No | `0` | Uncached token threshold above which P/D disaggregation is triggered. `0` disables the decider. |
215+
| `nonCachedTokens` | `int` | No | `0` | Uncached token threshold above which P/D disaggregation is triggered, and at or above which the conditional-decode 412 gate rejects. `0` disables both behaviors. |
199216

200-
##### Example
217+
##### Example — P/D disaggregation
201218
```yaml
202219
plugins:
203220
- type: prefix-based-pd-decider
@@ -209,11 +226,26 @@ plugins:
209226
prefill: prefix-based-pd-decider
210227
```
211228

229+
##### Example — conditional-decode 412 gate only (no P/D)
230+
```yaml
231+
plugins:
232+
- type: prefix-based-pd-decider
233+
parameters:
234+
nonCachedTokens: 512
235+
# No disagg-profile-handler / no `prefill` profile.
236+
# The plugin is auto-registered as the ConditionalDecodeDecider via
237+
# AddPlugins; the director consults it whenever a request carries the
238+
# RFC 7240 `Prefer: if-available` header.
239+
```
240+
241+
When the plugin is *not* declared in the EPP config, the conditional-decode gate is disabled and the director forwards every `Prefer: if-available` request unconditionally.
242+
212243
#### Limitations
213244

214-
- `nonCachedTokens: 0` disables disaggregation entirely (the decider always returns false).
215-
- A `token-producer` populates `TokenizedPrompt`; when none is configured the framework auto-creates one with the `estimate` backend, so disaggregation works without extra setup.
216-
- Requires `PrefixCacheMatchInfo` on the decode endpoint; if absent, disaggregation is skipped with an error log.
245+
- `nonCachedTokens: 0` disables both the disaggregation decision and the conditional-decode gate (the decider returns false for disaggregation and false for "should reject").
246+
- A `token-producer` populates `TokenizedPrompt`; when none is configured the framework auto-creates one with the `estimate` backend, so both behaviors work without extra setup.
247+
- Requires `PrefixCacheMatchInfo` on the decode endpoint. If absent: disaggregation is skipped with an error log (fail open); the conditional-decode gate rejects with 412 (fail closed).
248+
- Only one `ConditionalDecodeDecider` is consulted per director. If multiple plugins implement the interface, the first one registered through `Config.AddPlugins` wins; later instances are logged and ignored.
217249

218250
---
219251

0 commit comments

Comments
 (0)