Skip to content

Commit 8f5e2e6

Browse files
committed
docs: describe conditional-decode gate independently of coordinator
Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>
1 parent 765591d commit 8f5e2e6

2 files changed

Lines changed: 5 additions & 2 deletions

File tree

docs/disaggregation.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -415,7 +415,10 @@ The `prefix-based-pd-decider` plugin makes the disaggregation decision according
415415

416416
**Conditional-decode 412 gate**
417417

418-
When this plugin is configured, it also drives the EPP's RFC 7240 `Prefer: if-available` (conditional-decode) gate. The coordinator uses that header to mark a speculative early-decode attempt: forward to a decode worker only if its KV cache already covers the prompt, otherwise return HTTP 412 Precondition Failed so the coordinator restarts the pipeline at encode/prefill/decode.
418+
When this plugin is configured, it also gates requests carrying the RFC 7240 `Prefer: if-available` header: a request is forwarded to the chosen decode worker only when the worker's KV cache already covers the prompt; otherwise the EPP returns HTTP 412 Precondition Failed.
419+
420+
> **Note**
421+
> The llm-d coordinator uses this header to mark speculative early-decode attempts. A 412 response signals that the local cache was insufficient, prompting the coordinator to restart the pipeline at encode/prefill/decode.
419422

420423
The gate uses the same `nonCachedTokens` threshold as the disaggregation decision:
421424

pkg/epp/framework/plugins/scheduling/profilehandler/disagg/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -200,7 +200,7 @@ Setting `nonCachedTokens: 0` disables both decisions entirely (disaggregation ne
200200
| Decision | Behavior on unreadable prefix info | Rationale |
201201
|---|---|---|
202202
| Disaggregation | fail open — return false (no disaggregation) | A misconfigured prefix-cache producer should not silently route every request to remote prefill. |
203-
| Conditional-decode 412 gate | fail closed — return true (reject with 412) | The header is a "fast-fail" hint from the coordinator: if the EPP cannot prove the cache covers the prompt, the safe answer is to bounce so the coordinator falls back to the full pipeline. |
203+
| Conditional-decode 412 gate | fail closed — return true (reject with 412) | The `Prefer: if-available` header is a fast-fail hint: if the EPP cannot prove the cache covers the prompt, the safe answer is to reject with 412 so the caller can fall back. |
204204

205205
#### Inputs consumed
206206

0 commit comments

Comments
 (0)