Skip to content

Commit 541981e

Browse files
committed
docs: document conditional-decode 412 gate behavior change
Signed-off-by: Dmitri Pikus <DPIKUS@il.ibm.com>
1 parent 7bd1128 commit 541981e

1 file changed

Lines changed: 14 additions & 0 deletions

File tree

docs/disaggregation.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -413,6 +413,20 @@ The `prefix-based-pd-decider` plugin makes the disaggregation decision according
413413
- `nonCachedTokens`: Number of non-cached tokens that trigger disaggregation
414414
- If set to 0, disaggregation never occurs for any request
415415

416+
**Conditional-decode 412 gate**
417+
418+
When this plugin is configured, it also drives the EPP's RFC 7240 `Prefer: if-available` (conditional-decode) gate. The coordinator uses that header to mark a speculative early-decode attempt: forward to a decode worker only if its KV cache already covers the prompt, otherwise return HTTP 412 Precondition Failed so the coordinator restarts the pipeline at encode/prefill/decode.
419+
420+
The gate uses the same `nonCachedTokens` threshold as the disaggregation decision:
421+
422+
- Non-cached suffix `< nonCachedTokens` → forward to the chosen decode worker.
423+
- Non-cached suffix `≥ nonCachedTokens` → return 412.
424+
- `nonCachedTokens: 0` → gate disabled (forwards every conditional-decode request).
425+
- Plugin not declared in the config → gate disabled (the EPP forwards conditional-decode requests unconditionally).
426+
- Prefix-cache state unreadable on the chosen endpoint (e.g. no approximate-prefix producer wired up) → fail closed (412).
427+
428+
This behavior differs from earlier releases, where the gate was a hard-coded check inside the director that forwarded on any cache hit and rejected only on a complete cache miss. The threshold is now in tokens (not blocks), is operator-configurable, and is opt-in via the plugin.
429+
416430
#### Always-Disagg PD Decider
417431
The `always-disagg-pd-decider` is a simpler alternative used mainly for testing or benchmarking.
418432
It always triggers disaggregation, regardless of prefix cache state or prompt characteristics.

0 commit comments

Comments
 (0)