Plugins for disaggregated inference scheduling: a profile handler that selects the active stages: EPD (no disaggregation), P/D (Prefill/Decode), E/P/D (Encode/Prefill/Decode), or E/PD (Encode/Prefill-Decode), legacy headers handlers (deprecated) kept for backward compatibility, and decider plugins that control whether each disaggregation stage runs per request.
PrefixBasedPDDecider additionally drives the EPP director's conditional-decode 412 gate (RFC 7240 Prefer: if-available) via the ConditionalDecodeDecider extension point — see its section for details.
Type: disagg-profile-handler
Interfaces: scheduling.ProfileHandler
Orchestrates up to three scheduling stages per request — decode (always), and optionally encode and prefill — based on which decider plugins are configured.
Runs each scheduling stage in sequence and assembles the final result from all stages that ran.
- Run the decode profile (always).
- If an encode decider is configured and approves the request, run the encode profile.
- If a prefill decider is configured and approves the request, run the prefill profile.
- Return the assembled scheduling result with decode as the primary profile.
The handler is invoked repeatedly by the framework until all stages are complete. Each optional stage is gated by a decider: if the decider returns false for a request, the stage is marked as skipped so the handler doesn't revisit it on the next invocation. If the decode stage finds no suitable endpoint, all remaining stages are skipped and the request fails.
PrefixCacheMatchInfo— endpoint attribute fromapprox-prefix-cache-producer, read by the configured prefill decider (e.g.prefix-based-pd-decider) when deciding whether to run the prefill stage.
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
profiles.decode |
string |
No | "decode" |
Name of the decode scheduling profile. |
profiles.prefill |
string |
No | "prefill" |
Name of the prefill scheduling profile. |
profiles.encode |
string |
No | "encode" |
Name of the encode scheduling profile. |
deciders.prefill |
string |
No | — | Name of the prefill decider plugin. When set, enables P/D disaggregation. |
deciders.encode |
string |
No | — | Name of the encode decider plugin. When set, enables E disaggregation. |
Decode-only (no disaggregation):
plugins:
- type: disagg-profile-handlerP/D disaggregation:
plugins:
- type: disagg-profile-handler
parameters:
deciders:
prefill: prefix-based-pd-deciderE/P/D disaggregation:
plugins:
- type: disagg-profile-handler
parameters:
deciders:
prefill: prefix-based-pd-decider
encode: always-disagg-multimodal-decider- Without a configured decider, the corresponding stage is disabled for all requests — this is a static decision at startup, not per-request.
- The names in
deciders.prefillanddeciders.encodemust match plugin names declared earlier in the same configuration. - When using P/D disaggregation, a
PrefixCachePluginmust be configured in the prefill and decode scheduling profiles.
Type: pd-profile-handler
Interfaces: scheduling.ProfileHandler
Deprecated: Use
disagg-profile-handlerinstead.
Type: disagg-headers-handler
Interfaces: requestcontrol.PreRequest
Deprecated: Use
disagg-profile-handlerinstead.
disagg-profile-handlernow implementsrequestcontrol.PreRequestnatively.Planned removal:
v0.11.
Sets HTTP routing headers on the outgoing request so the inference proxy can forward prefill and encode work to the selected disaggregated pods.
Reads the scheduling result and writes pod addresses as request headers for each disaggregated stage that ran.
- If a prefill endpoint was selected, write its
ip:porttox-prefiller-host-port. - If one or more encode endpoints were selected, write their comma-separated
ip:portlist tox-encoder-hosts-ports. - If a stage did not run or found no endpoints, that header is omitted.
SchedulingResult.ProfileResults— per-profile endpoint selections produced bydisagg-profile-handler.
x-prefiller-host-portrequest header —<ip:port>of the selected prefill pod; absent when P/D disaggregation was skipped.x-encoder-hosts-portsrequest header — comma-separated<ip:port>list of selected encode pods; absent when encode disaggregation was skipped.
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
prefillProfile |
string |
No | "prefill" |
Name of the profile used for prefill scheduling. Only needed if the prefill profile is not named prefill. |
encodeProfile |
string |
No | "encode" |
Name of the profile used for encode scheduling. Only needed if the encode profile is not named encode. |
plugins:
- type: disagg-headers-handlerCustom profile names:
plugins:
- type: disagg-headers-handler
parameters:
prefillProfile: "my-prefill"
encodeProfile: "my-encode"Type: prefill-header-handler
Interfaces: requestcontrol.PreRequest
Deprecated: Use
disagg-profile-handlerinstead.Planned removal:
v0.11.
Type: prefix-based-pd-decider
Drives two decisions per request, both based on how much of the prompt is already cached on the selected decode pod:
- P/D disaggregation — whether to offload prefill to a remote prefill pod (consumed by
disagg-profile-handlerviadeciders.prefill). - Conditional-decode 412 gate — whether a request carrying RFC 7240
Prefer: if-availableshould be rejected with HTTP 412 instead of forwarded (consumed by the EPP director through theConditionalDecodeDeciderextension point).
Both decisions share the same nonCachedTokens threshold and the same uncached-suffix computation; they differ only in their failure semantics (see How It Works).
Compares the uncached portion of the request prompt against a configurable threshold:
- Read the prompt token count as
len(request.Body.TokenizedPrompt.TokenIDs). - Read
PrefixCacheMatchInfofrom the decode endpoint attributes. - Compute uncached suffix length.
- Disaggregation: return true (disaggregate) if uncached tokens ≥
nonCachedTokens. - Conditional-decode gate: return true (reject with 412) if uncached tokens ≥
nonCachedTokens.
The prompt token count is len(request.Body.TokenizedPrompt.TokenIDs), populated by a token-producer — auto-created with the tokenizer-free estimate backend when none is configured. Prefix cache state is read from the PrefixCacheMatchInfo attribute on the decode endpoint, populated by approx-prefix-cache-producer.
Setting nonCachedTokens: 0 disables both decisions entirely (disaggregation never runs, the conditional-decode gate always forwards).
Failure semantics when prefix-cache state is unreadable (attribute missing, malformed, or producer not configured) differ between the two decisions:
| Decision | Behavior on unreadable prefix info | Rationale |
|---|---|---|
| Disaggregation | fail open — return false (no disaggregation) | A misconfigured prefix-cache producer should not silently route every request to remote prefill. |
| Conditional-decode 412 gate | fail closed — return true (reject with 412) | The header is a "fast-fail" hint from the coordinator: if the EPP cannot prove the cache covers the prompt, the safe answer is to bounce so the coordinator falls back to the full pipeline. |
PrefixCacheMatchInfo— endpoint attribute fromapprox-prefix-cache-producer, read from the decode endpoint.request.Body.TokenizedPrompt.TokenIDs— token IDs from atoken-producerplugin; their count is the prompt token count.
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
nonCachedTokens |
int |
No | 0 |
Uncached token threshold above which P/D disaggregation is triggered, and at or above which the conditional-decode 412 gate rejects. 0 disables both behaviors. |
plugins:
- type: prefix-based-pd-decider
parameters:
nonCachedTokens: 512
- type: disagg-profile-handler
parameters:
deciders:
prefill: prefix-based-pd-deciderplugins:
- type: prefix-based-pd-decider
parameters:
nonCachedTokens: 512
# No disagg-profile-handler / no `prefill` profile.
# The plugin is auto-registered as the ConditionalDecodeDecider via
# AddPlugins; the director consults it whenever a request carries the
# RFC 7240 `Prefer: if-available` header.When the plugin is not declared in the EPP config, the conditional-decode gate is disabled and the director forwards every Prefer: if-available request unconditionally.
nonCachedTokens: 0disables both the disaggregation decision and the conditional-decode gate (the decider returns false for disaggregation and false for "should reject").- A
token-producerpopulatesTokenizedPrompt; when none is configured the framework auto-creates one with theestimatebackend, so both behaviors work without extra setup. - Requires
PrefixCacheMatchInfoon the decode endpoint. If absent: disaggregation is skipped with an error log (fail open); the conditional-decode gate rejects with 412 (fail closed). - Only one
ConditionalDecodeDecideris consulted per director. If multiple plugins implement the interface, the first one registered throughConfig.AddPluginswins; later instances are logged and ignored.
Type: always-disagg-pd-decider
Unconditionally approves P/D disaggregation for every request, regardless of cache state or prompt length.
Returns true for every request. Useful for testing or environments where P/D disaggregation should always run.
None — ignores request content and endpoint state.
None.
Type: always-disagg-multimodal-decider
Approves encode disaggregation for requests that contain multimodal content (images, audio, video); passes text-only requests through without disaggregation.
Inspects the chat completions message content blocks for image_url, video_url, or input_audio types and returns true when any such block is found.
- Request body (
ChatCompletions.Messages) — inspected for multimodal content blocks.
None.