diff --git a/pkg/epp/framework/plugins/scheduling/scorer/prefix/README.md b/pkg/epp/framework/plugins/scheduling/scorer/prefix/README.md index 8e8cfb16c2..df2ccf9aaf 100644 --- a/pkg/epp/framework/plugins/scheduling/scorer/prefix/README.md +++ b/pkg/epp/framework/plugins/scheduling/scorer/prefix/README.md @@ -31,6 +31,23 @@ The attribute is typically produced by the approximate prefix cache data produce This plugin does not define any plugin-specific parameters. +### Agentic Workload Example + +For agentic workloads, sessions typically consist of long-running, multi-turn interactions where the system prompt is shared but session-specific context diverges rapidly. + +To optimize routing for these workloads, configure both the `prefix-cache-scorer` (with length awareness enabled) and the `session-affinity-scorer`. Length awareness allows shorter sessions to migrate freely when pods are overloaded, while session affinity keeps the rapidly diverging chat history sticky to the original pod: + +```yaml +- type: prefix-cache-scorer + parameters: + matchLengthWeight: 0.8 + matchLengthScaleTokens: 16000 + +- type: session-affinity-scorer + parameters: + headerName: x-session-token +``` + ## Operational notes - The scorer itself does not hash prompts or maintain cache state. diff --git a/pkg/epp/framework/plugins/scheduling/scorer/sessionaffinity/README.md b/pkg/epp/framework/plugins/scheduling/scorer/sessionaffinity/README.md index df67edc6c0..3ca3ed1c18 100644 --- a/pkg/epp/framework/plugins/scheduling/scorer/sessionaffinity/README.md +++ b/pkg/epp/framework/plugins/scheduling/scorer/sessionaffinity/README.md @@ -42,6 +42,21 @@ To support session affinity with PD disaggregation, configure two separate insta The decode instance uses the default behavior (writing the decode pod to `x-session-token`). The prefill instance uses `profileName: prefill` to look up the prefill pod from the scheduling results and write it to `x-session-token-prefill`. This ensures that subsequent requests in the same session target both the same prefill pod and the same decode pod. +### Agentic Workload Example + +For agentic workloads (long-running, multi-turn interactions), it is highly recommended to pair this scorer with the `prefix-cache-scorer` (with length awareness enabled) to ensure rapidly diverging chat history remains sticky to the original pod, while allowing shorter new sessions to migrate to less loaded pods: + +```yaml +- type: prefix-cache-scorer + parameters: + matchLengthWeight: 0.8 + matchLengthScaleTokens: 16000 + +- type: session-affinity-scorer + parameters: + headerName: x-session-token +``` + ## Relationship to the session affinity filter The [session affinity filter](../../filter/sessionaffinity/README.md) (`session-affinity-filter`) provides the same affinity behavior as a hard constraint and writes the same response header. Configuring both alongside the scorer is unnecessary and can be misleading; see [Relationship to the session affinity scorer](../../filter/sessionaffinity/README.md#relationship-to-the-session-affinity-scorer) for details. Use the scorer for a soft preference that can be outweighed by other scorers, or the filter for a hard pin.