feat(epp): add encoder cache features to the latency predictor#2125
Open
kaushikmitr wants to merge 2 commits into
Open
feat(epp): add encoder cache features to the latency predictor#2125kaushikmitr wants to merge 2 commits into
kaushikmitr wants to merge 2 commits into
Conversation
github-actions
Bot
added
kind/feature
kaushikmitr
force-pushed
the
kaushikmitra/latency-predictor-encoder-cache-feature
branch
from
July 21, 2026 21:03
0fa4e83 to
735d47e
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds optional multimodal encoder-cache size features into the PredictedLatency data producer so TTFT predictions and training can account for per-pod encoder cache locality (via EncoderCacheMatchInfo), improving routing quality on multimodal workloads without changing default behavior.
Changes:
- Plumbs
encoder_input_sizeandencoder_matched_sizethrough prediction and training request payloads (including Bayesian Ridge coefficient support + validation). - Adds an opt-in plugin config (
useEncoderCacheFeatures,encoderCacheMatchInfoProducerName) that consumes encoder-cache match data as a required DAG dependency and captures per-endpoint sizes. - Extends tests to cover propagation and validation of the new encoder-cache fields.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/epp/framework/plugins/requestcontrol/dataproducer/predictedlatency/training.go | Extends bulk prediction/training request construction to include encoder-cache size features. |
| pkg/epp/framework/plugins/requestcontrol/dataproducer/predictedlatency/training_test.go | Adds/updates unit tests to verify encoder size propagation into outgoing prediction/training requests. |
| pkg/epp/framework/plugins/requestcontrol/dataproducer/predictedlatency/requestcontrol_hooks.go | Ensures TTFT training collection passes encoder-cache matched size for the serving/prefill endpoint. |
| pkg/epp/framework/plugins/requestcontrol/dataproducer/predictedlatency/README.md | Documents the new opt-in encoder-cache feature flags and data source. |
| pkg/epp/framework/plugins/requestcontrol/dataproducer/predictedlatency/prediction.go | Feeds per-endpoint encoder-cache sizes into bulk prediction requests. |
| pkg/epp/framework/plugins/requestcontrol/dataproducer/predictedlatency/plugin.go | Adds config knobs and request-scoped context fields for encoder-cache feature plumbing. |
| pkg/epp/framework/plugins/requestcontrol/dataproducer/predictedlatency/latencypredictorclient/types.go | Adds encoder-cache fields to JSON request/entry schemas. |
| pkg/epp/framework/plugins/requestcontrol/dataproducer/predictedlatency/latencypredictorclient/training.go | Validates encoder-cache fields for training entries. |
| pkg/epp/framework/plugins/requestcontrol/dataproducer/predictedlatency/latencypredictorclient/prediction.go | Validates encoder-cache fields for prediction requests and applies coefficients in Bayesian Ridge TTFT. |
| pkg/epp/framework/plugins/requestcontrol/dataproducer/predictedlatency/latencypredictorclient/prediction_test.go | Adds unit tests for encoder-cache validation and Bayesian Ridge coefficient application. |
| pkg/epp/framework/plugins/requestcontrol/dataproducer/predictedlatency/dataproducer_hooks.go | Captures encoder-cache input/matched sizes from endpoint attributes and adds required dependency when enabled. |
| pkg/epp/framework/plugins/requestcontrol/dataproducer/predictedlatency/dataproducer_hooks_test.go | Adds tests covering consumption gating and size capture behavior for enabled/disabled modes. |
Comment on lines
+137
to
+141
| inputSize := sumItemSizes(matchInfo.RequestItems()) | ||
| matchedSize := sumItemSizes(matchInfo.MatchedItems()) | ||
| predictedLatencyCtx.encoderInputSize = inputSize | ||
| predictedLatencyCtx.encoderMatchedSizeForEndpoints[endpoint.GetMetadata().NamespacedName.Name] = matchedSize | ||
| logger.V(logutil.DEBUG).Info("Encoder cache sizes for pod", |
Comment on lines
+226
to
+231
| if i < len(encoderInputSizes) { | ||
| bulkRequests[i].EncoderInputSize = encoderInputSizes[i] | ||
| } | ||
| if i < len(encoderMatchedSizes) { | ||
| bulkRequests[i].EncoderMatchedSize = encoderMatchedSizes[i] | ||
| } |
Collaborator
|
Did we do any evaluation? |
Contributor
Author
|
Multimodal requests incur encoder compute that dominates TTFT when the target pod's encoder cache misses, and the predictor cannot see this per-pod difference today. Feed the multimodal encoder-cache match data as two features, encoder_input_size and encoder_matched_size, opt-in via useEncoderCacheFeatures so existing deployments are unaffected. Signed-off-by: Kaushik Mitra <kaushikmitra@google.com>
The predictor rejects matched > input, so inconsistent match data or mismatched feature slices would fail validation and drop the whole prediction or training entry instead of degrading gracefully. Signed-off-by: Kaushik Mitra <kaushikmitra@google.com>
kaushikmitr
force-pushed
the
kaushikmitra/latency-predictor-encoder-cache-feature
branch
from
July 23, 2026 22:40
7dc0fa9 to
4d605c3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind feature
What this PR does / why we need it:
Adds two multimodal encoder-cache features to the latency predictor: encoder_input_size (total size of the request's unique multimodal items) and encoder_matched_size (portion likely present in the target pod's encoder cache, from the multimodal producer's per-pod match data).
For multimodal requests, encoder compute dominates TTFT when the target pod's encoder cache misses, and this cost differs per pod. The predictor cannot see this today, so TTFT predictions carry large unexplained variance on multimodal workloads, degrading SLO-aware routing and admission. input - matched approximates the encoder work a pod would actually perform.
The features are opt-in via a new useEncoderCacheFeatures plugin parameter (default false; no behavior change for existing deployments). When enabled, the plugin consumes EncoderCacheMatchInfo as a required dependency so the DAG orders the multimodal producer first, auto-creating it if absent. encoderCacheMatchInfoProducerName selects a specific producer instance. Sizes flow into bulk predictions per endpoint and into TTFT training entries for the serving pod (prefill pod in disaggregated mode); TPOT paths send zeros, following the prefix_cache_score convention. The Bayesian Ridge path applies matching TTFT coefficients when present; models trained without them are unaffected.
Note: the Python predictor server needs matching schema support (encoder_input_size, encoder_matched_size, both non-negative ints, matched <= input) before tree models learn from these fields; until then they are sent and ignored.
Which issue(s) this PR fixes:
Fixes #
Release note:
The latency predictor supports optional multimodal encoder-cache features (encoder_input_size, encoder_matched_size), enabled with the useEncoderCacheFeatures parameter, to improve TTFT predictions for multimodal requests.