feat(plugin-state): Allow customizing staleness threshold#2138
Open
asharkhan3101 wants to merge 5 commits into
Open
feat(plugin-state): Allow customizing staleness threshold#2138asharkhan3101 wants to merge 5 commits into
asharkhan3101 wants to merge 5 commits into
Conversation
In decode heavy with non-streaming, it is fairly common that a request may take over 5 minutes (default staleness threshold), which causes plugin state to remove those from accounting, when used with inflight flow control this results in prematurly releasing request or token budget which results in EPP admitting more request than configured. This PR allows us to customize this staleness threshold, it adds a new epp command line argument `--plugin-state-staleness-threshold` defaulting to 5m. Signed-off-by: mohammadkhan <mohammadkhan@digitalocean.com>
github-actions
Bot
added
size/M
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a configurable staleness threshold for PluginState cleanup so long-running non-streaming requests aren’t prematurely evicted from in-flight accounting, which can otherwise lead to EPP over-admission under request-concurrency flow control.
Changes:
- Adds a new EPP flag
--plugin-state-staleness-threshold(default 5m) and validates it is positive. - Plumbs the configured duration into the plugin framework via a process-wide default applied before plugin instantiation.
- Updates
PluginStateto store a per-instance threshold and extends tests to cover the new behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| pkg/epp/server/options.go | Adds the new CLI option, default value, flag wiring, and validation. |
| pkg/epp/framework/interface/plugin/plugin_state.go | Introduces exported default threshold and a configurable process-wide default captured by new PluginState instances. |
| pkg/epp/framework/interface/plugin/plugin_state_test.go | Updates stale-time calculations and adds a new test for the default-threshold override behavior. |
| cmd/epp/runner/runner.go | Applies the configured threshold early in startup before plugins are instantiated. |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Mohammad Ashar Khan <140482588+asharkhan3101@users.noreply.github.com>
Signed-off-by: mohammadkhan <mohammadkhan@digitalocean.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind feature
What this PR does / why we need it: In decode heavy with non-streaming, it is fairly common that a request may take over 5 minutes (default staleness threshold), which causes plugin state to remove those from accounting, when used with inflight flow control this results in prematurely releasing request or token budget which results in EPP admitting more request than configured.
This PR allows us to customize this staleness threshold, it adds a new epp command line argument
--plugin-state-staleness-thresholddefaulting to 5m.Which issue(s) this PR fixes: #2137
Fixes #2137
Release note (write
NONEif no user-facing change):