Skip to content

Fix flaky inference extension test timeouts caused by accumulated state #1453

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

Fix flaky inference extension test timeouts caused by accumulated state #1453

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b758ec6
Initial plan
Copilot Oct 28, 2025
6c03814
Fix flaky test by using inference extension default timeouts
Copilot Oct 28, 2025
e0d9589
Increase GetTimeout to handle slow API server responses
Copilot Oct 28, 2025
56045ea
Merge branch 'main' into copilot/fix-flaky-test-inference-pool
mathetake Oct 28, 2025
2b99900
Add test isolation delay to reduce API server load
Copilot Oct 28, 2025
dbaedc6
Increase TestIsolation delay to 5s for sustained test runs
Copilot Oct 28, 2025
0b6af49
Increase polling interval to 15s to reduce API server pressure
Copilot Oct 28, 2025
48ee480
Update comments to reflect true root cause: accumulated state
Copilot Oct 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 13 additions & 7 deletions tests/e2e-inference-extension/conformance_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ import (

"github.com/stretchr/testify/require"
gie "sigs.k8s.io/gateway-api-inference-extension/conformance"
gieconfig "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/config"
v1 "sigs.k8s.io/gateway-api/conformance/apis/v1"
"sigs.k8s.io/gateway-api/conformance/utils/config"

"github.com/envoyproxy/ai-gateway/tests/internal/e2elib"
)
Expand All @@ -35,12 +35,18 @@ func TestGatewayAPIInferenceExtension(t *testing.T) {
Version: "latest",
}
options.ConformanceProfiles.Insert(gie.GatewayLayerProfileName)
defaultTimeoutConfig := config.DefaultTimeoutConfig()
defaultTimeoutConfig.HTTPRouteMustHaveCondition = 10 * time.Second
defaultTimeoutConfig.HTTPRouteMustNotHaveParents = 10 * time.Second
defaultTimeoutConfig.GatewayMustHaveCondition = 10 * time.Second
config.SetupTimeoutConfig(&defaultTimeoutConfig)
options.TimeoutConfig = defaultTimeoutConfig
// Use the inference extension's default timeout config which has appropriate
// timeout values for InferencePool operations (e.g., 300s for conditions).
inferenceTimeoutConfig := gieconfig.DefaultInferenceExtensionTimeoutConfig()
// Add test isolation delay to mitigate accumulated state across test runs.
// Root cause: Resources/state accumulate in the kind cluster across consecutive tests,
// causing API server slowness and "context deadline exceeded" errors. Delays between
// tests help but don't fully solve it - proper fix requires cluster recreation or
// improved cleanup between test runs.
inferenceTimeoutConfig.TestIsolation = 5 * time.Second
// Increase polling interval to reduce API call frequency and pressure on API server
inferenceTimeoutConfig.InferencePoolMustHaveConditionInterval = 15 * time.Second
options.TimeoutConfig = inferenceTimeoutConfig.TimeoutConfig
options.GatewayClassName = "inference-pool"
options.SkipTests = []string{}

Expand Down
Loading