draft: fast e2e tests #12993

howardjohn · 2025-11-26T17:08:55Z

Description

Read #12981 for background.

This PR optimizes 2 e2e tests. I randomly picked these as samples, there is nothing special about these tests, and the same could be applied to most or all of our tests.

Before this PR:

--- PASS: TestAgentgatewayIntegration (35.97s)
    --- PASS: TestAgentgatewayIntegration/ApiKeyAuth (17.31s)
        --- PASS: TestAgentgatewayIntegration/ApiKeyAuth/TestGatewayPolicy (2.76s)
        --- PASS: TestAgentgatewayIntegration/ApiKeyAuth/TestRoutePolicy (0.84s)
    --- PASS: TestAgentgatewayIntegration/BasicAuth (18.53s)
        --- PASS: TestAgentgatewayIntegration/BasicAuth/TestGatewayPolicy (3.27s)
        --- PASS: TestAgentgatewayIntegration/BasicAuth/TestRoutePolicy (1.28s)
ok      github.com/kgateway-dev/kgateway/v2/test/e2e/tests      35.985s

After this PR:

2025-11-26T09:04:29.425069 --- PASS: TestAgentgatewayIntegration (0.22s)
2025-11-26T09:04:29.425072     --- PASS: TestAgentgatewayIntegration/BasicAuth (0.09s)
2025-11-26T09:04:29.425075         --- PASS: TestAgentgatewayIntegration/BasicAuth/TestGatewayPolicy (0.03s)
2025-11-26T09:04:29.425078         --- PASS: TestAgentgatewayIntegration/BasicAuth/TestRoutePolicy (0.03s)
2025-11-26T09:04:29.425081     --- PASS: TestAgentgatewayIntegration/ApiKeyAuth (0.10s)
2025-11-26T09:04:29.425084         --- PASS: TestAgentgatewayIntegration/ApiKeyAuth/TestGatewayPolicy (0.02s)
2025-11-26T09:04:29.425086         --- PASS: TestAgentgatewayIntegration/ApiKeyAuth/TestRoutePolicy (0.04s)

Changes I applied:

Deploy a single Gateway and test backend that is shared and re-used across tests (biggest change, by far). Replace curl pod with native Go code. Drops 35s->3s
Replace kubectl apply with Istio apply (cuts 200ms off each test)
Rebase on b96e132 (now merged) (cuts 100ms of each test)
Replace kubectl delete with istio delete: cuts 200ms
use istio apply on base setup as well, instead of just each test: cuts 300ms off suite
Parallel apply within each file (instead of across files): 50ms
Async delete: drops 20ms (note: at this point, the entire test is 50ms so this is a notable saving as well). I had to disable this though since it needs more fixes
Replace helm status with native code: Cuts 150ms off the test suite

edit: did some more.
--- PASS: TestAgentgatewayIntegration (1.15s)
--- PASS: TestAgentgatewayIntegration/ApiKeyAuth (0.16s)
--- PASS: TestAgentgatewayIntegration/ApiKeyAuth/TestGatewayPolicy (0.05s)
--- PASS: TestAgentgatewayIntegration/ApiKeyAuth/TestRoutePolicy (0.07s)
--- PASS: TestAgentgatewayIntegration/JwtAuth (0.21s)
--- PASS: TestAgentgatewayIntegration/JwtAuth/TestGatewayPolicy (0.04s)
--- PASS: TestAgentgatewayIntegration/JwtAuth/TestGatewayPolicyWithRbac (0.04s)
--- PASS: TestAgentgatewayIntegration/JwtAuth/TestRoutePolicy (0.04s)
--- PASS: TestAgentgatewayIntegration/JwtAuth/TestRoutePolicyWithRbac (0.04s)
--- PASS: TestAgentgatewayIntegration/CSRF (0.10s)
--- PASS: TestAgentgatewayIntegration/CSRF/TestGatewayLevelCSRF (0.06s)
--- PASS: TestAgentgatewayIntegration/RBAC (0.08s)
--- PASS: TestAgentgatewayIntegration/RBAC/TestRBACHeaderAuthorization (0.01s)
--- PASS: TestAgentgatewayIntegration/Transformation (0.11s)
--- PASS: TestAgentgatewayIntegration/Transformation/TestGatewayWithTransformedRoute (0.01s)
--- PASS: TestAgentgatewayIntegration/BackendTLSPolicy (0.27s)
--- PASS: TestAgentgatewayIntegration/BackendTLSPolicy/TestBackendTLSPolicyAndStatus (0.12s)
--- PASS: TestAgentgatewayIntegration/BasicAuth (0.20s)
--- PASS: TestAgentgatewayIntegration/BasicAuth/TestGatewayPolicy (0.04s)
--- PASS: TestAgentgatewayIntegration/BasicAuth/TestRoutePolicy (0.12s)

End to end full GHA flow: 8minutes
Actual time of testing: 23.956s

Change Type

/kind cleanup

Changelog

NONE

Additional Notes

sheidkamp · 2025-11-26T19:46:50Z

test/e2e/tests/agent_gateway_test.go

 	// Install kgateway
 	testInstallation.InstallKgatewayFromLocalChart(ctx)

+	common.SetupBaseConfig(ctx, t, testInstallation, filepath.Join("manifests", "agent-gateway-base.yaml"))


Want to make you aware of this PR which added nightly e2e tests with different Gateway API versions. It mostly works by tagging tests with minimum versions, but it also introduced a mechanism to use different manifests for suite setup. The main use case is to switch between a gateway that defines "allowedListeners" and one that doesn't based on the API version.

I don't think its relevant for agentgateway at the moment, but may be in the future, and will be if we extend these improvements to the kgateway tests.

Copilot

Pull request overview

This PR optimizes e2e test execution by implementing several performance improvements, reducing test time from ~36s to ~0.2s for sample tests. The main optimization is deploying a single shared Gateway and backend that is reused across tests, replacing individual per-test deployments. Additional improvements include replacing kubectl with Istio's native apply/delete, using native Go HTTP requests instead of curl pods, and optimizing helper functions.

Key changes:

Shared test infrastructure with reusable Gateway/backend in agentgateway-base namespace
Native Go HTTP client implementation replacing curl pods
Istio client integration for faster resource operations
Test manifest updates to reference shared resources

Reviewed changes

Copilot reviewed 89 out of 92 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
test/e2e/common/base.go	New shared gateway setup and native HTTP client
test/e2e/tests/base/base_suite.go	Replaced kubectl with Istio apply/delete
test/e2e/testutils/cluster/kind.go	Added IstioClient integration
pkg/utils/requestutils/curl/native_request.go	New native Go HTTP request implementation
test/e2e/tests/manifests/agent-gateway-base.yaml	New shared base resources
Various test manifests	Updated to use shared agentgateway-base namespace
pkg/agentgateway/translator/	TLS/mTLS support improvements
go.mod	Local replace directive and dependency updates

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

Copilot · 2025-11-27T00:44:27Z

test/e2e/tests/base/base_suite.go

+	//for _, manifest := range testCase.Manifests {
+	//	log.Errorf("howardjohn: apply..")
+	err := s.TestInstallation.ClusterContext.IstioClient.ApplyYAMLFiles("", testCase.Manifests...)
+	gomega.Expect(err).NotTo(gomega.HaveOccurred())
+	//gomega.Eventually(func() error {
+	//	err := s.TestInstallation.Actions.Kubectl().ApplyFile(s.Ctx, manifest)
+	//	return err
+	//}, 10*time.Second, 1*time.Second).Should(gomega.Succeed(), "can apply "+manifest)
+	//}


Remove commented-out debug code before merging. The commented code includes a debug log statement and the old implementation, which should be cleaned up for production code.

Copilot · 2025-11-27T00:44:27Z

test/e2e/tests/base/base_suite.go

+	// TODO: this is a bottleneck. Doing it async is great but we need barriers to make sure if we do
+	// delete(foo); add(foo) we end up in the right state.
+	// We also need to make sure we finish the deletion before exiting the process
+	//go func() {
+	err := s.TestInstallation.ClusterContext.IstioClient.DeleteYAMLFiles("", testCase.Manifests...)
+	gomega.Expect(err).NotTo(gomega.HaveOccurred())
+	//}()
+	return


Address the TODO comment about async deletion barriers. The commented async implementation and early return suggest incomplete work that needs proper synchronization before enabling.

Copilot · 2025-11-27T00:44:27Z

test/e2e/tests/base/base_suite.go

+	//for _, manifest := range testCase.Manifests {
+	//gomega.Eventually(func() error {
+	//	err := s.TestInstallation.Actions.Kubectl().DeleteFileSafe(s.Ctx, manifest)
+	//	return err
+	//}, 10*time.Second, 1*time.Second).Should(gomega.Succeed(), "can delete "+manifest)
+	//}


Remove commented-out old delete implementation code. This dead code should be cleaned up before merging.

Copilot · 2025-11-27T00:44:28Z

go.mod

 	sigs.k8s.io/kind
 )
+
+replace github.com/agentgateway/agentgateway => /home/john/solo/agentgateway


Remove local replace directive that points to a developer's local filesystem path. This will break builds for anyone else and should not be committed.

Suggested change

replace github.com/agentgateway/agentgateway => /home/john/solo/agentgateway

Copilot · 2025-11-27T00:44:28Z

test/e2e/tests/agent_gateway_tests.go

+	//agentgatewaySuiteRunner.Register("A2A", a2a.NewTestingSuite)
+	//agentgatewaySuiteRunner.Register("BasicRouting", agentgateway.NewTestingSuite)


Multiple test suites are commented out. If these are intentionally disabled for this draft PR, add a comment explaining why or create a tracking issue to re-enable them with the new optimizations.

Copilot · 2025-11-27T00:44:28Z

pkg/utils/requestutils/curl/native_request.go

+	// Handle SNI with custom host resolution
+	// TODO
+	if c.sni != "" {
+		panic("sni is not implemented")
+	}


SNI support is not implemented and will panic at runtime if used. Either implement this feature or remove SNI as an option until it can be properly supported.

Copilot · 2025-11-27T00:44:29Z

test/e2e/testutils/cluster/kind.go

 	kubeCtx := os.Getenv(testutils.KubeCtx)
-	if len(kubeCtx) == 0 {
-		kubeCtx = fmt.Sprintf("kind-%s", clusterName)
-	}
-
 	restCfg, err := kubeutils.GetRestConfigWithKubeContext(kubeCtx)


Removed the default context name fallback logic. If KubeCtx environment variable is empty, this will attempt to use an empty context which may fail. Consider restoring the default logic or documenting the requirement to set KubeCtx.

Copilot · 2025-11-27T00:44:29Z

pkg/agentgateway/translator/conversion.go

+	case *gatewayx.XListenerSet:
+		return wellknown.XListenerSetGVK
+	default:
+		panic("Uknown GVK")


Corrected spelling of 'Unknown' to 'Unknown'.

Suggested change

panic("Uknown GVK")

panic("Unknown GVK")

yuval-k · 2025-12-01T00:54:38Z

how do you make sure there's no test pollution? i.e. that a test passes and/or flakes because some policy from previous test didn't propagate yet for whatever reason (maybe async delete took its time)?
i know we had that problem with gloo-edge-v1

for example, when testing proxy protocol, no other test will pass until the proxy protocol is fully removed from the data plane

howardjohn · 2025-12-01T15:11:28Z

how do you make sure there's no test pollution? i.e. that a test passes and/or flakes because some policy from previous test didn't propagate yet for whatever reason (maybe async delete took its time)? i know we had that problem with gloo-edge-v1

FWIW I disabled async delete, I don't think its worth the benefit. However, you can still have pollution due to eventual consistency (as could you before this change, though it was much less likely).

In my experience in Istio we would occasionally have this issue. Usually it shows up pretty quickly as flakes that are then diagnosed as "test was really broken and incorrectly passed sometimes". In kgw I think we also randomize test order which would make the chance of it correctly failing much much higher.

If we need to, we can add some additional barriers to check the config is cleaned up end to end (e.g. send requests that should fail?). Or, we can also make the tests use non-overlapping configs like distinct hostnames. Given 90% of the tests are just "apply a policy, send some traffic" we could also just have the framework entirely manage an assigned hostname too.

Signed-off-by: John Howard <[email protected]>

gateway-bot added kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. release-note-none labels Nov 26, 2025

sheidkamp reviewed Nov 26, 2025

View reviewed changes

howardjohn marked this pull request as ready for review November 27, 2025 00:34

Copilot AI review requested due to automatic review settings November 27, 2025 00:34

Copilot started reviewing on behalf of howardjohn November 27, 2025 00:34 View session

Copilot finished reviewing on behalf of howardjohn November 27, 2025 00:35

Copilot AI reviewed Nov 27, 2025

View reviewed changes

howardjohn force-pushed the agw/fast-e2e branch from f252461 to 0a9f683 Compare November 30, 2025 23:55

howardjohn force-pushed the agw/fast-e2e branch 2 times, most recently from 71c90d3 to b172b1e Compare December 1, 2025 15:44

howardjohn mentioned this pull request Dec 2, 2025

ci: minimal CI changes for faster tests #13031

Merged

puertomontt mentioned this pull request Dec 4, 2025

e2e tests: Replace curl pod with native Go code #13053

Open

howardjohn force-pushed the agw/fast-e2e branch 4 times, most recently from e257db3 to 207ac79 Compare December 5, 2025 22:26

howardjohn added 3 commits December 8, 2025 10:38

Initial

322baf2

Signed-off-by: John Howard <[email protected]>

fixup

769fb21

Signed-off-by: John Howard <[email protected]>

set default ns

f68db2a

Signed-off-by: John Howard <[email protected]>

howardjohn force-pushed the agw/fast-e2e branch from df6efa1 to f68db2a Compare December 8, 2025 18:51

howardjohn added 3 commits December 8, 2025 12:34

ignore ns

a57262d

Signed-off-by: John Howard <[email protected]>

fix

03b5a7e

Signed-off-by: John Howard <[email protected]>

use SSA

605a91a

Signed-off-by: John Howard <[email protected]>

		//agentgatewaySuiteRunner.Register("A2A", a2a.NewTestingSuite)
		//agentgatewaySuiteRunner.Register("BasicRouting", agentgateway.NewTestingSuite)

draft: fast e2e tests #12993

Are you sure you want to change the base?

draft: fast e2e tests #12993

Uh oh!

Conversation

howardjohn commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Change Type

Changelog

Additional Notes

Uh oh!

sheidkamp Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

yuval-k commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

howardjohn commented Dec 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

howardjohn commented Nov 26, 2025 •

edited

Loading

yuval-k commented Dec 1, 2025 •

edited

Loading