Support running e2e tests across different Gw API versions

[sheidkamp]

Description

• Update the base TestCase
  • Add MinGwApiVersion and MaxGwApiVersion to allow tests to be skipped if version constraints aren't met
• Update the base TestSuite
  • Add MinGwApiVersion to allow an entire suite to be skipped. This is important because even if tests are skipped, suite level configuration will still be applied and may contain resources incompatible with the installed API version
• Update tests to run successfully (or skip) across versions 1.2.1 - 1.4.0 on both the conformance and expiermental channels
• Add running these tests to the nightly jobs.

Passing nightly tests in fork: https://github.com/sheidkamp/kgateway/actions/runs/19080780416/job/54508476958 (minus the Lambda suite which I did not have the environment setup for)

Example running locally:

 CONFORMANCE_CHANNEL=standard CONFORMANCE_VERSION=v1.2.0 make setup
 go test -v ./test/kubernetes/e2e/... -run "TestKgateway$/^HeaderModifiers" -timeout 15m -tags e2e

You should see

--- PASS: TestKgateway (31.12s)
    --- PASS: TestKgateway/HeaderModifiers (22.08s)
        --- PASS: TestKgateway/HeaderModifiers/TestGatewayLevelHeaderModifiers (0.58s)
        --- SKIP: TestKgateway/HeaderModifiers/TestListenerSetLevelHeaderModifiers (0.00s)
        --- PASS: TestKgateway/HeaderModifiers/TestMultiLevelHeaderModifiers (0.79s)
        --- SKIP: TestKgateway/HeaderModifiers/TestMultiLevelHeaderModifiersWithListenerSet (0.00s)
        --- PASS: TestKgateway/HeaderModifiers/TestRouteLevelHeaderModifiers (0.60s)
PASS

Resolves #12721

Change Type

/kind design
/kind cleanup

Changelog

NONE

Additional Notes

The first commit is base test suite changes and CI jobs
The second commit is the test updates.
Further commits are tweaks/updates/fixes/cleanup

[Copilot]

Pull Request Overview

This PR implements support for running E2E tests against different versions and channels (standard/experimental) of the Gateway API. It introduces a versioning framework that allows tests to declare minimum/maximum version requirements and automatically skip tests when requirements are not met.

Key changes:

• Adds Gateway API version detection and caching mechanism based on CRD annotations
• Implements test-level and suite-level version constraints via MinGwApiVersion and MaxGwApiVersion fields
• Introduces SetupByVersion to allow different suite setups based on installed Gateway API version
• Updates multiple test suites to specify appropriate version requirements for experimental features

Reviewed Changes

Copilot reviewed 45 out of 45 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/e2e/tests/base/base_suite.go	Core implementation of version detection, constraint checking, and versioned setup selection
test/e2e/tests/kgateway_tests.go	Comments out Lambda test (temporarily for local/fork testing)
test/e2e/tests/listenerset_test.go	Removes manual CRD existence check (replaced by version requirements)
test/e2e/features/*/types.go, suite.go	Updates test suites to declare Gateway API version requirements for experimental features
.github/workflows/nightly-tests.yaml	Adds workflow to run E2E tests across different Gateway API versions
go.mod	Promotes semver package from indirect to direct dependency
design/12721.md	Design document explaining the versioning approach

Comments suppressed due to low confidence (1)

test/e2e/features/metrics/testdata/setup.yaml:1

• The removed 'allowedListeners' configuration appears to be a necessary change for backwards compatibility with older Gateway API versions, but the comment explaining this removal is missing. Consider adding a comment explaining why this was removed from the base setup and moved to the versioned setup.

kind: Gateway

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[davidjumani]

Suggested change

	type ThreadSafeWriter struct {
	type Writer struct {

since this is in the threadsafe package

[sheidkamp]

renamed to "WriterWrapper"

[davidjumani]

cleanup ?

[sheidkamp]

cleaned up.

[davidjumani]

Could you help me understand why this is a map ? Why can't this be a string / enum of channel_version ?

[sheidkamp]

The biggest reason was to make it impossible to enter bad/ambiguous configuration.

With the type you can only use known channels and valid semvars, and are limited to one version per channel by the map. It also stores the data in a way that makes it easy to access when checking.

Maybe I'm misunderstanding, but are you thinking the string would look something like "experimental:1.2.0;standard:1.40"?

[timflannagan]

Overall, approach sounds reasonable to me. I think we could investigate the race issues around the buffer a bit more but that can be tackled over time imo.

[sheidkamp]

Overall, approach sounds reasonable to me. I think we could investigate the race issues around the buffer a bit more but that can be tackled over time imo.

Thanks, once this merges I will create an issue with links to the relevant spots in code.