Add Gateway API conformance tests to CI pipeline #11581
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
marvin-tigera
added
release-note-required
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds Gateway API conformance testing to the Calico CI pipeline by integrating the upstream Gateway API v1.3.0 conformance test suite to validate Calico's Gateway API implementation through Envoy Gateway.
Key Changes:
- Adds Gateway API v1.3.0 dependency and conformance test infrastructure
- Creates new e2e test binary for Gateway API conformance testing
- Integrates Gateway API conformance tests into SemaphoreCI pipeline with automatic artifact collection
Reviewed changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| go.mod | Adds Gateway API v1.3.0 dependency and updates jsonreference version |
| gateway/test/conformance/manifests/gatewayclass.yaml | Defines GatewayClass resource for conformance testing |
| gateway/test/conformance/manifests/gateway.yaml | Defines Gateway resource for conformance testing infrastructure |
| gateway/test/conformance/README.md | Comprehensive documentation for running and debugging Gateway API conformance tests |
| e2e/cmd/gateway/e2e_test.go | Implements conformance test runner that integrates upstream Gateway API test suite |
| e2e/Makefile | Adds build target for Gateway API conformance test binary |
| Makefile | Adds Makefile target to run Gateway API conformance tests against existing cluster |
| .semaphore/semaphore.yml.d/blocks/20-gateway-conformance.yml | Defines SemaphoreCI block configuration for Gateway API conformance tests |
| .semaphore/semaphore.yml | Integrates Gateway API conformance test block into main CI pipeline |
| .semaphore/semaphore-scheduled-builds.yml | Integrates Gateway API conformance tests into scheduled builds |
.semaphore/semaphore.yml
Outdated
| - name: google-service-account-for-gce | ||
| - name: "E2E tests: Gateway API Conformance" | ||
| run: | ||
| when: "false or change_in(['/gateway', '/e2e/cmd/gateway', '/third_party/envoy-gateway'], {pipeline_file: 'ignore', exclude: ['/**/.gitignore', '/**/README.md', '/**/LICENSE']})" |
There was a problem hiding this comment.
The condition starts with "false or" which means the Gateway API conformance tests will only run when files change in the specified directories, never by default. This differs from the block definition in 20-gateway-conformance.yml which uses "${FORCE_RUN} or". This inconsistency means FORCE_RUN won't work as expected. Change "false" to "${FORCE_RUN}" to align with the block definition and allow manual triggering.
Suggested change
| when: "false or change_in(['/gateway', '/e2e/cmd/gateway', '/third_party/envoy-gateway'], {pipeline_file: 'ignore', exclude: ['/**/.gitignore', '/**/README.md', '/**/LICENSE']})" | |
| when: "${FORCE_RUN} or change_in(['/gateway', '/e2e/cmd/gateway', '/third_party/envoy-gateway'], {pipeline_file: 'ignore', exclude: ['/**/.gitignore', '/**/README.md', '/**/LICENSE']})" |
| - name: google-service-account-for-gce | ||
| - name: "E2E tests: Gateway API Conformance" | ||
| run: | ||
| when: "true or change_in(['/gateway', '/e2e/cmd/gateway', '/third_party/envoy-gateway'], {pipeline_file: 'ignore', exclude: ['/**/.gitignore', '/**/README.md', '/**/LICENSE']})" |
There was a problem hiding this comment.
The condition "true or change_in(...)" means the Gateway API conformance tests will always run in scheduled builds regardless of file changes, which makes the change_in() check redundant. This appears intentional for scheduled builds but the redundant condition reduces clarity. Consider using just "true" or documenting why both conditions are present.
Suggested change
| when: "true or change_in(['/gateway', '/e2e/cmd/gateway', '/third_party/envoy-gateway'], {pipeline_file: 'ignore', exclude: ['/**/.gitignore', '/**/README.md', '/**/LICENSE']})" | |
| when: "true" |
electricjesus
added
release-note-not-required
| KUBECONFIG=$(KIND_KUBECONFIG) ./e2e/bin/gateway/e2e.test \ | ||
| -gateway-class=calico-gateway \ | ||
| -supported-features=Gateway,HTTPRoute \ | ||
| -cleanup-base-resources=true \ |
There was a problem hiding this comment.
The -allow-crds-mismatch flag is included without explanation. This flag appears to allow running tests even when CRD versions don't match exactly, which could mask compatibility issues. Consider adding a comment explaining why this flag is necessary for the Calico Gateway API conformance tests.
Suggested change
| -cleanup-base-resources=true \ | |
| -cleanup-base-resources=true \ | |
| # Allow minor version skew between the Gateway API CRDs installed by Calico | |
| # and the versions expected by the upstream conformance test binary. This | |
| # keeps the tests runnable while our bundled CRDs may lag slightly behind | |
| # the latest upstream definitions. |
electricjesus
force-pushed
the
gateway-api-conformance-tests
branch
from
8fca315 to
4251945
Compare
Implements Gateway API conformance testing in the Calico CI pipeline using Tigera's hardened Envoy Gateway distribution. Implementation: - e2e test binary integrating upstream Gateway API v1.3.0 conformance suite - GatewayAPI operator resource to enable Tigera Gateway support - Test manifests (Gateway resource using tigera-gateway-class) - Makefile targets for building and running tests - SemaphoreCI pipeline block for automated testing - Comprehensive README with local testing workflow Testing: - Supports Gateway and HTTPRoute conformance profiles - 260+ conformance test cases covering hostname matching, cross-namespace routing, invalid references, and HTTP routing - Uses embedded manifests from upstream conformance suite - Handles experimental channel CRDs via -allow-crds-mismatch flag Dependencies: - sigs.k8s.io/gateway-api v1.3.0 The conformance tests validate that Calico's Gateway API implementation meets the standard specification and catch regressions early.
Auto-generated changes from previous commit: - semaphore.yml: Add Gateway API conformance test block - semaphore-scheduled-builds.yml: Add Gateway API conformance test block - deps.txt: Update module dependencies for Gateway API v1.3.0
Use hardcoded path hack/test/kind/kind-kubeconfig.yaml instead of
${KIND_KUBECONFIG} variable which is not available in shell context.
KIND_KUBECONFIG is a Makefile variable, not a shell environment variable.
Add kubectl wait for gatewayclasses.gateway.networking.k8s.io CRD to be established before attempting to wait for tigera-gateway-class resource. The Tigera operator needs time to install Gateway API CRDs after the GatewayAPI resource is applied. Without this wait, the subsequent kubectl wait for the gatewayclass resource fails with 'server doesn't have a resource type gatewayclass'.
Replace single kubectl wait command with two-step approach: 1. Poll until CRD exists (handles operator installation delay) 2. Wait for Established condition (ensures CRD is ready) This fixes the issue where kubectl wait fails immediately if the gatewayclasses.gateway.networking.k8s.io CRD doesn't exist yet when the Tigera operator is still installing it.
Build envoy-gateway, envoy-proxy, and envoy-ratelimit images from current commit and load them into KIND cluster before running conformance tests. This ensures Gateway API conformance tests validate the current codebase rather than pulling released images from external registries. Changes: - Build Gateway images before creating KIND cluster - Load built images into KIND after cluster creation - Scale tigera-operator back up before applying GatewayAPI resource (operator was scaled to 0 by kind-k8st-setup) - Wait for operator to be available before proceeding This follows the same pattern as other e2e tests: build current code, deploy to KIND, and run tests against it.
Extract Gateway image building and setup steps from CI configuration into a reusable Makefile target (e2e-gateway-setup). This simplifies the CI configuration from 38 lines to 7 lines while making the Gateway setup process reusable for local development. The new target handles: - Building Gateway images (envoy-gateway, envoy-proxy, envoy-ratelimit) - Loading images into KIND cluster - Scaling tigera-operator back up - Applying GatewayAPI resource and waiting for readiness Changes: - Add e2e-gateway-setup target to root Makefile - Simplify Gateway conformance CI block to call new target - Add progress messages for better visibility Net reduction: 31 lines of code
electricjesus
force-pushed
the
gateway-api-conformance-tests
branch
from
aa796e9 to
41c5c81
Compare
…references - Use $(KIND) and $(KUBECTL) Make variables instead of bare commands to ensure tools are found from correct cached locations - Add envoy-gateway, envoy-proxy, envoy-ratelimit to K8ST_IMAGE_TARS - Build and tag envoy images as calico/envoy-*:test-build in node/Makefile - Update load_images_on_kind_cluster.sh to load envoy image tars - Simplify e2e-gateway-setup to only scale operator and apply resources (images are now loaded by kind-k8st-setup) - Restore envoy CI blocks that were temporarily deleted - Update gateway conformance block change detection paths - Regenerate semaphore.yml from updated blocks
Add BUILD_CACHE_GROUP=node to Gateway API Conformance block to share Go build cache with other node-related blocks, avoiding rebuilding from scratch.
- Poll for GatewayClass creation instead of immediately waiting - Show operator logs if GatewayClass is not created - Verify envoy images are loaded in KIND before applying GatewayAPI - Display GatewayAPI resource status after applying This will help diagnose why tigera-gateway-class is not being created by the operator.
| @@ -1,15 +0,0 @@ | |||
| - name: API | |||
Member
There was a problem hiding this comment.
@electricjesus was it intentional to remove all of these blocks? Seems wrong?
Member
Author
There was a problem hiding this comment.
hey casey - just got back to this. intentional but only temporary so i can "focus" semaphore only on the conformance tests that i want
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds Gateway API conformance testing to the Calico CI pipeline.
Automated conformance testing to ensure Calico's Gateway API implementation meets the standard specification and to catch regressions early.
Installs latest calicoreates an
operator.tigera.io/GatewayAPIresource with matchinggateway.networking.k8s.io/Gatewaythen Gateway Conformance tests are run.Related issues/PRs
N/A
Todos
Release Note