tests: Add conformace tests for listenersets #3890

davidjumani · 2025-06-30T15:26:36Z

What type of PR is this?

/kind test
/area conformance-test

What this PR does / why we need it:

This PR contains an initial conformance test for listenersets. This aims to verify the following :

Translation of a listenerset by an implementer
Route inheritance of gateway-level routes by the listenerset
Route isolation between gateways and listenersets based on parentRef and sectionName
Gateway, ListenerSet, Route and Service in different namespaces
ListenerSets not allowed on the gateway
ListenerSets with protocol and hostname conflict

Run against kgateway

--- PASS: TestConformance/ListenerSetHostnameConflict (4.21s)
    --- PASS: TestConformance/ListenerSetHostnameConflict/5_request_to_'hostname-conflict-listener-1.com/listenerset-2-route'_should_receive_one_of_[] (0.03s)
    --- PASS: TestConformance/ListenerSetHostnameConflict/7_request_to_'hostname-conflict-listener-2.com/listenerset-2-route'_should_receive_one_of_[] (0.03s)
    --- PASS: TestConformance/ListenerSetHostnameConflict/4_request_to_'hostname-conflict-listener-1.com/listenerset-1-route'_should_receive_one_of_[] (0.03s)
    --- PASS: TestConformance/ListenerSetHostnameConflict/6_request_to_'hostname-conflict-listener-2.com/listenerset-1-route'_should_go_to_infra-backend-v2 (0.03s)
    --- PASS: TestConformance/ListenerSetHostnameConflict/1_request_to_'listenerset-1-listener.com/listenerset-1-route'_should_go_to_infra-backend-v2 (0.03s)
    --- PASS: TestConformance/ListenerSetHostnameConflict/3_request_to_'hostname-conflict-listener-1.com/gateway-route'_should_go_to_infra-backend-v1 (0.04s)
    --- PASS: TestConformance/ListenerSetHostnameConflict/2_request_to_'listenerset-2-listener.com/listenerset-2-route'_should_go_to_infra-backend-v3 (0.04s)
    --- PASS: TestConformance/ListenerSetHostnameConflict/0_request_to_'gateway-listener.com/gateway-route'_should_go_to_infra-backend-v1 (0.04s)
--- PASS: TestConformance/ListenerSetCrossNamespace (3.33s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/6_request_to_'listenerset-1-listener-1.com/gateway-listener-2-listener-route'_should_receive_one_of_[] (0.03s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/8_request_to_'gateway-listener-1.com/listenerset-1-route'_should_receive_one_of_[] (0.03s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/16_request_to_'listenerset-2-listener-1.com/listenerset-2-route'_should_receive_one_of_[] (0.04s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/9_request_to_'gateway-listener-2.com/listenerset-1-route'_should_receive_one_of_[] (0.04s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/4_request_to_'gateway-listener-1.com/gateway-listener-2-listener-route'_should_receive_one_of_[] (0.03s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/2_request_to_'listenerset-1-listener-1.com/gateway-route'_should_receive_one_of_[] (0.04s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/7_request_to_'listenerset-1-listener-2.com/gateway-listener-2-listener-route'_should_receive_one_of_[] (0.04s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/1_request_to_'gateway-listener-2.com/gateway-route'_should_go_to_infra-backend-v1 (0.04s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/5_request_to_'gateway-listener-2.com/gateway-listener-2-listener-route'_should_go_to_infra-backend-v1 (0.04s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/0_request_to_'gateway-listener-1.com/gateway-route'_should_go_to_infra-backend-v1 (0.05s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/12_request_to_'gateway-listener-1.com/listenerset-1-listener-2-listener-route'_should_receive_one_of_[] (0.01s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/14_request_to_'listenerset-1-listener-1.com/listenerset-1-listener-2-listener-route'_should_receive_one_of_[] (0.01s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/3_request_to_'listenerset-1-listener-2.com/gateway-route'_should_receive_one_of_[] (0.02s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/13_request_to_'gateway-listener-2.com/listenerset-1-listener-2-listener-route'_should_receive_one_of_[] (0.02s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/15_request_to_'listenerset-1-listener-2.com/listenerset-1-listener-2-listener-route'_should_go_to_infra-backend-v1 (0.02s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/11_request_to_'listenerset-1-listener-2.com/listenerset-1-route'_should_go_to_infra-backend-v1 (0.02s)
    --- PASS: TestConformance/ListenerSetCrossNamespace/10_request_to_'listenerset-1-listener-1.com/listenerset-1-route'_should_go_to_infra-backend-v1 (0.02s)
--- PASS: TestConformance/ListenerSetNotAllowed (3.11s)
    --- PASS: TestConformance/ListenerSetNotAllowed/2_request_to_'bar.com/route'_should_receive_one_of_[] (0.04s)
    --- PASS: TestConformance/ListenerSetNotAllowed/1_request_to_'foo.com/route'_should_receive_one_of_[] (0.04s)
    --- PASS: TestConformance/ListenerSetNotAllowed/0_request_to_'example.com/route'_should_go_to_infra-backend-v1 (0.05s)
--- PASS: TestConformance/ListenerSetProtocolConflict (4.21s)
    --- PASS: TestConformance/ListenerSetProtocolConflict/4_request_to_'protocol-conflict-listener-1.com/listenerset-1-route'_should_receive_one_of_[] (0.03s)
    --- PASS: TestConformance/ListenerSetProtocolConflict/5_request_to_'protocol-conflict-listener-1.com/listenerset-2-route'_should_receive_one_of_[] (0.03s)
    --- PASS: TestConformance/ListenerSetProtocolConflict/3_request_to_'protocol-conflict-listener-1.com/gateway-route'_should_go_to_infra-backend-v1 (0.04s)
    --- PASS: TestConformance/ListenerSetProtocolConflict/0_request_to_'gateway-listener.com/gateway-route'_should_go_to_infra-backend-v1 (0.04s)
    --- PASS: TestConformance/ListenerSetProtocolConflict/7_request_to_'protocol-conflict-listener-2.com/listenerset-2-route'_should_receive_one_of_[] (0.04s)
    --- PASS: TestConformance/ListenerSetProtocolConflict/6_request_to_'protocol-conflict-listener-2.com/listenerset-1-route'_should_go_to_infra-backend-v2 (0.04s)
    --- PASS: TestConformance/ListenerSetProtocolConflict/1_request_to_'listenerset-1-listener.com/listenerset-1-route'_should_go_to_infra-backend-v2 (0.04s)
    --- PASS: TestConformance/ListenerSetProtocolConflict/2_request_to_'listenerset-2-listener.com/listenerset-2-route'_should_go_to_infra-backend-v3 (0.04s)

Which issue(s) this PR fixes:

Fixes #3785

Does this PR introduce a user-facing change?:

Adding initial conformance tests for XListenerSets

k8s-ci-robot · 2025-06-30T15:26:39Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2025-06-30T15:26:44Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: davidjumani
Once this PR has been reviewed and has the lgtm label, please assign youngnick for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

davidjumani · 2025-07-01T01:58:09Z

/test all

davidjumani · 2025-08-16T09:54:58Z

/retest

robscott · 2025-08-18T22:34:02Z

/cc @rikatz @dprotaso

youngnick · 2025-08-20T05:07:13Z

conformance/tests/listenerset-domain-name-conflict.go

+			// Requests to the listener with domain name conflict should not work
+			{
+				Request:  http.Request{Host: "conflict.com", Path: "/gateway-route"},
+				Response: http.Response{StatusCode: 404},
+			},


Hmm, I don't know if we were clear enough about this in the GEP, but in this case, because there is a Gateway Listener, it should win over the ListenerSet listener. (see https://gateway-api.sigs.k8s.io/geps/gep-1713/#listener-precedence for the details here)

In this and other conflict cases, there should always be one "winning" Listener from somewhere, and that Listener should end up Accepted, and traffic should flow.

The intent of that is to stop the creation of conflicts from stopping traffic flowing (that's why "oldest first wins" for ListenerSets).

I'm confused since the GEP mentions

Implementations MUST treat the parent Gateways as having the merged list of all listeners from itself and attached ListenerSets and validation of this list of listeners MUST behave the same as if the list were part of a single Gateway with the relaxed listener name constraints.

and so I based the validation tests as though they were defined on a single gateway - My understanding was that the ordering was purely for merging them into a gateway and not validating them any different

Okay I see #3978 has been updated. Shall I wait for it to merge before making the changes here ?

conformance/utils/kubernetes/helpers.go

rikatz · 2025-10-23T19:09:47Z

conformance/tests/listenerset-same-namespace.yaml

+    hostname: "listenerset-1-listener-2.com"
+    allowedRoutes:
+      namespaces:
+        from: All


interesting: do we expect this to work, if the allowedListener defines just ListenerSet of the same namespace, or not?

I think the idea is to allow, just checking

I'm a bit confused. Are you asking if the Listener on the ListenerSet will accept routes from all namespaces if the Gateway's allowedListener is set to only listenerSets from the same namespace?
If so, it should allow since the allowedListener restricts listenersets and should have no effect on the routes

rikatz · 2025-10-23T19:51:41Z

I still have some questions, but overall lgtm and I would like to see these tests running with other implementations. I will try to trigger it tomorrow against Istio and kgateway to see how they behave (failing is fine, as this may not be implemented yet)

I have mapped some missing tests, but not for this PR:

ListenerSet entry name unique on same set
count of attached routes (maybe it is ther,e but I didn't spotted it)
Port is now optional to allow for dynamic port assignment. If the port is unspecified or set to zero, the implementation will assign a unique port.
When a ReferenceGrant is applied to a Gateway it MUST NOT be inherited by child ListenerSets. Thus a ListenerSet listener MUST NOT access secrets granted to the Gateway listeners

davidjumani · 2025-10-27T14:50:54Z

Thanks @rikatz I've updated the PR and description with the passing run on kgateway
I'll also add the tests that have been suggested in a follow up

davidjumani · 2025-10-27T17:55:23Z

/retest

This reverts commit 3ec5493.

davidjumani · 2025-10-28T15:42:49Z

I have mapped some missing tests, but not for this PR:

Thanks @rikatz
I've raised a PR with the list of conformance tests I believe would be sufficient to validate this feature
Add conformance details to ListenerSets (GEP-1713)

rikatz · 2025-12-02T19:20:15Z

/lgtm
/hold cancel

The tests looks fine and reflect what we've been discussing. Some tests will need to be updated after GEP updates are merged, but this is a great start for implementations

/cc @robscott @shaneutt

rikatz · 2025-12-02T19:20:32Z

/remove-label release-blocker

k8s-ci-robot · 2025-12-02T19:20:34Z

@rikatz: The label(s) /remove-label release-blocker cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, ci-short, ci-extended, ci-full. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

Details

In response to this:

/remove-label release-blocker

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-12-04T22:59:26Z

New changes are detected. LGTM label has been removed.

davidjumani · 2025-12-15T19:51:24Z

I'll update these tests and the API to use AttachedListenerSets as a count once merged

tests: Add conformace tests for listenersets

3b1f44b

k8s-ci-robot requested review from robscott and sunjayBhatia June 30, 2025 15:26

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jun 30, 2025

fix lint

9e130f2

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 21, 2025

davidjumani added 2 commits August 15, 2025 22:25

Merge remote-tracking branch 'origin/main' into add-ls-conformace-tests

77ebf23

add more tests

1d02411

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 16, 2025

add conflict tests

c108086

davidjumani marked this pull request as ready for review August 16, 2025 07:44

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 16, 2025

k8s-ci-robot requested review from LiorLieberman and mlavacca August 16, 2025 07:44

davidjumani mentioned this pull request Aug 16, 2025

Conformance Tests for ListenerSets #3785

Open

k8s-ci-robot requested review from dprotaso and rikatz August 18, 2025 22:34

youngnick reviewed Aug 20, 2025

View reviewed changes

rikatz reviewed Oct 23, 2025

View reviewed changes

conformance/utils/kubernetes/helpers.go Outdated Show resolved Hide resolved

rikatz reviewed Oct 23, 2025

View reviewed changes

davidjumani added 2 commits October 27, 2025 10:42

optimize matching listenerconditions

8d27faa

Merge remote-tracking branch 'origin/main' into add-ls-conformace-tests

6473f3e

dont return errors

3ec5493

Revert "dont return errors"

f9423b7

This reverts commit 3ec5493.

davidjumani mentioned this pull request Nov 19, 2025

STD: ListenerSets (GEP-1713) #4249

Open

1 task

k8s-ci-robot requested a review from shaneutt December 2, 2025 19:20

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Dec 2, 2025

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 4, 2025

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. kind/gep PRs related to Gateway Enhancement Proposal(GEP) labels Dec 4, 2025

davidjumani force-pushed the add-ls-conformace-tests branch from 16f861a to f9423b7 Compare December 4, 2025 23:05

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Dec 4, 2025

davidjumani force-pushed the add-ls-conformace-tests branch from afe4530 to f9423b7 Compare December 5, 2025 02:59

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 5, 2025

Merge branch 'main' into add-ls-conformace-tests

121d6c6

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 5, 2025

codgen

5638e97

tests: Add conformace tests for listenersets #3890

Are you sure you want to change the base?

tests: Add conformace tests for listenersets #3890

Uh oh!

Conversation

davidjumani commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jun 30, 2025

Uh oh!

k8s-ci-robot commented Jun 30, 2025

Uh oh!

davidjumani commented Jul 1, 2025

Uh oh!

davidjumani commented Aug 16, 2025

Uh oh!

robscott commented Aug 18, 2025

Uh oh!

youngnick Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

davidjumani Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidjumani Aug 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rikatz Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

davidjumani Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

rikatz commented Oct 23, 2025

Uh oh!

davidjumani commented Oct 27, 2025

Uh oh!

davidjumani commented Oct 27, 2025

Uh oh!

davidjumani commented Oct 28, 2025

Uh oh!

rikatz commented Dec 2, 2025

Uh oh!

rikatz commented Dec 2, 2025

Uh oh!

k8s-ci-robot commented Dec 2, 2025

Uh oh!

k8s-ci-robot commented Dec 4, 2025

Uh oh!

davidjumani commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

davidjumani commented Jun 30, 2025 •

edited

Loading

davidjumani Aug 21, 2025 •

edited

Loading