conformance: wait for mesh weighted routes to be programmed

[Automaat]

What type of PR is this?

/kind flake
/area conformance-test

What this PR does / why we need it:

MeshGRPCRouteWeight and MeshHTTPRouteWeight start sampling the traffic distribution before the route is programmed, so they fail against implementations that take more than a second to converge.

Neither test gates on the route. MakeRequestAndExpectEventuallyConsistentResponse only needs three consecutive 200s, and in the mesh fixtures the parent echo Service already fronts the echo-v1 and echo-v2 pods, so requests succeed with no route at all. In one failing run that gate passed 164ms after the route was created. The check then retries MaxTestRetries (10) times with no delay, so the real budget is however long 10 batches take: under two seconds. Every attempt sees the pre-route 50/50 split.

ExpectWeightedDistributionBatch replaces the fixed retry count. It polls at DefaultPollInterval until MaxTimeToConsistency, so the budget becomes the configured timeout rather than the sampling speed, and the fast path still returns on the first success. #5040 and #5122 fixed the same problem for the TCP and UDP tests by adding an explicit readiness gate; the mesh tests never got one. The other four MaxTestRetries callers are unchanged.

Measured on Kuma, from apply to weights visible in the proxy config: HTTPRoute 0.31-1.81s, GRPCRoute 0.18-2.03s. Both straddle the current budget. That matches the roughly 50% failure rate in CI. kumahq/kuma#18322 skips the gRPC test downstream until this lands.

Does this PR introduce a user-facing change?:

NONE

Was AI used in preparing this PR?

AIL:3 - the change and this description were AI-drafted from my investigation of the failure, and reviewed by me before submission.

https://claude.ai/code/session_014tscnFJQ4AwC4CDdsz6P1m

[kubernetes-prow]

Welcome @Automaat!

It looks like this is your first PR to kubernetes-sigs/gateway-api 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/gateway-api has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[kubernetes-prow]

Hi @Automaat. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[robscott]

Thanks @Automaat! This change looks correct to me, but I'll defer to one of the mesh leads to confirm.

/ok-to-test
/approve
/cc @howardjohn @mikemorris @kflynn @LiorLieberman

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Automaat, robscott

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• conformance/OWNERS [robscott]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mikemorris]

I think this change might be somewhat conflating three distinct concerns - config propagation time, statistical variance (which is the stated reason for MaxTestRetries, and an actual logical issue.

Providing an improved allowance for variable config propagation times and respecting maxTimeToConsistency with configurable retry interval would likely be an improvement and reduce flakiness for mesh implementations running conformance (but somewhat pulling double-duty by waiting to retry if config has propagated but the failure was just due to statistical variance, which is likely fine because we don't have a discrete way to confirm programmed config has been applied - I'm not sure if a fix closer to #5040 and #5122 would make more sense or not).

Neither test gates on the route. MakeRequestAndExpectEventuallyConsistentResponse only needs three consecutive 200s, and in the mesh fixtures the parent echo Service already fronts the echo-v1 and echo-v2 pods, so requests succeed with no route at all.

If the test can pass before the route config is applied though, then allowing a longer waiting period to ensure the config is applied doesn't seem to address an actual logical problem in this test case?

[Automaat]

@mikemorris Fair point; my description wasn't clear about this.

There's no signal meaning "config is programmed" for a mesh, so the distribution check covers propagation and variance together, and I don't see a way to split them. It's safe to lean on because it can't pass early, and it requires the exact number of active backends. A pass implies the route is programmed.

On your last question, I think that's two different checks. MakeRequestAndExpectEventuallyConsistentResponse passes with no route, because the parent echo Service already fronts the echo-v1 and echo-v2 pods. The distribution assertion can't. Its budget is, however, long; 10 back-to-back batches take under two seconds since #4740, and that isn't written down or configurable.

I looked at #5040 and #5122 first. Those gate on a signal that means ready. The closest mesh analogue is route status, so I measured it on Kuma (local k3d, apply to weights live in the sidecar):

route	Accepted	programmed	gap
GRPCRoute	1.23s	2.59s	1.36s
GRPCRoute	0.68s	1.90s	1.22s
HTTPRoute	0.48s	1.62s	1.14s
HTTPRoute	0.15s	1.45s	1.30s

Accepted lands ~1.2s before the dataplane has the config, more than the whole current budget, so a status gate alone wouldn't fix this. That's the gap you meant about confirming applied config. Happy to add one if you want the shape to match, but it wouldn't change flakiness.

Across 13 runs: HTTPRoute 0.31-1.81s, GRPCRoute 0.18-2.03s. Both straddle the current budget.

[youngnick]

I'm increasingly feeling like the best approach here would be to split all of the weighted tests into two phases:

• wait until you get a successful response from each possible backend, subject to retry intervals, max retries, and overall timeouts. This could be by just probing the implementation, or maybe something more clever like using the backend pods to connect to themselves - that won't succeed until they are up and running.
• Once you can confirm that all the backends are up and receiving traffic, then run the weighting test.

That splits the convergence time from the functionality that's actually being tested - does the configured weight actually take effect?

[Automaat]

@youngnick this wouldn't solve the actual issue I have since echo backends are available. The issue is the propagation time from applying policy on cp to it actually taking effect on dataplanes. There are multiple steps that need to be synced and computed in the meantime, before acutal config is propagated to dataplane

[youngnick]

Ah, I see now.

I agree that this will help with the problem you describe then.