fix(e2e): wait for endpoints and retry service trace in kubectl-ko test

[oilbeater]

Summary

• The "kubectl ko trace ..." should work with network policy test flakes because ovn-trace is executed against a Service ClusterIP before the OVN load balancer rules are fully programmed
• The test only waited for the ClusterIP to be assigned, but the async chain (EndpointSlice creation → kube-ovn LB VIP sync → OVN northd propagation) may not complete in time
• Without LB rules, the trace routes the packet via default routing to a node instead of DNAT-ing to the target pod

Root Cause

When the Service is created, serviceClient.CreateSync only waits for len(s.Spec.ClusterIPs) != 0, which is assigned synchronously by kube-apiserver. However, OVN LB rules require three async steps:

1. Kubernetes EndpointSlice controller detects Service + matching pod → creates EndpointSlice
2. kube-ovn controller detects EndpointSlice → calls LoadBalancerAddVip() to write OVN NB LB rules
3. OVN northd processes LB rules → updates southbound DB

If ovn-trace runs before step 3 completes, the Service ClusterIP is not DNAT'd and the packet follows the default route to the node.

Fix

Two layers of protection:

1. Wait for Endpoints readiness: After Service creation, poll until the Endpoints object has at least one ready address (consistent with the pattern used in switch_lb_rule tests)
2. Retry Service ClusterIP trace: Wrap the Service trace in framework.WaitUntil with a 30s timeout to cover the window between Endpoints readiness and OVN LB rule propagation

Test plan

• Verify the kubectl-ko e2e test passes consistently in CI
• Confirm no new lint issues (make lint passes with 0 issues)

🤖 Generated with Claude Code

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a flakiness issue in the kubectl ko trace e2e test, specifically when testing network policies with Service ClusterIPs. The root cause was identified as a race condition where ovn-trace was executed before OVN load balancer rules were fully propagated, leading to incorrect trace results. The changes introduce more robust waiting mechanisms, first for Kubernetes Endpoints readiness and then for the ovn-trace command itself, to ensure the OVN network state is stable before verification.

Highlights

• Enhanced Service Readiness Check: Implemented a polling mechanism to wait for a Service's Endpoints object to have at least one ready address, addressing a timing issue where OVN load balancer rules might not be fully programmed.
• Service Trace Retry Logic: Introduced a retry loop with a 30-second timeout for kubectl ko trace commands targeting Service ClusterIPs, ensuring that OVN LB rule propagation has sufficient time to complete before the trace is evaluated.
• Refactored Output Verification: Extracted the logic for checking ovn-trace command output into a dedicated helper function for improved readability and maintainability.

Changelog

• test/e2e/kube-ovn/kubectl-ko/kubectl-ko.go
  • Imported time and k8serrors packages.
  • Added a framework.WaitUntil block to poll for Service Endpoints readiness, ensuring at least one ready address.
  • Introduced a new helper function checkOutput to encapsulate the logic for verifying ovn-trace command output.
  • Replaced direct framework.ExpectContainSubstring calls with the new checkOutput function within the checkFunc.
  • Wrapped the kubectl ko trace execution for Service ClusterIPs in a framework.WaitUntil loop, retrying the trace for up to 30 seconds until the expected output is observed.

Activity

• The pull request was created to fix a flaky e2e test related to kubectl ko trace and network policies.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The pull request effectively addresses the flakiness in the kubectl ko trace e2e test by implementing two key improvements. First, it introduces a wait condition to ensure that service endpoints are ready before proceeding with the trace. Second, it wraps the service ClusterIP trace execution in a retry mechanism with a 30-second timeout, accounting for potential delays in OVN LB rule propagation. The refactoring of the output checking logic into a dedicated checkOutput function also improves code readability and maintainability. These changes directly tackle the root cause of the test instability, making the e2e tests more robust.