Skip to content

fix(e2e): poll for ovs-ovn pod readiness instead of single snapshot check#6364

Merged
oilbeater merged 1 commit intomasterfrom
fix/e2e-ovs-ovn-pod-recreate-race
Feb 27, 2026
Merged

fix(e2e): poll for ovs-ovn pod readiness instead of single snapshot check#6364
oilbeater merged 1 commit intomasterfrom
fix/e2e-ovs-ovn-pod-recreate-race

Conversation

@oilbeater
Copy link
Copy Markdown
Collaborator

Summary

  • Fix flaky "Recreating ovs-ovn pod" e2e test caused by RolloutStatus returning with stale DaemonSet status
  • Replace single-snapshot GetPods + ExpectEqual assertion with WaitUntil polling that waits for the replacement pod to be Running and Ready

Root Cause

After deleting an ovs-ovn pod, RolloutStatus checks DaemonSet.Status.NumberReady via DaemonSetStatusViewer. When the DaemonSet controller hasn't yet updated the status counters to reflect the deleted pod, RolloutStatus reads stale "all ready" status and returns immediately. The subsequent GetPods then finds the replacement pod still in Pending state, causing the assertion to fail.

Test plan

  • Verify the "Recreating ovs-ovn pod" disaster recovery e2e test passes consistently
  • Confirm no regressions in other disaster recovery tests

🤖 Generated with Claude Code

…heck

RolloutStatus may return immediately with stale DaemonSet status when
the controller hasn't yet updated NumberReady to reflect the deleted
pod, causing the subsequent GetPods to find the replacement pod still
in Pending state.

Replace the single-snapshot GetPods + ExpectEqual assertion with
WaitUntil polling that waits for the pod on the target node to be
Running and Ready.

Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
@dosubot dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Feb 27, 2026
@gemini-code-assist
Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a persistent flakiness issue in the ovs-ovn pod recreation e2e test. The previous implementation was prone to failure due to race conditions where the DaemonSet status was not immediately updated after a pod deletion, causing the test to check for pod readiness too early. The core change involves transitioning from a single, immediate check for pod status to a robust polling mechanism, ensuring the test accurately reflects the actual state of the ovs-ovn pod.

Highlights

  • Flaky E2E Test Fix: Addressed the flakiness in the "Recreating ovs-ovn pod" e2e test, which was caused by stale DaemonSet status leading to premature assertions.
  • Polling Mechanism Implemented: Replaced the single-snapshot GetPods and ExpectEqual assertion with a WaitUntil polling loop to reliably wait for the replacement ovs-ovn pod to be in a Running and Ready state.
Changelog
  • test/e2e/connectivity/e2e_test.go
    • Replaced a static pod status check with a dynamic polling mechanism to wait for the ovs-ovn pod to become ready.
Activity
  • The pull request description indicates it was generated with Claude Code.
  • No human comments or reviews have been provided in the context.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@dosubot dosubot bot added the test automation tests label Feb 27, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a flaky e2e test by replacing a single snapshot check for pod readiness with a more robust polling mechanism using framework.WaitUntil. This change correctly handles the eventual consistency of pod statuses in Kubernetes and should resolve the flakiness. The implementation looks good. I have one suggestion to improve code readability within the polling logic.

Comment on lines +264 to +272
for i := range pods.Items {
if pods.Items[i].Spec.NodeName == suiteCtx.Node &&
pods.Items[i].Status.Phase == corev1.PodRunning &&
len(pods.Items[i].Status.ContainerStatuses) != 0 &&
pods.Items[i].Status.ContainerStatuses[0].Ready {
pod = &pods.Items[i]
return true, nil
}
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The if condition with multiple && operators is a bit long and can be hard to read. To improve readability and maintainability, you could refactor this to first check for the pod on the correct node, and then, in a nested block, check for its readiness. This separation of concerns makes the logic clearer.

            for i := range pods.Items {
                p := &pods.Items[i]
                if p.Spec.NodeName != suiteCtx.Node {
                    continue
                }

                if p.Status.Phase == corev1.PodRunning &&
                    len(p.Status.ContainerStatuses) != 0 &&
                    p.Status.ContainerStatuses[0].Ready {
                    pod = p
                    return true, nil
                }
            }

@oilbeater oilbeater merged commit cf48799 into master Feb 27, 2026
73 of 76 checks passed
@oilbeater oilbeater deleted the fix/e2e-ovs-ovn-pod-recreate-race branch February 27, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M This PR changes 30-99 lines, ignoring generated files. test automation tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant