tests: Fixes to run E2E suites in parallel - #144
Merged
Conversation
indradhanush
force-pushed
the
dg/e2e-parallel-and-upgrade-fixes
branch
from
July 20, 2026 12:45
1d6adb6 to
f6d3720
Compare
indradhanush
force-pushed
the
dg/e2e-parallel-and-upgrade-fixes
branch
from
July 20, 2026 13:32
f6d3720 to
d63282a
Compare
indradhanush
force-pushed
the
dg/e2e-parallel-and-upgrade-fixes
branch
from
July 20, 2026 13:34
d63282a to
54ba71a
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
indradhanush
force-pushed
the
dg/e2e-parallel-and-upgrade-fixes
branch
from
July 20, 2026 13:55
54ba71a to
1fea548
Compare
indradhanush
force-pushed
the
dg/e2e-parallel-and-upgrade-fixes
branch
from
July 20, 2026 14:41
00dd888 to
12342d9
Compare
indradhanush
changed the title
indradhanush
changed the title
indradhanush
marked this pull request as ready for review
July 20, 2026 15:43
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
indradhanush
force-pushed
the
dg/e2e-parallel-and-upgrade-fixes
branch
from
July 20, 2026 17:38
12342d9 to
891a1da
Compare
indradhanush
force-pushed
the
dg/e2e-parallel-and-upgrade-fixes
branch
from
July 31, 2026 16:01
891a1da to
37e2e21
Compare
sebastian-pf9
previously approved these changes
Aug 5, 2026
sebastian-pf9
left a comment
sebastian-pf9
left a comment
Collaborator
There was a problem hiding this comment.
approve to unblock
indradhanush
force-pushed
the
dg/e2e-parallel-and-upgrade-fixes
branch
from
August 7, 2026 16:03
37e2e21 to
cb792da
Compare
indradhanush
force-pushed
the
dg/e2e-parallel-and-upgrade-fixes
branch
from
August 11, 2026 13:31
cb792da to
55c5d1d
Compare
indradhanush
force-pushed
the
dg/e2e-parallel-and-upgrade-fixes
branch
from
August 11, 2026 13:58
55c5d1d to
48f5912
Compare
Fix two concurrency bugs under Ginkgo's in-process parallelism (GINKGO_NODES>1). 1. Setup routine's IPC payload omitted two critical environment variables; N-1 of N parallel processes got empty strings, triggering nil-pointer panics. Observed: 6 of 7 processes panicking on each run. 2. Control-plane endpoint IP hardcoded per-cluster; multiple concurrent specs' clusters collided on the same IP, causing apiserver TLS certs to duplicate the contested IP and flooding logs with certificate authority errors. Observed: 651 errors in one run. Both fixes are isolated to test-harness code and covered by table-driven unit tests using TDD. Verified: panics 6/7 → 0/7; cert errors 651 → 0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The generateBootstrapKubeconfig helper function was using Gomega's implicit
default timeout (~1 second) when waiting for a BootstrapKubeconfig controller
to reconcile and populate status. This timeout is far too tight for a real
Kubernetes controller reconcile against a real API server, especially under
resource contention on GitHub-hosted runners (4 vCPUs).
This bug was pre-existing but masked by a concurrent Ginkgo state-propagation
bug (fixed earlier) that crashed processes before reaching this Eventually call.
Once that crash was fixed, execution began reaching this line under real load,
immediately exposing the timeout: CI runs started failing with "Timed out after
1.074s" on specs that had passed consistently in prior validation runs.
Fix: pass e2eConfig.GetIntervals("", "wait-controllers") as the timeout and
polling-interval arguments to the Eventually call. This resolves to the 3-minute
timeout and 10-second poll interval already defined in test/e2e/config/provider.yaml
and already used elsewhere in the codebase for "wait for controller reconcile"
style waits. The empty spec-name string deliberately falls through to the
default wait-controllers config entry, which is the correct fallback since this
helper is called from all 7 e2e spec files with different spec names.
The "Cluster upgrade test" and "Clusterclass upgrade test" specs upgrade from a source to a target Kubernetes version. Both specs were hardcoded to v1.25.11 → v1.26.6, but publish bundles for only v1.31.0 and newer. Bumped both to v1.31.0 → v1.31.2 with matching etcdUpgradeVersion and coreDNSUpgradeVersion. However, this project's vendored CAPI v1.4.4 cannot decode the v1beta4 kubeadm API written by v1.31+ bundles. No currently-published bundle version can complete an upgrade against this CAPI version — upgrading CAPI itself is required as a separate change. Both specs now call Fail(...) as the first statement to signal this blocker clearly and fail fast instead of timing out after 10+ minutes. Additionally, the shared dumpSpecResourcesAndCleanup helper (used by all 7 e2e specs in AfterEach) unconditionally dereferenced a *clusterv1.Cluster pointer that is nil whenever a spec fails before calling ApplyClusterTemplateAndWait — exactly this new fail-fast scenario, but a real latent crash risk for any spec failing early. Added a nil guard around cluster-specific log-dumping calls and rewrote one log line to use the always-available namespace.Name instead of cluster fields.
indradhanush
force-pushed
the
dg/e2e-parallel-and-upgrade-fixes
branch
from
August 11, 2026 14:23
48f5912 to
0ff6fc5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently we only ever ran the PR-Blocking and the ClusterClass suites in E2E. There are 5 other suites that we should run, but it will take ~5x the time. To avoid this, run all E2E suites but use
GINKGO_NODES=7to run them in parallel.This does mean that the E2E CI suite will now start failing, because we never fixed the other 5 suites. E2E CI passed because we only were focused on the other 2 suites instead (which we fixed previously).