test(control-plane): de-flake auth-link integration tests#1829
Merged
Conversation
The auth-link integration tests failed intermittently in CI (a different `test_link_*` test each run, always at the same two assertion sites, and on `main` too) under slow, heavily-parallel jobs such as the llvm-cov coverage run. Two independent causes: 1. Wrong-link race. `auth_link()` matched the first link sourced from *either* the connector or the protected group, but `wait_auth_link()` then asserts the source is the connector. During reconciliation a transient protected-sourced link can reach the target status first, tripping the assertion. Select the connector-sourced link directly so transient links are skipped and the loop keeps polling for the one every caller actually expects. 2. Timeouts too tight under instrumentation. Reconciliation occasionally needs longer than 30s/15s when the runner is saturated (llvm-cov, parallel matrix), causing `wait_*` timeout panics. Double DEFAULT_TIMEOUT (30->60s) and SHORT_TIMEOUT (15->30s); the happy path returns immediately, so passing tests are unaffected. No production code changes; test-only. Signed-off-by: Luca Muscariello <muscariello@ieee.org>
muscariello
force-pushed
the
deflake/control-plane-integration-tests
branch
from
fed688c to
211974d
Compare
…rvation Gate each control-plane integration test through a shared semaphore (2-4 slots, scaled to available parallelism), with the permit held for the test's lifetime via TestControlPlane. Prevents the ~30 node-spinning tests from starving each other of CPU under llvm-cov / matrix CI, which stalled reconciliation past the wait timeouts and caused the rotating flake. Signed-off-by: Luca Muscariello <muscariello@ieee.org>
… flake A persistently-failing link is only marked LINK_FAILED once reconciler requeues exhaust, with exponential backoff (base x 2^n). At max_requeues=10 that's ~51s of cumulative backoff, which exceeds the wait_auth_link timeout under slow llvm-cov runs and caused failure-expecting tests (wrong creds, missing workload API) to flake. Lower to 7 (~6s to fail). Recovery tests are unaffected — they recover via a fresh node registration, not the requeue chain (verified locally). Signed-off-by: Luca Muscariello <muscariello@ieee.org>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
De-flake for the flaky control-plane integration tests (rotating
test_link_*failures under CI load, red on main too). Root cause: ~30 node-spinning tests run under plaincargo testand starve each other, stalling reconciliation past the timeouts.auth_link(): select the connector-sourced link (skips transient reverse links that trip the source assert).TestControlPlane.Contention only reproduces under CI, so needs a couple of
coverage/unittest-matrixruns to confirm.