Add retry mechanism for failing integration tests#1879
Merged
sondavidb merged 2 commits intoMar 3, 2026
Conversation
github-actions
Bot
added
the
github_actions
sondavidb
force-pushed
the
retry-failing-integration-tests
branch
3 times, most recently
from
5a24a34 to
9aed58d
Compare
Before this change we would have to empty the covdir before re-running any integration tests or unit tests with coverage, which made rerunning them in CI difficult. Signed-off-by: David Son <davbson@amazon.com>
sondavidb
force-pushed
the
retry-failing-integration-tests
branch
from
9aed58d to
600de95
Compare
Contributor
Author
|
Looks like failed tests retry: https://github.com/awslabs/soci-snapshotter/actions/runs/22537189541/job/65286713612?pr=1879#step:5:865 |
Contributor
Can we use |
sondavidb
force-pushed
the
retry-failing-integration-tests
branch
from
600de95 to
102045f
Compare
Contributor
|
Test seem to fail. |
Signed-off-by: David Son <davbson@amazon.com>
sondavidb
force-pushed
the
retry-failing-integration-tests
branch
from
102045f to
551542c
Compare
Contributor
Author
Wrong version 🤦 correct one should run them now |
Shubhranshu153
approved these changes
Mar 2, 2026
Contributor
Author
|
Nice, we got a failure that got caught: https://github.com/awslabs/soci-snapshotter/actions/runs/22596642133/job/65468271427?pr=1879 Wish we could do something about these network errors though. Would be nice only to fail on those. |
Swapnanil-Gupta
approved these changes
Mar 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
Our integration tests have become flakier for networking reasons. I decided to add a wrapper script to catch failing tests and retry only the ones that have failed.
This isn't perfect as the
-runflag ingo testtakes regex, so if the failing test name happens to be the prefix for another test, it will run more tests than necessary (e.g.-run TestConvertwould also runTestConvertWithForceRecreateZtocsandTestConvertAndPush). However, this is still better than what we have currently, so I guess it's fine for now.Testing performed:
Ran
./scripts/integration-with-retries 3locally and ensured it passed.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.