Coordinate Attrition targets within a test phase#13742
Draft
tclinkenbeard-oai wants to merge 1 commit into
Draft
Coordinate Attrition targets within a test phase#13742tclinkenbeard-oai wants to merge 1 commit into
tclinkenbeard-oai wants to merge 1 commit into
Conversation
Contributor
Result of foundationdb-pr-clang-ide on Linux RHEL 9
|
Contributor
Result of foundationdb-pr-clang-arm on Linux RHEL 9
|
Contributor
Result of foundationdb-pr-macos-m1 on macOS 14.x
|
Contributor
Result of foundationdb-pr-clang on Linux RHEL 9
|
Contributor
Result of foundationdb-pr on Linux RHEL 9
|
Contributor
Result of foundationdb-pr-macos on macOS 14.x
|
Contributor
Result of foundationdb-pr-cluster-tests on Linux RHEL 9
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Coordinate machine Attrition targets within a test phase so concurrent Attrition workloads share the strongest
machinesToLeavefloor and cannot claim the same zone. Recheck immediately before killing, while allowing a replacement workload to revisit its own zone, and add a focused lifecycle/floor/replacement regression.Failure evidence
Managed correctness job failed
tests/slow/SwizzledDdBalance.tomlonce in 190,859 runs with RandomSeed1023295228, JoshuaSeed978392406014025441, buggify and fault injection enabled. Immutable artifact:The exact artifact replay reproduced the timeout and 1,000,001-line trace overflow. Three explicit Attrition workloads independently targeted all five zones despite two declaring
machinesToLeave=3, leaving recovery unavailable whileChangeConfigattempted a coordinator move. This is a harness availability failure, distinct from the earlier old-TLog QuietDatabase assertion family.Validation
Managed clang
-Werrorcompile/link and the focused/fdbserver/workloads/MachineAttrition/phaseCoordinationregression passed on both integrated revisione86c1a7e64346cd7aa4d9e445189cac79b7ccad3and upstream base1ae8e7f2383bbd79aa860940507db6ec0754055d.Exact-seed patched replays passed on both binaries. Trace audits show only two unique killed zones (preserving the three-zone floor), successful
ChangeConfigcompletion, and zeroTestFailure,TestUnexpectedlyNotFinished, orTracedTooManyLinesevents. Additional managed smoke runs passed:slow/SwizzledDdBalance.toml --seed 1023295229 --buggify on -fi onfast/StatusDuringOutage.toml --seed 2456305 --buggify on -fi onfast/CycleTest.toml --seed 4101 --buggify on -fi onThe managed build wrapper reported a post-link nonzero status because the worker image lacks
sccache; compilation/link, binary execution, source-hash identity, and focused tests were verified explicitly.