Skip to content

Coordinate Attrition targets within a test phase#13742

Draft
tclinkenbeard-oai wants to merge 1 commit into
apple:mainfrom
tclinkenbeard-oai:dev/tclinkenbeard/swizzled-dd-balance-timeout-trace-overflow
Draft

Coordinate Attrition targets within a test phase#13742
tclinkenbeard-oai wants to merge 1 commit into
apple:mainfrom
tclinkenbeard-oai:dev/tclinkenbeard/swizzled-dd-balance-timeout-trace-overflow

Conversation

@tclinkenbeard-oai

@tclinkenbeard-oai tclinkenbeard-oai commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

Coordinate machine Attrition targets within a test phase so concurrent Attrition workloads share the strongest machinesToLeave floor and cannot claim the same zone. Recheck immediately before killing, while allowing a replacement workload to revisit its own zone, and add a focused lifecycle/floor/replacement regression.

Failure evidence

Managed correctness job failed tests/slow/SwizzledDdBalance.toml once in 190,859 runs with RandomSeed 1023295228, JoshuaSeed 978392406014025441, buggify and fault injection enabled. Immutable artifact:

The exact artifact replay reproduced the timeout and 1,000,001-line trace overflow. Three explicit Attrition workloads independently targeted all five zones despite two declaring machinesToLeave=3, leaving recovery unavailable while ChangeConfig attempted a coordinator move. This is a harness availability failure, distinct from the earlier old-TLog QuietDatabase assertion family.

Validation

Managed clang -Werror compile/link and the focused /fdbserver/workloads/MachineAttrition/phaseCoordination regression passed on both integrated revision e86c1a7e64346cd7aa4d9e445189cac79b7ccad3 and upstream base 1ae8e7f2383bbd79aa860940507db6ec0754055d.

Exact-seed patched replays passed on both binaries. Trace audits show only two unique killed zones (preserving the three-zone floor), successful ChangeConfig completion, and zero TestFailure, TestUnexpectedlyNotFinished, or TracedTooManyLines events. Additional managed smoke runs passed:

  • slow/SwizzledDdBalance.toml --seed 1023295229 --buggify on -fi on
  • fast/StatusDuringOutage.toml --seed 2456305 --buggify on -fi on
  • fast/CycleTest.toml --seed 4101 --buggify on -fi on

The managed build wrapper reported a post-link nonzero status because the worker image lacks sccache; compilation/link, binary execution, source-hash identity, and focused tests were verified explicitly.

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 348b22e
  • Duration 0:33:02
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux RHEL 9

  • Commit ID: 348b22e
  • Duration 0:45:50
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS 14.x

  • Commit ID: 348b22e
  • Duration 0:58:01
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 348b22e
  • Duration 1:02:41
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 348b22e
  • Duration 1:06:43
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS 14.x

  • Commit ID: 348b22e
  • Duration 1:28:18
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 348b22e
  • Duration 1:32:28
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants