Re-recruit backup workers to avoid recoveries #12564

jzhou77 · 2025-11-19T18:01:38Z

Similar to #12558, this PR changes how backup workers are monitored and recruited to avoid recoveries. After fully_recovered state, a new actor monitorAndRecruitBackupWorkers takes over the monitoring and re-recruitment for backup workers of the current generation. Note before fully_recovered state, the monitoring is done in the original TagPartitionedLogSystem::onError_internal() actor. Monitoring backup workers for older generations is possible, but complicated by the fact that the number of log routers can be modified via a configuration change. However, this PR should be strictly better in terms of availability, as in most cases backup workers are running at fully_recovered state. Reducing recoveries improves availability.

Details of the implementation:

During normal operation: Backup workers periodically call saveProgress() to save their progress to the database using key backupProgressKeyFor(workerID) with value containing {epoch, version, tag, totalTags}.
When a backup worker fails:
- monitorAndRecruitBackupWorkers() in ClusterController detects the failure
- Calls recruitFailedBackupWorkers() which creates InitializeBackupRequest with:
  - isReplacement = true
  - startVersion = 0 (to be determined by backup worker)
  - backupEpoch and routerTag identifying which worker failed
New backup worker starts:
- Checks if (req.isReplacement && req.startVersion == 0)
- Calls getSavedVersion(cx, workerID, backupEpoch, routerTag)
- Scans all backup progress entries to find matching (epoch, tag)
- Resumes from savedVersion + 1
Refactor log router and backup worker re-recruitment so that they share the same monitorAndRecruitWorkerSet() logic for monitoring and re-recruiting after fully_recovered.
This feature is controlled by knob CC_RERECRUIT_BACKUP_WORKER_ENABLED, default is on for more test coverage.

20260108-005523-jzhou-2753e9ae122ef17e compressed=True data_size=34255231 duration=7047094 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:59:39 sanity=False started=100000 stopped=20260108-015502 submitted=20260108-005523 timeout=5400 username=jzhou

20260108-033820-jzhou-0107822af36074ff compressed=True data_size=34255687 duration=5578744 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:53:28 sanity=False started=100000 stopped=20260108-043148 submitted=20260108-033820 timeout=5400 username=jzho

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2025-11-19T18:26:57Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 06720ba
Duration 0:25:06
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-19T18:40:39Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 06720ba
Duration 0:38:50
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-19T18:48:02Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 06720ba
Duration 0:46:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-19T18:56:20Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 06720ba
Duration 0:54:30
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-19T19:04:02Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 06720ba
Duration 1:02:15
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-19T19:12:25Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 06720ba
Duration 1:10:36
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-11-19T19:16:16Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 06720ba
Duration 1:14:25
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-18T00:27:09Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 4ab570c
Duration 0:23:59
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-18T00:38:05Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 4ab570c
Duration 0:34:54
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-18T00:48:22Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 4ab570c
Duration 0:45:09
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-18T00:51:14Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 4ab570c
Duration 0:48:05
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-18T00:53:08Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 4ab570c
Duration 0:50:01
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-18T00:59:48Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 4ab570c
Duration 0:56:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-12-18T02:03:41Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 4ab570c
Duration 2:00:28
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

The idea is that backup workers are stateless roles that can reconstruct its state after failures. So we don't need to trigger recoveries unnecessarily.

By removing an unused parameter.

Before fully recovered, there are backup workers for the old generations. The APIs for tracking these old generations and updating them are not well defined. To simplify the work, I'll limit the monitoring to the current generation only. Before fully recovered, any backup worker failures will cause another recovery, i.e., old behavior.

During normal operation, backup workers periodically call saveProgress() to save their progress to the database using key backupProgressKeyFor(workerID) with value containing {epoch, version, tag, totalTags}. So for the re-recruited one, let the new worker to take over old worker's progress and use that as its start version. 20251119-040547-jzhou-78a15d307baeed3f

20251119-043027-jzhou-94321348b7ca1c02

It was running infinite loops because no backup workers to monitor. Change so that the function returns Never() in such cases. 20251119-050212-jzhou-c3691b1ed4c330a3

When there is a new recovery, let the recovery process to handle initial backup worker recruitment. 20251119-054841-jzhou-1e367b6d83439c39

There could be a new recovery, thus the worker is popping from a previous epoch. As a result, this assertion doesn't make sense. The pop() is always safe to do because mutations are saved already. 20251119-173704-jzhou-31139c54035a3755

20251218-041558-jzhou-bbf8243bea1e71d0

Backup worker could be hanging around for too long and trigger assertions. 20251218-214722-jzhou-b9547780971cc150

20260105-220908-jzhou-57c53cd7ff2b8c1a compressed=True data_size=34250222 duration=5755330 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:57:34 sanity=False started=100000 stopped=20260105-230642 submitted=20260105-220908 timeout=5400 username=jzhou

foundationdb-ci · 2026-01-05T23:38:14Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 142029b
Duration 0:25:59
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-05T23:57:27Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 142029b
Duration 0:45:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T03:56:53Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 601a868
Duration 0:21:54
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T03:59:51Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 601a868
Duration 0:24:52
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-01-10T04:00:46Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 601a868
Duration 0:25:46
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T04:08:22Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 601a868
Duration 0:33:24
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T04:18:49Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 601a868
Duration 0:43:51
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T04:25:04Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 601a868
Duration 0:50:07
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T04:29:05Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 601a868
Duration 0:54:06
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T04:33:58Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: d8b058d
Duration 0:23:43
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T04:34:48Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: d8b058d
Duration 0:24:37
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-01-10T04:35:10Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: d8b058d
Duration 0:24:59
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T04:43:36Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: d8b058d
Duration 0:33:24
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T04:56:51Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: d8b058d
Duration 0:46:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T04:58:48Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: d8b058d
Duration 0:48:38
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

type has no linkage [-Werror=subobject-linkage]

foundationdb-ci · 2026-01-10T06:16:40Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: ff6f93c
Duration 0:21:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T06:28:01Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: ff6f93c
Duration 0:33:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T06:38:00Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: ff6f93c
Duration 0:43:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T06:44:47Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: ff6f93c
Duration 0:50:01
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T06:45:37Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: ff6f93c
Duration 0:50:49
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T06:48:51Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: ff6f93c
Duration 0:54:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-01-10T07:50:08Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: ff6f93c
Duration 1:55:17
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

spraza · 2026-01-15T18:11:33Z

fdbclient/include/fdbclient/ServerKnobs.h

+	double CC_RERECRUIT_BACKUP_WORKER_TIMEOUT;
+	bool CC_RERECRUIT_BACKUP_WORKER_ENABLED;


Can you add comments describing the semantics of these knobs as well as the knob above (BACKUP_WORKER_REPLACEMENT_GRACE_PERIOD)?

spraza · 2026-01-15T18:12:14Z

fdbclient/ServerKnobs.cpp

 	init( CC_RERECRUIT_LOG_ROUTER_TIMEOUT,                       5.0 );
 	init( CC_RERECRUIT_LOG_ROUTER_ENABLED,                      true ); if ( randomize && BUGGIFY ) CC_RERECRUIT_LOG_ROUTER_ENABLED = deterministicRandom()->coinflip();
+	init( CC_RERECRUIT_BACKUP_WORKER_TIMEOUT,                    5.0 );
+	init( CC_RERECRUIT_BACKUP_WORKER_ENABLED,                   true ); if ( randomize && BUGGIFY ) CC_RERECRUIT_BACKUP_WORKER_ENABLED = deterministicRandom()->coinflip();


I think default of true is ok but just wanted to make sure this is an intentional choice

Yes, it's intentional.

jzhou77 · 2026-01-15T20:37:40Z

Convert to draft and will split this into multiple PRs.

jzhou77 changed the title ~~Rerecruit backup workers to avoid recoveries~~ Re-recruit backup workers to avoid recoveries Nov 19, 2025

jzhou77 force-pushed the rerecruit-backup-worker branch from 06720ba to 4ab570c Compare December 18, 2025 00:03

jzhou77 added 11 commits January 5, 2026 15:11

Add monitorAndRecruitBackupWorkers to CC

8853ca2

The idea is that backup workers are stateless roles that can reconstruct its state after failures. So we don't need to trigger recoveries unnecessarily.

Clean up monitorAndRecruitBackupWorkers()

054ce09

By removing an unused parameter.

Fix an assertion failure

e8f8dcc

20251119-043027-jzhou-94321348b7ca1c02

Fix monitorBackupWorkers when no backups are running

969cd42

It was running infinite loops because no backup workers to monitor. Change so that the function returns Never() in such cases. 20251119-050212-jzhou-c3691b1ed4c330a3

Fix recruitFailedBackupWorkers() to skip recruiting

d112964

When there is a new recovery, let the recovery process to handle initial backup worker recruitment. 20251119-054841-jzhou-1e367b6d83439c39

Remove an assertion in backup worker's pop()

bbb7115

There could be a new recovery, thus the worker is popping from a previous epoch. As a result, this assertion doesn't make sense. The pop() is always safe to do because mutations are saved already. 20251119-173704-jzhou-31139c54035a3755

Add knob CC_RERECRUIT_BACKUP_WORKER_ENABLED to control this feature

9d23cdb

20251218-041558-jzhou-bbf8243bea1e71d0

Exit loop in checkRemoved() when there are new recoveries

74c9994

Backup worker could be hanging around for too long and trigger assertions. 20251218-214722-jzhou-b9547780971cc150

jzhou77 force-pushed the rerecruit-backup-worker branch from 4ab570c to 142029b Compare January 5, 2026 23:12

apple deleted a comment from foundationdb-ci Jan 10, 2026

Merge branch 'main' into rerecruit-backup-worker

d8b058d

Fix gcc compiling error

ff6f93c

type has no linkage [-Werror=subobject-linkage]

spraza reviewed Jan 15, 2026

View reviewed changes

jzhou77 marked this pull request as draft January 15, 2026 20:37

jzhou77 mentioned this pull request Jan 16, 2026

Refactor log router monitoring and re-recruitment #12642

Merged

5 tasks

		double CC_RERECRUIT_BACKUP_WORKER_TIMEOUT;
		bool CC_RERECRUIT_BACKUP_WORKER_ENABLED;

Re-recruit backup workers to avoid recoveries #12564

Are you sure you want to change the base?

Re-recruit backup workers to avoid recoveries #12564

Uh oh!

Conversation

jzhou77 commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code-Reviewer Section

For Release-Branches

Uh oh!

foundationdb-ci commented Nov 19, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 19, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Nov 19, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 19, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Nov 19, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 19, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 19, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Dec 18, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Dec 18, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Dec 18, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Dec 18, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Dec 18, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Dec 18, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Dec 18, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jan 5, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jan 5, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jan 10, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jan 10, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jan 10, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jan 10, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jan 10, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jan 10, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jan 10, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jan 10, 2026

jzhou77 commented Nov 19, 2025 •

edited

Loading