Avoid writing additional data to nearly full tlogs by tclinkenbeard-oai · Pull Request #12809 · apple/foundationdb

tclinkenbeard-oai · 2026-03-19T08:28:36Z

Primary and satellite tlogs rely on ratekeeper to throttle once available disk space drops below MIN_AVAILABLE_SPACE_RATIO (by default 5%). However, remote tlogs don't have a similar safeguard. It makes sense for ratekeeper not to throttle on remote tlog disk space, because remote tlogs shouldn't directly affect availability. However, this can lead to a situation where remote tlogs completely run out of space and are unable to restart in order to serve peek and pop requests from previous log generations. This causes remote storage servers' lag to grow indefinitely.

This PR introduces a TLOG_MIN_AVAILABLE_SPACE_RATIO knob that tlogs can use to protect themselves, without relying on ratekeeper. By default the knob is set to 0, effectively disabling this feature. However, if set, tlogs will stop accepting commits or pulling async data once their available space ratio reaches the configured threshold. If remote tlogs hit this situation, they will lag behind, but they will still be available to serve peek and pop requests from remote storage servers.

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2026-03-19T08:54:34Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 6c723f9
Duration 0:25:47
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-19T09:01:03Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 6c723f9
Duration 0:32:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-19T09:12:02Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 6c723f9
Duration 0:42:19
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-19T09:17:27Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 6c723f9
Duration 0:48:39
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-19T09:24:26Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 6c723f9
Duration 0:55:36
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-19T09:25:44Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 6c723f9
Duration 0:56:54
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-19T10:38:21Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 6c723f9
Duration 2:09:35
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

gridem-openai · 2026-03-19T19:08:25Z

fdbclient/ServerKnobs.cpp

 	init( TLOG_SPILL_THRESHOLD,                               1500e6 ); if( smallTlogTarget ) TLOG_SPILL_THRESHOLD = 1500e3; if( randomize && BUGGIFY ) TLOG_SPILL_THRESHOLD = 0;
 	init( REFERENCE_SPILL_UPDATE_STORAGE_BYTE_LIMIT,            20e6 ); if( (randomize && BUGGIFY) || smallTlogTarget ) REFERENCE_SPILL_UPDATE_STORAGE_BYTE_LIMIT = 1e6;
 	init( TLOG_HARD_LIMIT_BYTES,                              3000e6 ); if( smallTlogTarget ) TLOG_HARD_LIMIT_BYTES = 30e6;
+	init( TLOG_MIN_AVAILABLE_SPACE_RATIO,                        0.0 ); if( randomize && BUGGIFY ) TLOG_MIN_AVAILABLE_SPACE_RATIO = 0.5;


Should we have like 0.2 for our case?

Yeah, I think we'll want to set this knob to something higher. 0.05 should be sufficient, because tlog disk usage will stop growing once the threshold is hit

gridem-openai · 2026-03-19T19:10:40Z

fdbserver/tlog/TLogServer.actor.cpp

+		auto ratio = [](StorageBytes const& storageBytes) -> double {
+			return storageBytes.total > 0 ? double(storageBytes.available) / storageBytes.total : 1.0;
+		};
+		return std::min(ratio(kvStoreBytes), ratio(queueBytes));


Should we have max of (kvStoreBytes, queueBytes), and then apply ratio? I think it would be simpler to understand.

That could be simpler, but in theory kvStoreBytes.total and queueBytes.total could be different. They usually aren't, but there's a valid use case to use different disks for spilling and the disk queue, and the current logic handles that case

gridem-openai · 2026-03-19T19:13:53Z

fdbserver/TLogServer.actor.cpp

+	while (!logData->stopped()) {
+		StorageBytes kvStoreBytes = self->persistentData->getStorageBytes();
+		StorageBytes queueBytes = self->rawPersistentQueue->getStorageBytes();
+		if (self->shouldAcceptNewData(kvStoreBytes, queueBytes, SERVER_KNOBS->TLOG_MIN_AVAILABLE_SPACE_RATIO)) {


Do you think it would be better to introduce hysteresis here - and start accepting if it's > TLOG_MIN_AVAILABLE_SPACE_RATIO.

I pushed some commits to now fail the tlog role when the tlog runs out of space, so there's no coroutine waiting on disk space to recover anymore

…-stop-accepting

foundationdb-ci · 2026-03-19T21:45:37Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: a064b13
Duration 0:31:16
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-19T21:57:33Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: a064b13
Duration 0:43:13
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-19T22:04:58Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: a064b13
Duration 0:50:35
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-19T22:08:50Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: a064b13
Duration 0:54:30
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-19T23:28:41Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: a064b13
Duration 2:14:19
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

tclinkenbeard-oai added 4 commits March 18, 2026 23:22

Add TLOG_MIN_AVAILABLE_SPACE_RATIO knob

8b37839

Extract waitUntilTLogAcceptsNewData helper function

c96b120

Avoid spawning coroutine on happy path in waitUntilTLogAcceptsNewData

845d6d4

Add more comments in pullAsyncData

6c723f9

tclinkenbeard-oai marked this pull request as draft March 19, 2026 08:36

gridem-openai reviewed Mar 19, 2026

View reviewed changes

tclinkenbeard-oai added 2 commits March 19, 2026 19:57

Avoid blocking commits on tlog disk space

7fb9f42

Throw worker_removed when tlog is running out of space in pullAsyncData

bfe9f14

tclinkenbeard-oai marked this pull request as ready for review March 19, 2026 21:10

Merge remote-tracking branch 'apple/main' into dev/tclinkenbeard/tlog…

a064b13

…-stop-accepting

tclinkenbeard-oai requested a review from gridem-openai March 19, 2026 21:18

Conversation

tclinkenbeard-oai commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code-Reviewer Section

For Release-Branches

Uh oh!

foundationdb-ci commented Mar 19, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 19, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 19, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Mar 19, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 19, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 19, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 19, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

gridem-openai Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

tclinkenbeard-oai Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

gridem-openai Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

tclinkenbeard-oai Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

gridem-openai Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

tclinkenbeard-oai Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

foundationdb-ci commented Mar 19, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 19, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Mar 19, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 19, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 19, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tclinkenbeard-oai commented Mar 19, 2026 •

edited

Loading