Skip to content

Make throttling on tlog disk space more gradual #12807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tclinkenbeard-oai wants to merge 3 commits into
base: main
Choose a base branch
from
+34 −13
Open

Make throttling on tlog disk space more gradual #12807

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6aaa40b
Create TLOG_THROTTLE_START_AVAILABLE_SPACE_RATIO knob
tclinkenbeard-oai Mar 18, 2026
036ebd5
Add comments
tclinkenbeard-oai Mar 18, 2026
2b70f7e
Simplify clamping
tclinkenbeard-oai Mar 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions fdbclient/ServerKnobs.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -929,6 +929,8 @@ void ServerKnobs::initialize(Randomize randomize, ClientKnobs* clientKnobs, IsSi
init( SPRING_BYTES_TLOG, 400e6 ); if( smallTlogTarget ) SPRING_BYTES_TLOG = 200e3;
init( TARGET_BYTES_PER_TLOG_BATCH, 1400e6 ); if( smallTlogTarget ) TARGET_BYTES_PER_TLOG_BATCH = 1400e3;
init( SPRING_BYTES_TLOG_BATCH, 300e6 ); if( smallTlogTarget ) SPRING_BYTES_TLOG_BATCH = 150e3;
// Match MIN_AVAILABLE_SPACE by default; buggified simulations exercise the earlier ramp.
init( TLOG_THROTTLE_START_AVAILABLE_SPACE_RATIO, 0.05 ); if( randomize && isSimulated && BUGGIFY ) TLOG_THROTTLE_START_AVAILABLE_SPACE_RATIO = 0.20;
Copy link

@gridem-openai gridem-openai Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to have START and STOP as configurable params?

Copy link
Collaborator Author

@tclinkenbeard-oai tclinkenbeard-oai Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

STOP is effectively MIN_AVAILABLE_SPACE_RATIO, there isn't a separate knob for this yet, but I don't think we need one

init( TLOG_SPILL_THRESHOLD, 1500e6 ); if( smallTlogTarget ) TLOG_SPILL_THRESHOLD = 1500e3; if( randomize && BUGGIFY ) TLOG_SPILL_THRESHOLD = 0;
init( REFERENCE_SPILL_UPDATE_STORAGE_BYTE_LIMIT, 20e6 ); if( (randomize && BUGGIFY) || smallTlogTarget ) REFERENCE_SPILL_UPDATE_STORAGE_BYTE_LIMIT = 1e6;
init( TLOG_HARD_LIMIT_BYTES, 3000e6 ); if( smallTlogTarget ) TLOG_HARD_LIMIT_BYTES = 30e6;
Expand Down
1 change: 1 addition & 0 deletions fdbclient/include/fdbclient/ServerKnobs.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -956,6 +956,7 @@ class SWIFT_CXX_IMMORTAL_SINGLETON_TYPE ServerKnobs : public KnobsImpl<ServerKno
int64_t SPRING_BYTES_TLOG;
int64_t TARGET_BYTES_PER_TLOG_BATCH;
int64_t SPRING_BYTES_TLOG_BATCH;
double TLOG_THROTTLE_START_AVAILABLE_SPACE_RATIO;
int64_t TLOG_SPILL_THRESHOLD;
int64_t TLOG_HARD_LIMIT_BYTES;
int64_t TLOG_RECOVER_MEMORY_LIMIT;
Expand Down
44 changes: 31 additions & 13 deletions fdbserver/ratekeeper/Ratekeeper.actor.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
* limitations under the License.
*/

#include <algorithm>

#include "fdbclient/ClientKnobs.h"
#include "fdbserver/core/Knobs.h"
#include "fdbserver/core/ServerDBInfo.h"
Expand Down Expand Up @@ -978,17 +980,31 @@ void Ratekeeper::updateRate(RatekeeperLimits* limits) {

limitReason_t tlogLimitReason = limitReason_t::log_server_write_queue;

int64_t minFreeSpace = std::max(SERVER_KNOBS->MIN_AVAILABLE_SPACE,
(int64_t)(SERVER_KNOBS->MIN_AVAILABLE_SPACE_RATIO * tl.getSmoothTotalSpace()));

worstFreeSpaceTLog =
std::min(worstFreeSpaceTLog, std::max((int64_t)tl.getSmoothFreeSpace() - minFreeSpace, (int64_t)0));

int64_t springBytes = std::max<int64_t>(
1, std::min<int64_t>(limits->logSpringBytes, (tl.getSmoothFreeSpace() - minFreeSpace) * 0.2));
int64_t targetBytes =
std::max<int64_t>(1, std::min(limits->logTargetBytes, (int64_t)tl.getSmoothFreeSpace() - minFreeSpace));
if (targetBytes != limits->logTargetBytes) {
const auto smoothTotalSpace = tl.getSmoothTotalSpace();
const auto smoothFreeSpace = tl.getSmoothFreeSpace();
const auto minFreeSpaceByRatio =
static_cast<int64_t>(SERVER_KNOBS->MIN_AVAILABLE_SPACE_RATIO * smoothTotalSpace);
const auto minFreeSpace = std::max(SERVER_KNOBS->MIN_AVAILABLE_SPACE, minFreeSpaceByRatio);
// Start shrinking the tlog queue budget once available space drops below the configured ratio,
// then ramp down linearly until reaching minFreeSpace.
const auto throttleStartSpaceByRatio =
static_cast<int64_t>(SERVER_KNOBS->TLOG_THROTTLE_START_AVAILABLE_SPACE_RATIO * smoothTotalSpace);
const auto throttleStartSpace = std::max(minFreeSpace + 1, throttleStartSpaceByRatio);
const auto availableAboveMin = smoothFreeSpace - minFreeSpace;
const auto availableAboveMinBytes = static_cast<int64_t>(availableAboveMin);
const auto throttleWindow = static_cast<double>(throttleStartSpace - minFreeSpace);
const auto diskBudgetRatio = std::clamp(availableAboveMin / throttleWindow, 0.0, 1.0);

worstFreeSpaceTLog = std::min(worstFreeSpaceTLog, std::max(availableAboveMinBytes, int64_t{ 0 }));

const auto scaledSpringBytes = static_cast<int64_t>(limits->logSpringBytes * diskBudgetRatio);
const auto springBytes =
std::max(int64_t{ 1 }, std::min(std::max(int64_t{ 1 }, scaledSpringBytes), availableAboveMinBytes));
const auto scaledTargetBytes = static_cast<int64_t>(limits->logTargetBytes * diskBudgetRatio);
const auto targetBytes =
std::max(int64_t{ 1 }, std::min(std::max(int64_t{ 1 }, scaledTargetBytes), availableAboveMinBytes));
if (diskBudgetRatio < 1.0) {
CODE_PROBE(true, "Ratekeeper tlog disk budget ratio below one");
if (minFreeSpace == SERVER_KNOBS->MIN_AVAILABLE_SPACE) {
tlogLimitReason = limitReason_t::log_server_min_free_space;
} else {
Expand All @@ -998,9 +1014,11 @@ void Ratekeeper::updateRate(RatekeeperLimits* limits) {
TraceEvent("RatekeeperLimitReasonDetails")
.detail("TLogID", tl.id)
.detail("Reason", tlogLimitReason)
.detail("TLSmoothFreeSpace", tl.getSmoothFreeSpace())
.detail("TLSmoothTotalSpace", tl.getSmoothTotalSpace())
.detail("TLSmoothFreeSpace", smoothFreeSpace)
.detail("TLSmoothTotalSpace", smoothTotalSpace)
.detail("LimitsLogTargetBytes", limits->logTargetBytes)
.detail("ThrottleStartSpace", throttleStartSpace)
.detail("DiskBudgetRatio", diskBudgetRatio)
.detail("TargetBytes", targetBytes)
.detail("MinFreeSpace", minFreeSpace);
}
Expand Down