Add commit proxy long-running commit request tracing

[kakaiu]

We aim to understand the root cause of peak commit latency in the cluster—a challenging problem. At first, we want to identify which components are contributing most to the performance degradation, such as the network, CPU, tLogs, or resolvers.

This PR addresses this question by tracing long-running requests (those exceeding 0.5 seconds). Specifically, when a commit request arrives at a commit proxy (CP), the CP records the latency for each step in the request's processing. If the total serving time of the request exceeds the threshold, the CP tracks it accordingly.

The key idea of the role based approach is that: each role provides a "service". Those roles work together to provide some functionality like transaction commit. A commit slow down eventually can be explained by slow down of some services or the network.

Notably, we do not attempt to log every individual long-running request, as the volume of such requests can be substantial. Instead, our focus is on identifying which commit step is the "bottleneck".

Bottleneck Definition:
For a given set of long-running commit requests, a step is the bottleneck if and only if its removal would yield the greatest reduction in the total time span of those requests.

To identify the bottleneck step, the CP aggregates the latency of each step across all slow requests and generates a trace event every 10 seconds. The aggregation algorithm for a step's time span simply sums the durations of that step across all long-running requests observed in the past 10 seconds. The aggregated time span represents the cumulative time that the step was actively executing, reflecting its probability of blocking other steps within that period.

To minimize the overhead associated with latency tracking at each step, we leverage shared_ptr to avoid unnecessary memory copies and employ inline functions to eliminate function call overhead. We avoid passing any metrics over the network to save bandwidth.

Why don't we sum up all requests in the last 10 seconds?
Here, we assume that there is a "correlation" between long-running commits and specific commit steps. By filtering out fast commits, we effectively remove noise introduced by their shorter durations. For example, if the majority of commits in the last 10 seconds are completed quickly while only a few are long-running, aggregating all commit steps—including the fast ones—can obscure the visibility of the slow steps. This is because the cumulative time of fast commits can overwhelm the contribution of the long-running steps, masking potential bottlenecks.

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

• The PR has a description, explaining both the problem and the solution.
• The description mentions which forms of testing were done and the testing seems reasonable.
• Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

• This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
• There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux RHEL 9

• Commit ID: 6053204
• Duration 0:32:05
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux RHEL 9

• Commit ID: a8284c6
• Duration 0:33:49
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: a8284c6
• Duration 0:38:29
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 6053204
• Duration 0:48:16
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: a8284c6
• Duration 0:51:04
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: a8284c6
• Duration 1:01:14
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 6053204
• Duration 1:03:10
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: a8284c6
• Duration 1:05:51
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: a8284c6
• Duration 1:22:02
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 6053204
• Duration 1:21:15
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 6053204
• Duration 1:26:50
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: a8284c6
• Duration 1:32:56
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)