Guard Resolver Private Mutations Work #12498

dlambrig · 2025-10-22T22:31:06Z

On the resolver, only construct ResolverData, read databaseLockedKey, and invoke applyMetadataMutations when the batch actually contains state transactions. Keep the LogPushData object alive in each batch so the commit proxy still receives the metadata replay.

100K USE_RESOLVER_PRIVATE_MUTATIONS enabled
20251023-171528-dlambrig-4eb9e1acb2dc2864
20251103-205727-dlambrig-5ffe617bc6aa1d72 (11-3-25)

100K USE_RESOLVER_PRIVATE_MUTATIONS disabled
20251023-171420-dlambrig-735d7123309317ab
20251103-160029-dlambrig-7dd92c5eee5f38c2 (11-3-25)

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2025-10-22T22:56:18Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 64d056f
Duration 0:25:00
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T23:15:58Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 64d056f
Duration 0:43:45
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T23:18:10Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 64d056f
Duration 0:46:49
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T23:20:35Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 64d056f
Duration 0:49:19
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T23:26:12Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 64d056f
Duration 0:54:53
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-22T23:40:12Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 64d056f
Duration 1:08:56
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-23T16:16:28Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 7b6009b
Duration 0:12:28
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T16:29:14Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 7b6009b
Duration 0:25:14
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T16:40:43Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 7b6009b
Duration 0:36:46
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T16:46:48Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 7b6009b
Duration 0:42:50
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T16:47:07Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 7b6009b
Duration 0:43:08
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T16:50:48Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 7b6009b
Duration 0:46:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T16:57:31Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: b4292b9
Duration 0:29:20
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T17:09:30Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: b4292b9
Duration 0:41:21
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T17:13:29Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 7b6009b
Duration 1:09:29
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-23T17:14:13Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: b4292b9
Duration 0:46:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T17:21:54Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: b4292b9
Duration 0:53:44
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

This reverts commit 505e0cb.

foundationdb-ci · 2025-10-23T17:38:15Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: b4292b9
Duration 1:10:06
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-23T17:59:44Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 505e0cb
Duration 0:33:30
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T18:09:28Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 4ff89fe
Duration 0:39:05
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T18:10:10Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 505e0cb
Duration 0:43:51
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T18:12:23Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 505e0cb
Duration 0:46:07
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T18:15:52Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 4ff89fe
Duration 0:45:34
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T18:49:17Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 53713c2
Duration 1:12:33
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-23T18:49:56Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 53713c2
Duration 1:13:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

fdbserver/Resolver.actor.cpp

sbodagala · 2025-10-24T20:44:09Z

fdbserver/Resolver.actor.cpp

 			// merge mutation tags with sent client tags
 			toCommit->saveTags(reply.writtenTags);
-			reply.privateMutationCount = toCommit->getMutationCount();
+			reply.privateMutationCount = applyResolverPrivateMutations ? toCommit->getMutationCount() : 0;


Same question: why checking for "applyResolverPrivateMutations" here?

why checking for "applyResolverPrivateMutations" here?

privateMutationCount should only be non-zero when we actually applied resolver private mutations for this batch. Gating with applyResolverPrivateMutations keeps this consistent with the nearby haveLogsChanged() checks. i.e. "no state txns ⇒ count must be 0.”

privateMutationCount should only be non-zero when we actually applied resolver private mutations for this batch.

Probably we can add an assert and run a Joshua job to validate this (I don't fully understand this code so I can't say for sure this is true).

private mutations are only generated when we call applyMetadataMutations() above, which is conditioned on req.txnStateTransactions.

shouldApplyResolverPrivateMutations = SERVER_KNOBS->PROXY_USE_RESOLVER_PRIVATE_MUTATIONS && !req.txnStateTransactions.empty(); So the change is correct. But honestly I don't know if this is an optimization, toCommit->getMutationCount() is very fast anyway,

uint32_t getMutationCount() const { return subsequence - 1; }

honestly I don't know if this is an optimization

its not an optimization; we cannot deref toCommit here unless applyResolverPrivateMutations is true

fdbserver/Resolver.actor.cpp

foundationdb-ci · 2025-10-27T22:35:31Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 847cd99
Duration 0:12:25
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-27T22:46:41Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 847cd99
Duration 0:23:34
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-27T23:07:45Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 847cd99
Duration 0:44:40
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-27T23:08:24Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 847cd99
Duration 0:45:18
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-27T23:11:37Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 847cd99
Duration 0:48:36
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-27T23:12:06Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 847cd99
Duration 0:49:03
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-27T23:32:33Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 847cd99
Duration 1:09:28
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

fdbserver/Resolver.actor.cpp

foundationdb-ci · 2025-10-31T15:29:14Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 90d2222
Duration 0:12:09
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-31T15:40:46Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 90d2222
Duration 0:23:40
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-31T15:47:51Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 90d2222
Duration 0:30:43
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /opt/homebrew/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-31T15:56:45Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 90d2222
Duration 0:39:39
Result: ❌ FAILED
Error: Error while executing command: docker tag ${REGISTRY}/rockylinux/rockylinux:9.6 rockylinux/rockylinux:9.6-minimal. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-31T16:01:43Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 90d2222
Duration 0:44:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-31T16:11:19Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 90d2222
Duration 0:54:13
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-31T16:18:34Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 90d2222
Duration 1:01:26
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Guard Resolver Private Mutations Work

64d056f

Optimize Resolver private mutations handling

b4292b9

dlambrig force-pushed the resolver-private-mutations-fastpath branch from 7b6009b to b4292b9 Compare October 23, 2025 16:27

Revert to 64d056f

bda5049

Dan Lambright added 2 commits October 23, 2025 13:25

revert previous change

505e0cb

Revert "revert previous change"

53713c2

This reverts commit 505e0cb.

dlambrig force-pushed the resolver-private-mutations-fastpath branch from 4ff89fe to 53713c2 Compare October 23, 2025 17:36

dlambrig marked this pull request as ready for review October 23, 2025 19:22

dlambrig requested review from jzhou77 and sbodagala October 23, 2025 19:22

sbodagala reviewed Oct 24, 2025

View reviewed changes

fdbserver/Resolver.actor.cpp Show resolved Hide resolved

sbodagala reviewed Oct 24, 2025

View reviewed changes

fdbserver/Resolver.actor.cpp Show resolved Hide resolved

sbodagala reviewed Oct 24, 2025

View reviewed changes

fdbserver/Resolver.actor.cpp Outdated Show resolved Hide resolved

sbodagala reviewed Oct 24, 2025

View reviewed changes

fdbserver/Resolver.actor.cpp Outdated Show resolved Hide resolved

Respond to review comments

847cd99

dlambrig requested a review from sbodagala October 28, 2025 11:57

This comment was marked as outdated.

Sign in to view

sbodagala reviewed Oct 28, 2025

View reviewed changes

fdbserver/Resolver.actor.cpp Outdated Show resolved Hide resolved

Respond to review comments.

90d2222

jzhou77 approved these changes Oct 31, 2025

View reviewed changes

Guard Resolver Private Mutations Work #12498

Are you sure you want to change the base?

Guard Resolver Private Mutations Work #12498

Uh oh!

Conversation

dlambrig commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code-Reviewer Section

For Release-Branches

Uh oh!

foundationdb-ci commented Oct 22, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 22, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 22, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 22, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 22, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 22, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 23, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 23, 2025

dlambrig commented Oct 22, 2025 •

edited

Loading