Coalesce encrypted blobstore backup reads

[arnav-ag]

Problem

Encrypted blobstore backups previously defaulted to 4,096-byte encryption blocks, while the blobstore read cache used 1,048,576-byte blocks. Because the cache wrapped AsyncFileEncrypted, a 1 MiB restore read was split into roughly 256 small S3 requests before reaching the cache.

The current encryption default is 1 MiB, which gives a 1:1 ratio with the default cache block. Existing backups and new backups configured with --encryption-block-size below 1 MiB still encounter the amplification.

Change

Cache ciphertext between AsyncFileEncrypted and AsyncFileBlobStoreRead, allowing adjacent encryption-block reads to share larger object-store range reads. Unencrypted reads, writes, and files under properties/ are unchanged.

Validation

With 4 KiB encryption blocks, we observed approximately 256 times fewer S3 requests. A sample restore improved from 123 minutes to 103 minutes, primarily from removing S3 request overhead. The same result was achieved with default concurrent-read and blobstore-connection settings, removing the tuning previously needed to absorb the inflated request volume.

• git diff --check
• clang-format --dry-run --Werror fdbclient/BackupContainerBlobStore.cpp

[gxglass]

AI review seems fine. @akankshamahajan15 do you want to double check this?

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux RHEL 9

• Commit ID: 8107504
• Duration 0:41:12
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux RHEL 9

• Commit ID: 8107504
• Duration 0:45:47
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 8107504
• Duration 1:24:47
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 8107504
• Duration 1:26:25
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 8107504
• Duration 1:41:18
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: 8107504
• Duration 2:14:16
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: 8107504
• Duration 4:13:40
• Result: ❌ FAILED
• Error: `Error while executing command: # compatible with bash and zsh
  ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key
  -o ServerAliveInterval=9 ec2-user@${MAC_EC2_HOST} "
  set -e -o noclobber
  BUILDNUM=${CODEBUILD_BUILD_NUMBER}
  WAIT_TIME=14400 # 4 hours
  LOCK_STALE=4500 # 75 minutes
  LOCK=/tmp/ci.lock
  "'

macOS stat flags, not linux compatible

lock_mtime() { stat -f %m $LOCK || echo 0 }
I=0
while ! (echo $BUILDNUM >$LOCK) 2>/dev/null ; do
if (( ++I >= WAIT_TIME )); then
echo "timeout waiting for ci.lock after $I seconds"
exit 1
fi
if (( $(date +%s) - $(lock_mtime) >= LOCK_STALE )); then
echo "lock expired after $LOCK_STALE seconds ..."
# racy but sleep so chances are low
sleep $(( 3 + RANDOM % 8 ))
if (( $(date +%s) - $(lock_mtime) >= LOCK_STALE )); then
echo "lock holder is gone"
rm -f $LOCK
else
echo "new lock holder"
fi
fi
sleep 1
done
echo "acquired ci.lock"
'
. Reason: exit status 1`

• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)