Save Mutations in Range Partitioned Log Files for Backup V3 by akankshamahajan15 · Pull Request #12752 · apple/foundationdb

akankshamahajan15 · 2026-03-05T04:51:15Z

This PR implements saving mutations to range-partitioned backup log files.

Main Logic:

onBackupChanges is called with list of active backups. BackupWorker updates the backups map it maintains and removes stopped backups from the map.
- It recomputes the mapping of KeyRange -> Vector of (BackupUIDs, PartitionIds) keyRangeToBackupAssignment based on intersection of backup's key range request and partitions key ranges
- Recomputation is triggered only when there is a change in backup added/stopped.
In SaveMutationsToFile:
- Iterates over keyRangeToBackupAssignment to create the log file based, stores it in activeFiles and records the mapping in fileIndexByBackupPartition Map of (backupUid, partitionId) -> index into activeFiles to make mutation routing easier.
- Introduces RangePartitionedLogFileInfo to hold per-file metadata instead of maintaining multiple parallel vectors.
- Writes a file header (Version, partition id and key range) for each log file.
- . For each mutation, looks up the list of log file indexes covering its key from keyRangeToBackupAssignment and appends the mutation to all log files using fileIndexByBackupPartition
- For ClearRange mutations, finds all intersecting key ranges and writes the full mutation once per file, using a set to avoid duplicates.

Example of keyRangetoFileIdxMap:

Given:
2 backups: 
Backup1 backs up [A, M), 
Backup2 backs up [E,  P)
2 partitions for this backup worker: 
Partition0 [E, G), Partition1 [K, R)

keyRangeToBackupAssignment would contain:

[E, G) -> [(B1, 0), (B2, 0)]   // Both backups cover this range in Partition0
[K, M) -> [(B1, 1), (B2, 1)]   // Both backups cover this in Partition1
[M, P) -> [(B2, 1)]             // Only Backup2 continues to P in Partition1

fileIndexByBackupPartition would contain:

(B1, 0) -> 0    // Backup1, Partition0 -> activeFiles[0]
(B1, 1)  -> 1    // Backup1, Partition1 -> activeFiles[1]
(B2, 0) -> 2    // Backup2, Partition0 -> activeFiles[2]
(B2, 1) -> 3    // Backup2, Partition1 -> activeFiles[3]


activeFiles:
[0] = B1, Partition0, [E, G)
[1] = B2, Partition0, [E, G)
[2] = B1, Partition1, [K, M)
[3] = B2, Partition1, [K, M)


and 
- A ClearRange [G,O) would intersect with map entry [K, M), and [M,P) collecting unique files {1, 3}, and write the full mutation to both files once.

Each log files has

┌─────────────────────────────────────────────────────────┐
│ FILE HEADER (written once)                              │
├─────────────────────────────────────────────────────────┤
│ int32_t version = 5001                                  │  4 bytes
│ int32_t partitionId                                     │  4 bytes
│ int32_t beginKeyLength                                  │  4 bytes
│ uint8_t beginKey[beginKeyLength]                        │  variable
│ int32_t endKeyLength                                    │  4 bytes
│ uint8_t endKey[endKeyLength]                            │  variable
├═════════════════════════════════════════════════════════┤
│ BLOCK 0 (1MB)                                           │
├─────────────────────────────────────────────────────────┤
│ int32_t blockVersion = 5001  (for independent parsing) │  4 bytes
│ Mutation 1: header + data                              │
│ Mutation 2: header + data                              │
│ ...                                                     │
│ Mutation N                                             │
│ 0xFF padding to block boundary                         │
├═════════════════════════════════════════════════════════┤
│ BLOCK 1 (1MB)                                           │
├─────────────────────────────────────────────────────────┤
│ int32_t blockVersion = 5001                           │  4 bytes
│ Mutation 1: header + data                              │
│ ...                                                     │
│ 0xFF padding                                           │
├═════════════════════════════════════════════════════════┤
│ ... more blocks ...                                     │

Each block has

|                    BLOCK HEADER                     │
│                     (4 bytes)                       │
│              RANGE_PARTITIONED_MLOG_VERSION               │
│                    value: 5001                      │
├─────────────────────────────────────────────────────┤
│                   MUTATION 1                        │
│  ┌───────────────────────────────────────────────┐  │
│  │ Version (8 bytes, big endian)                 │  │
│  │ Subversion (4 bytes, big endian)              │  │
│  │ Mutation Size (4 bytes, big endian)           │  │
│  │ Mutation Data (variable length)               │  │
│  └───────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────┤
│                   MUTATION 2                        │
│  ┌───────────────────────────────────────────────┐  │
│  │ Version (8 bytes, big endian)                 │  │
│  │ Subversion (4 bytes, big endian)              │  │
│  │ Mutation Size (4 bytes, big endian)           │  │
│  │ Mutation Data (variable length)               │  │

.......   
|                  MUTATION N                        │
│  ┌───────────────────────────────────────────────┐  │
│  │ Version (8 bytes, big endian)                 │  │
│  │ Subversion (4 bytes, big endian)              │  │
│  │ Mutation Size (4 bytes, big endian)           │  │
│  │ Mutation Data (variable length)               │  │
│  └───────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────┤
│         PADDING (0xFF bytes to fill block)          │
│            (fills to blockSize boundary)            │
└─────────────────────────────────────────────────────┘

TODO:
Pop and update saved version will be handled in next PR that implement suploadData which will call saveMutationsToFile and pop and update savedVersion accordingly.

Additional Note how everything works:

pullAsyncData continuously pulls mutations from TLogs into an in-memory queue self->messages while the backup is running.
A FlowLock with capacity BACKUP_WORKER_LOCK_BYTES (3GB default) caps the total estimated bytes of buffered messages; if the queue hits this limit, pulling blocks until uploads free space.
uploadData wakes roughly every BACKUP_UPLOAD_DELAY (10 sec default) seconds, scans self->messages, and uploads a prefix that ends on a safe, committed version boundary using saveMutationsToFile.
After each upload, it erases those messages, releases their lock bytes, advances savedVersion, and pops TLog data up to that version, preventing unbounded growth of both memory and TLog history.

In the current code (V2 and V3 backup), onBackupChanges and saveMutationsToFile can run at the same time, so a backup may be marked “stopped” while a file write is already in progress, and that in‑flight write can still include one last batch of data for it. This race is expected and benign: the stopped backup might get one extra file, but backup progress only moves forward, so we never miss or corrupt any data that should be backed up, and we don’t create gaps, just possibly a bit more data than strictly necessary

Testing

Wrote a simple test using mock and it creates a file:

[okteto:rocky ~] # ls -ltrh simfdb/backups/saveMutations_9c44dbf394e9b5a38386d56f86903177/rlogs/0000000000000000200/log,150,151,7,1048576
-rw-r--r-- 1 root root 48 Mar 14 01:22 simfdb/backups/saveMutations_9c44dbf394e9b5a38386d56f86903177/rlogs/0000000000000000200/log,150,151,7,1048576

foundationdb-ci · 2026-03-05T04:55:44Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: f454532
Duration 0:04:18
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (f454532b09df3150c77573fbe0596fdde58346c1) for primary source and source version f454532b09df3150c77573fbe0596fdde58346c1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-05T04:55:44Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: f454532
Duration 0:04:17
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (f454532b09df3150c77573fbe0596fdde58346c1) for primary source and source version f454532b09df3150c77573fbe0596fdde58346c1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-05T04:55:46Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: f454532
Duration 0:04:17
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (f454532b09df3150c77573fbe0596fdde58346c1) for primary source and source version f454532b09df3150c77573fbe0596fdde58346c1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-05T04:55:47Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: f454532
Duration 0:04:19
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (f454532b09df3150c77573fbe0596fdde58346c1) for primary source and source version f454532b09df3150c77573fbe0596fdde58346c1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-03-05T04:55:52Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: f454532
Duration 0:04:24
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (f454532b09df3150c77573fbe0596fdde58346c1) for primary source and source version f454532b09df3150c77573fbe0596fdde58346c1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-05T04:57:19Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: f454532
Duration 0:05:54
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (f454532b09df3150c77573fbe0596fdde58346c1) for primary source and source version f454532b09df3150c77573fbe0596fdde58346c1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-05T04:58:08Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: f454532
Duration 0:06:39
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (f454532b09df3150c77573fbe0596fdde58346c1) for primary source and source version f454532b09df3150c77573fbe0596fdde58346c1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-05T05:16:12Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: b0372f6
Duration 0:23:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-05T05:36:52Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: b0372f6
Duration 0:44:17
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-05T05:43:06Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: b0372f6
Duration 0:50:31
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-05T05:43:30Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: b0372f6
Duration 0:50:56
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-05T07:03:00Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: b0372f6
Duration 2:10:27
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-03-09T03:39:21Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 8dcd539
Duration 0:22:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-09T03:49:39Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 8dcd539
Duration 0:32:25
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-09T03:58:34Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 8dcd539
Duration 0:41:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-09T04:05:29Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 8dcd539
Duration 0:48:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-09T04:07:43Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 8dcd539
Duration 0:50:29
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-09T04:08:47Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 8dcd539
Duration 0:51:33
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-09T05:18:24Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 8dcd539
Duration 2:01:06
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

fdbclient/BackupContainerFileSystem.actor.cpp

fdbserver/BackupWorkerRangePartitioned.actor.cpp

gxglass · 2026-03-10T22:45:30Z

Thanks for the detailed commit message. In the block header in addition to int32_t blockVersion = 5001 I think we should also have a checksum (e.g. crc32c) over the data in the block. This will help detect end to end problems in data transfer. We can also maintain a file-level checksum (perhaps computed over each block level checksum) stored in a trailer after the last block.

If we end up with a backup with bad data in it this will help a) notify of this fact, and b) maybe help pinpoint where the corruption came from.

This can be tested by generating a backup file to an in-memory target, then flipping a random bit in it, then uploading the file, then downloading / restoring it (presumably we put the checksum check in the download/restore logic somewhere).

akankshamahajan15 · 2026-03-10T22:57:47Z

Thanks for the detailed commit message. In the block header in addition to int32_t blockVersion = 5001 I think we should also have a checksum (e.g. crc32c) over the data in the block. This will help detect end to end problems in data transfer. We can also maintain a file-level checksum (perhaps computed over each block level checksum) stored in a trailer after the last block.

If we end up with a backup with bad data in it this will help a) notify of this fact, and b) maybe help pinpoint where the corruption came from.

This can be tested by generating a backup file to an in-memory target, then flipping a random bit in it, then uploading the file, then downloading / restoring it (presumably we put the checksum check in the download/restore logic somewhere).

Yes, that should be doable and should work. Let me add that as well.

foundationdb-ci · 2026-03-12T18:40:18Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 48491a9
Duration 0:42:44
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-12T18:58:28Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 48491a9
Duration 1:01:17
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-12T19:08:05Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 48491a9
Duration 1:10:56
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

gxglass · 2026-03-12T20:00:32Z

Thanks for the detailed commit message. In the block header in addition to int32_t blockVersion = 5001 I think we should also have a checksum (e.g. crc32c) over the data in the block. This will help detect end to end problems in data transfer. We can also maintain a file-level checksum (perhaps computed over each block level checksum) stored in a trailer after the last block.
If we end up with a backup with bad data in it this will help a) notify of this fact, and b) maybe help pinpoint where the corruption came from.
This can be tested by generating a backup file to an in-memory target, then flipping a random bit in it, then uploading the file, then downloading / restoring it (presumably we put the checksum check in the download/restore logic somewhere).

Yes, that should be doable and should work. Let me add that as well.

Thought about this some more. What do you think about the following:

In addition to the above per-chunk checksums and a per-file overall checksum, combine the per-file checksums with a per-manifest / index file (forgot the name, I think there is a file like this) to get a single overall per-backup checksum. Store this in a file by itself written/uploaded as the last thing in backup creation.

In end to end testing, after writing a backup, (a) read it back and confirm that all of the files are OK and that the whole-backup checksum checks out, and (b) restore the backup to a new database and run some kind of distributed computation to compare the old database with the new database and ensure they're identical.

This leverages two separate principles:

Everything we write down on disk or network should be covered by checksums, and

Broad-based end to end verification is the most valuable because it catches bugs at any step of the way

Does this sound like something we can do or have plans to do?

Ack!! I'm making some changes requested by Neethu. Once that done, I look at these changes and suggestions as well. Overall I do agree with adding checksum block level as well as file level.

Discussed offline. My enthusiasm for the principle of end to end verification has caused me to forget the principle that we are backing up log files, so there is no discrete "start of this backup" and "end of this backup" and thus no clear "checksum for this backup".

So for now, I think checksums on files only makes sense. On that point I do think that any new file format which is relied upon for data integrity (certainly this format counts) should build in application level checksumming in case of errors in dependent storage systems, errors in subsequent retrievals, etc.

foundationdb-ci · 2026-03-12T20:20:15Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 48491a9
Duration 2:23:02
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

akankshamahajan15 · 2026-03-12T21:03:13Z

Thanks for the detailed commit message. In the block header in addition to int32_t blockVersion = 5001 I think we should also have a checksum (e.g. crc32c) over the data in the block. This will help detect end to end problems in data transfer. We can also maintain a file-level checksum (perhaps computed over each block level checksum) stored in a trailer after the last block.
If we end up with a backup with bad data in it this will help a) notify of this fact, and b) maybe help pinpoint where the corruption came from.
This can be tested by generating a backup file to an in-memory target, then flipping a random bit in it, then uploading the file, then downloading / restoring it (presumably we put the checksum check in the download/restore logic somewhere).

Yes, that should be doable and should work. Let me add that as well.

Thought about this some more. What do you think about the following:

In addition to the above per-chunk checksums and a per-file overall checksum, combine the per-file checksums with a per-manifest / index file (forgot the name, I think there is a file like this) to get a single overall per-backup checksum. Store this in a file by itself written/uploaded as the last thing in backup creation.

In end to end testing, after writing a backup, (a) read it back and confirm that all of the files are OK and that the whole-backup checksum checks out, and (b) restore the backup to a new database and run some kind of distributed computation to compare the old database with the new database and ensure they're identical.

This leverages two separate principles:

Everything we write down on disk or network should be covered by checksums, and

Broad-based end to end verification is the most valuable because it catches bugs at any step of the way

Does this sound like something we can do or have plans to do?

Ack!! I'm making some changes requested by Neethu. Once that done, I look at these changes and suggestions as well. Overall I do agree with adding checksum block level as well as file level.

Discussed offline. My enthusiasm for the principle of end to end verification has caused me to forget the principle that we are backing up log files, so there is no discrete "start of this backup" and "end of this backup" and thus no clear "checksum for this backup".

So for now, I think checksums on files only makes sense. On that point I do think that any new file format which is relied upon for data integrity (certainly this format counts) should build in application level checksumming in case of errors in dependent storage systems, errors in subsequent retrievals, etc.

Yes, I'm discussing it with Neethu for file level checksums and will add that in a separate/next PR.

foundationdb-ci · 2026-03-12T22:49:41Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: e5dd468
Duration 0:27:08
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-12T23:06:42Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: e5dd468
Duration 0:44:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-12T23:22:29Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: e5dd468
Duration 1:00:00
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-12T23:28:38Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: e5dd468
Duration 1:06:08
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-13T00:23:13Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: e5dd468
Duration 2:00:40
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-03-14T01:56:47Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 4f5bb22
Duration 0:25:50
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-14T02:03:56Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 4f5bb22
Duration 0:33:01
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-14T02:11:35Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 4f5bb22
Duration 0:40:39
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-14T02:16:24Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 4f5bb22
Duration 0:45:24
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-14T02:20:26Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 4f5bb22
Duration 0:49:28
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-14T02:28:07Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 4f5bb22
Duration 0:57:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-14T03:31:25Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 4f5bb22
Duration 2:00:27
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-03-17T17:21:28Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: d129444
Duration 0:23:49
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-17T17:41:00Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: d129444
Duration 0:42:47
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-17T17:54:51Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: d129444
Duration 0:57:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-17T18:22:07Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: d129444
Duration 1:24:26
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-03-17T19:18:40Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: d129444
Duration 2:20:59
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

akankshamahajan15 force-pushed the bk_save_mutations branch from f454532 to b0372f6 Compare March 5, 2026 04:52

akankshamahajan15 changed the title ~~[Draft] Save Mutations in Range Paritioned Log File~~ [Draft] Save Mutations in Range Partitioned Log File Mar 5, 2026

akankshamahajan15 force-pushed the bk_save_mutations branch from b0372f6 to 8dcd539 Compare March 9, 2026 03:17

akankshamahajan15 marked this pull request as ready for review March 9, 2026 03:31

akankshamahajan15 changed the title ~~[Draft] Save Mutations in Range Partitioned Log File~~ Save Mutations in Range Partitioned Log Files for Backup V3 Mar 9, 2026

akankshamahajan15 added the Backup_v3 label Mar 9, 2026

akankshamahajan15 requested a review from neethuhaneesha March 9, 2026 04:54

akankshamahajan15 requested a review from gxglass March 9, 2026 16:54

neethuhaneesha reviewed Mar 9, 2026

View reviewed changes

fdbclient/BackupContainerFileSystem.actor.cpp Outdated Show resolved Hide resolved

fdbserver/BackupWorkerRangePartitioned.actor.cpp Outdated Show resolved Hide resolved

fdbserver/BackupWorkerRangePartitioned.actor.cpp Show resolved Hide resolved

akankshamahajan15 requested a review from neethuhaneesha March 12, 2026 22:16

akankshamahajan15 force-pushed the bk_save_mutations branch from e5dd468 to 4f5bb22 Compare March 14, 2026 01:30

akankshamahajan15 added 4 commits March 17, 2026 09:56

Upload data to files

ea0b0ef

Addressed comments

bce668e

Add some comments as TODO

ae023e5

Fix some errors

d129444

akankshamahajan15 force-pushed the bk_save_mutations branch from 4f5bb22 to d129444 Compare March 17, 2026 16:57

Conversation

akankshamahajan15 commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

foundationdb-ci commented Mar 5, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 5, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 5, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 5, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 5, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Mar 5, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 5, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 5, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 5, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Mar 5, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 5, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 5, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 9, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 9, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 9, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Mar 9, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Mar 9, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 9, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 9, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gxglass commented Mar 10, 2026

Uh oh!

akankshamahajan15 commented Mar 10, 2026

Uh oh!

foundationdb-ci commented Mar 12, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Mar 12, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Mar 12, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

gxglass commented Mar 12, 2026

Uh oh!

akankshamahajan15 commented Mar 5, 2026 •

edited

Loading