PXB-3502 [9.7]: Process largest files first in all parallel phases - #1782
Merged
jakub-nowakowski-percona merged 2 commits intoAug 24, 2026
Merged
Conversation
https://perconadev.atlassian.net/browse/PXB-3502 Problem ======= When xtrabackup processes files in parallel (backup, copy-back, decompress, incremental apply), the work queue is FIFO. If the largest tablespace or SST file happens to be discovered last, it starts processing after all other threads have finished their smaller files, resulting in a single thread running alone for the duration of that large file. This "tail latency" problem makes parallel restores and backups slower than necessary. Fix === Replace FIFO scheduling with largest-first scheduling across all parallel processing phases so that the longest-running task always starts immediately, minimizing wall-clock time. Affected scenarios (all use --parallel): 1. xtrabackup --backup: InnoDB tablespace copying is now ordered largest-first via sorted datafiles_iter_t. 2. xtrabackup --backup (RocksDB): SST and meta file copying is now ordered largest-first via queue-based work-stealing. 3. xtrabackup --backup --stream=xbstream: files are written to the stream in largest-first order, so downstream xbstream extraction and decompression inherits that order. 4. xtrabackup --prepare --incremental-dir (delta apply): InnoDB .delta files are applied largest-first via the priority queue. 5. xtrabackup --prepare --incremental-dir (RocksDB): SST files from the incremental directory are copied/moved largest-first. 6. xtrabackup --copy-back / --move-back: all data files (InnoDB, RocksDB, non-InnoDB) are processed largest-first. 7. xtrabackup --decompress: compressed files are decompressed largest-first via the priority queue. Implementation ============== 1. datadir_queue (backup_copy.cc): Convert the internal std::queue to std::priority_queue with a comparator that orders entries by descending file_size. Tie-break on path for determinism. Empty directories get lowest priority. 2. datafiles_iter_t (xtrabackup.cc): Sort the tablespace node vector by descending byte size (page count * physical page size) at the end of datafiles_iter_new(). 3. RocksDB (backup_copy.cc): Replace par_for()-based contiguous slicing with a queue-based work-stealing approach. RocksDB data and meta file lists are merged into a single datadir_queue so that all files compete globally for largest-first scheduling. Extract run_worker_threads() as a common template for thread lifecycle management shared between run_data_threads() and run_rocksdb_threads(). 4. File size propagation: Populate file_size in datadir_entry_t during directory scanning (xb_process_datadir via stat()) and in Myrocks_datadir::scan_dir(). 5. Log enhancement: Print both raw byte size and human-readable size in file copy/move log messages. Tests ===== Five test scripts verify ordering with --parallel=1 (which makes scheduling deterministic): - t/pxb_largest_first_innodb.sh: InnoDB backup and copy-back - t/pxb_largest_first_delta.sh: InnoDB incremental delta apply - suites/rocksdb/largest_first.sh: RocksDB backup and copy-back - suites/rocksdb/largest_first_incremental.sh: RocksDB incremental - suites/compression/largest_first_decompress.sh: LZ4 decompression
Contributor
Author
satya-bodapati
approved these changes
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PXB-3502: Process largest files first in all parallel phases
https://perconadev.atlassian.net/browse/PXB-3502
Problem
When xtrabackup processes files in parallel (backup, copy-back,
decompress, incremental apply), the work queue is FIFO. If the
largest tablespace or SST file happens to be discovered last, it
starts processing after all other threads have finished their
smaller files, resulting in a single thread running alone for the
duration of that large file. This "tail latency" problem makes
parallel restores and backups slower than necessary.
Fix
Replace FIFO scheduling with largest-first scheduling across all
parallel processing phases so that the longest-running task always
starts immediately, minimizing wall-clock time.
Affected scenarios (all use --parallel):
xtrabackup --backup: InnoDB tablespace copying is now ordered
largest-first via sorted datafiles_iter_t.
xtrabackup --backup (RocksDB): SST and meta file copying is
now ordered largest-first via queue-based work-stealing.
xtrabackup --backup --stream=xbstream: files are written to
the stream in largest-first order, so downstream xbstream
extraction and decompression inherits that order.
xtrabackup --prepare --incremental-dir (delta apply): InnoDB
.delta files are applied largest-first via the priority queue.
xtrabackup --prepare --incremental-dir (RocksDB): SST files
from the incremental directory are copied/moved largest-first.
xtrabackup --copy-back / --move-back: all data files (InnoDB,
RocksDB, non-InnoDB) are processed largest-first.
xtrabackup --decompress: compressed files are decompressed
largest-first via the priority queue.
Implementation
datadir_queue (backup_copy.cc): Convert the internal std::queue
to std::priority_queue with a comparator that orders entries by
descending file_size. Tie-break on path for determinism. Empty
directories get lowest priority.
datafiles_iter_t (xtrabackup.cc): Sort the tablespace node
vector by descending byte size (page count * physical page size)
at the end of datafiles_iter_new().
RocksDB (backup_copy.cc): Replace par_for()-based contiguous
slicing with a queue-based work-stealing approach. RocksDB data
and meta file lists are merged into a single datadir_queue so
that all files compete globally for largest-first scheduling.
Extract run_worker_threads() as a common template for thread
lifecycle management shared between run_data_threads() and
run_rocksdb_threads().
File size propagation: Populate file_size in datadir_entry_t
during directory scanning (xb_process_datadir via stat()) and
in Myrocks_datadir::scan_dir().
Log enhancement: Print both raw byte size and human-readable
size in file copy/move log messages.
Tests
Five test scripts verify ordering with --parallel=1 (which makes
scheduling deterministic):