Skip to content

PXB-3502 [9.7]: Process largest files first in all parallel phases - #1782

Merged
jakub-nowakowski-percona merged 2 commits into
percona:9.7from
jakub-nowakowski-percona:PXB-3502-9.7
Aug 24, 2026
Merged

PXB-3502 [9.7]: Process largest files first in all parallel phases#1782
jakub-nowakowski-percona merged 2 commits into
percona:9.7from
jakub-nowakowski-percona:PXB-3502-9.7

Conversation

@jakub-nowakowski-percona

Copy link
Copy Markdown
Contributor

PXB-3502: Process largest files first in all parallel phases

https://perconadev.atlassian.net/browse/PXB-3502

Problem

When xtrabackup processes files in parallel (backup, copy-back,
decompress, incremental apply), the work queue is FIFO. If the
largest tablespace or SST file happens to be discovered last, it
starts processing after all other threads have finished their
smaller files, resulting in a single thread running alone for the
duration of that large file. This "tail latency" problem makes
parallel restores and backups slower than necessary.

Fix

Replace FIFO scheduling with largest-first scheduling across all
parallel processing phases so that the longest-running task always
starts immediately, minimizing wall-clock time.

Affected scenarios (all use --parallel):

  1. xtrabackup --backup: InnoDB tablespace copying is now ordered
    largest-first via sorted datafiles_iter_t.

  2. xtrabackup --backup (RocksDB): SST and meta file copying is
    now ordered largest-first via queue-based work-stealing.

  3. xtrabackup --backup --stream=xbstream: files are written to
    the stream in largest-first order, so downstream xbstream
    extraction and decompression inherits that order.

  4. xtrabackup --prepare --incremental-dir (delta apply): InnoDB
    .delta files are applied largest-first via the priority queue.

  5. xtrabackup --prepare --incremental-dir (RocksDB): SST files
    from the incremental directory are copied/moved largest-first.

  6. xtrabackup --copy-back / --move-back: all data files (InnoDB,
    RocksDB, non-InnoDB) are processed largest-first.

  7. xtrabackup --decompress: compressed files are decompressed
    largest-first via the priority queue.

Implementation

  1. datadir_queue (backup_copy.cc): Convert the internal std::queue
    to std::priority_queue with a comparator that orders entries by
    descending file_size. Tie-break on path for determinism. Empty
    directories get lowest priority.

  2. datafiles_iter_t (xtrabackup.cc): Sort the tablespace node
    vector by descending byte size (page count * physical page size)
    at the end of datafiles_iter_new().

  3. RocksDB (backup_copy.cc): Replace par_for()-based contiguous
    slicing with a queue-based work-stealing approach. RocksDB data
    and meta file lists are merged into a single datadir_queue so
    that all files compete globally for largest-first scheduling.
    Extract run_worker_threads() as a common template for thread
    lifecycle management shared between run_data_threads() and
    run_rocksdb_threads().

  4. File size propagation: Populate file_size in datadir_entry_t
    during directory scanning (xb_process_datadir via stat()) and
    in Myrocks_datadir::scan_dir().

  5. Log enhancement: Print both raw byte size and human-readable
    size in file copy/move log messages.

Tests

Five test scripts verify ordering with --parallel=1 (which makes
scheduling deterministic):

  • t/pxb_largest_first_innodb.sh: InnoDB backup and copy-back
  • t/pxb_largest_first_delta.sh: InnoDB incremental delta apply
  • suites/rocksdb/largest_first.sh: RocksDB backup and copy-back
  • suites/rocksdb/largest_first_incremental.sh: RocksDB incremental
  • suites/compression/largest_first_decompress.sh: LZ4 decompression

https://perconadev.atlassian.net/browse/PXB-3502

Problem
=======

When xtrabackup processes files in parallel (backup, copy-back,
decompress, incremental apply), the work queue is FIFO. If the
largest tablespace or SST file happens to be discovered last, it
starts processing after all other threads have finished their
smaller files, resulting in a single thread running alone for the
duration of that large file. This "tail latency" problem makes
parallel restores and backups slower than necessary.

Fix
===

Replace FIFO scheduling with largest-first scheduling across all
parallel processing phases so that the longest-running task always
starts immediately, minimizing wall-clock time.

Affected scenarios (all use --parallel):

1. xtrabackup --backup: InnoDB tablespace copying is now ordered
   largest-first via sorted datafiles_iter_t.

2. xtrabackup --backup (RocksDB): SST and meta file copying is
   now ordered largest-first via queue-based work-stealing.

3. xtrabackup --backup --stream=xbstream: files are written to
   the stream in largest-first order, so downstream xbstream
   extraction and decompression inherits that order.

4. xtrabackup --prepare --incremental-dir (delta apply): InnoDB
   .delta files are applied largest-first via the priority queue.

5. xtrabackup --prepare --incremental-dir (RocksDB): SST files
   from the incremental directory are copied/moved largest-first.

6. xtrabackup --copy-back / --move-back: all data files (InnoDB,
   RocksDB, non-InnoDB) are processed largest-first.

7. xtrabackup --decompress: compressed files are decompressed
   largest-first via the priority queue.

Implementation
==============

1. datadir_queue (backup_copy.cc): Convert the internal std::queue
   to std::priority_queue with a comparator that orders entries by
   descending file_size. Tie-break on path for determinism. Empty
   directories get lowest priority.

2. datafiles_iter_t (xtrabackup.cc): Sort the tablespace node
   vector by descending byte size (page count * physical page size)
   at the end of datafiles_iter_new().

3. RocksDB (backup_copy.cc): Replace par_for()-based contiguous
   slicing with a queue-based work-stealing approach. RocksDB data
   and meta file lists are merged into a single datadir_queue so
   that all files compete globally for largest-first scheduling.
   Extract run_worker_threads() as a common template for thread
   lifecycle management shared between run_data_threads() and
   run_rocksdb_threads().

4. File size propagation: Populate file_size in datadir_entry_t
   during directory scanning (xb_process_datadir via stat()) and
   in Myrocks_datadir::scan_dir().

5. Log enhancement: Print both raw byte size and human-readable
   size in file copy/move log messages.

Tests
=====

Five test scripts verify ordering with --parallel=1 (which makes
scheduling deterministic):

- t/pxb_largest_first_innodb.sh: InnoDB backup and copy-back
- t/pxb_largest_first_delta.sh: InnoDB incremental delta apply
- suites/rocksdb/largest_first.sh: RocksDB backup and copy-back
- suites/rocksdb/largest_first_incremental.sh: RocksDB incremental
- suites/compression/largest_first_decompress.sh: LZ4 decompression
@jakub-nowakowski-percona

jakub-nowakowski-percona commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

@jakub-nowakowski-percona
jakub-nowakowski-percona merged commit 18e8462 into percona:9.7 Aug 24, 2026
3 checks passed
@jakub-nowakowski-percona
jakub-nowakowski-percona deleted the PXB-3502-9.7 branch August 24, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants