PXB-3543 [trunk] : Xtrabackup is extremely slow when performing incremenal backup for 1 Million Tables - #1723
Merged
Merged
Conversation
…al backup for 1 Million tables https://perconadev.atlassian.net/browse/PXB-3543 Optimize incremental backup buffer initialization **Problem:** Incremental backups were suffering from extremely high CPU usage (up to 82% in perf reports) attributed to `__memset_avx512_unaligned_erms`. The root cause was the initialization of the delta buffer (~67MB per thread). The kernel lazily allocates physical RAM, so the initial `memset` triggered millions of Page Faults (`asm_exc_page_fault`), forcing the kernel to pause execution, allocate RAM, and zero it out. Additionally, flushing the buffer involved a redundant `memset` that wasted memory bandwidth. **Solution:** 1. Removed the full-buffer `memset` in `wf_incremental_init`. 2. Removed the full-buffer `memset` in `wf_incremental_process` (flush loop). 3. Replaced both with a "surgical" `memset` that only zeroes the first page (the Index Page). **Safety Analysis:** The delta buffer consists of Page 0 (Index/Header) and Pages 1..N (Data). - **Page 0:** Must be zeroed to ensure the file headers are clean and no garbage data exists after the `0xFFFFFFFF` sentinel. The new `memset` handles this. - **Pages 1..N:** These are populated sequentially via `memcpy`. We track the number of valid pages (`npages`) and only write `npages * page_size` to disk. Any "dirty" or uninitialized memory at the tail of the buffer is never read and never written to the file. **Impact:** - Eliminates the "First Touch" page fault storm during initialization. - Saves ~67MB of memory write bandwidth per flush cycle per thread. - Drastically reduces system CPU time during incremental backups.
jakub-nowakowski-percona
approved these changes
Jan 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.