PXB-3862 : [9.7] Merge scattered changed-page reads in page-tracking backups - #1793
Open
satya-bodapati wants to merge 2 commits into
Open
PXB-3862 : [9.7] Merge scattered changed-page reads in page-tracking backups#1793satya-bodapati wants to merge 2 commits into
satya-bodapati wants to merge 2 commits into
Conversation
https://perconadev.atlassian.net/browse/PXB-3862 Problem ------- A page-tracking incremental reads only the pages the server marked as changed. Consecutive changed pages are already read together in one request; the problem is scattered changes. If the changed pages are 1, 3, 5, 7, 9 xtrabackup issues five single-page reads. Every read request costs a full I/O round trip, so with scattered changes the copy phase is bound by the number of requests instead of the amount of data, and an incremental can take several times longer than a full scan of the same file. Fix --- Read the whole range 1-9 in one request. The unchanged pages 2, 4, 6, 8 are read as filler and discarded by the existing incremental write filter, which drops every page whose FIL_PAGE_LSN is older than incremental_lsn. The backup's content and size are byte-for-byte unchanged; only the read pattern changes. A gap is worth combining across exactly when its bytes cost less than one read request. That cost, in bytes of sequential transfer, is round_trip * bandwidth - and no fixed number fits both a local NVMe and a network volume, so it is measured at backup start (probe_storage, xb_io_probe.h): twelve scattered single-page reads give the round trip, 16MB of sequential reads the bandwidth, on the largest changed data file (at least 64MB). Each sample region is first dropped from the OS page cache in whole 2MB-aligned units (newer kernels cache sequentially read files in blocks of up to 2MB and ignore a posix_fadvise(DONTNEED) that covers only part of a block), so a file warmed by a buffered server or a previous scan is still measured at device speed - while the copy itself keeps reading the warm file from RAM. Then read_request_cost = round_trip * bandwidth / 1.5, clamped to [64KB, 1MB], 512KB if unmeasurable merge_gap = read_request_cost / physical_page_size The /1.5 margin absorbs copy-pipeline overhead and measurement noise, erring toward reading less; it was calibrated on two instrumented machines whose break-evens bound it from both sides. The per-tablespace conversion lets compressed tablespaces combine across the same byte cost. Filler bytes are always the actual gap sizes present in the data, never the limit, so a generous limit reads nothing extra. --page-tracking-merge-gap exposes the behaviour: "auto" (default) as described; a page count pins one value for all tables and skips the probe; 0 keeps the previous strict-consecutive reads. Log messages ------------ Once per backup, the measurement: pagetracking: calibrated storage (./test/t1.ibd): request round trip 124 us, sequential read 984 MB/s -> one read request costs ~83KB of sequential transfer; gaps cheaper than this are combined Per table with at least 1000 changed pages, when its copy finishes, accumulated from what was actually read: pagetracking: test/t1.ibd: 3196 changed pages in 3196 ranges (avg gap 2.0 pages); merge-gap=4 (auto) combined them into 2 reads: request reduction 1598.0x, read amplification 2.98x; issued 16 read batches "ranges" is the requests merge-gap=0 would issue; "request reduction" is the benefit and "read amplification" its price (bytes read divided by changed bytes - read volume only, backup size is unaffected); "issued" exceeds the group count only when a group is larger than --read-buffer-size and is read in buffer-sized pieces. When the typical gap costs more than one read request, an extra line names both numbers and the pinned value to try, so a boundary case is diagnosable from the log alone: pagetracking: test/t1.ibd: typical gap 8.9 pages (143KB) costs more than one read request (83KB); reads stay individual - if sequential read throughput is high, --page-tracking-merge-gap=9 may be faster Testing: unit tests (xb_page_group-t) cover the read request cost model across device classes and the storage probe's failure modes; a framework testcase sweeps change densities and asserts only hardware-independent invariants - ranges vs combined reads vs issued requests, auto vs strict, the cost floor and ceiling, and restore correctness - and was verified to fail against a build with the combining silently disabled.
satya-bodapati
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
9.7 port of #1780, done as a merge of the 8.4 branch (train model) on top of the 8.4->9.7 null merge #1791.
Merge #1791 first: the commit list then shows only the fix commit and its merge commit. Zero conflicts; the merged content is byte-identical to the 8.4 change. Build + xb_pagetracking_merge_gap test pass against PS 9.7.0-1.
https://perconadev.atlassian.net/browse/PXB-3862