PXB-3862 : Handle OS-cached pages in the storage probe

On a buffered server the probe's sample regions may already sit in the
OS page cache (warmed by the server or a previous scan), and a cached
read times RAM, not the device: on a fully cached tablespace the round
trip measured 1us and the read request cost collapsed to the 64KiB
floor, understating how much merging the storage deserves.

The probe already dropped its samples with posix_fadvise(DONTNEED), but
newer kernels cache a sequentially read file in blocks of up to 2MB
(XFS since kernel 5.17, ext4 since 6.13) and ignore the advice unless
it covers a whole block, so the page-sized evictions did nothing there.

Evict a whole 2MB-aligned window around every sample instead, right
before reading it. The probe gives up ~40MB of cache on a multi-GB
file; the copy itself still reads a warm file from RAM. Verified
against a fully cached 21GB tablespace: calibration now reports device
numbers (178us round trip) while the backup reads only 79MB from the
device.

Author: satya-bodapati

storage/innobase/xtrabackup/src/xb_io_probe.h

@@ -57,6 +57,35 @@ struct Probe_result {
   uint64_t bw_bytes_per_sec{0}; /*!< sequential read bandwidth */
 };
 
+/** Drop a byte range of the file from the OS page cache, so the probe
+times the device rather than RAM (the range may be warm from the server
+or a previous scan). Newer kernels cache a sequentially read file in
+blocks of up to 2MB and ignore the advice unless it covers a whole
+block, so the range is widened to full 2MB multiples. The advice skips
+dirty pages; under O_DIRECT it is a no-op.
+
+@param[in] fd      file the probe reads from
+@param[in] offset  start of the byte range about to be read
+@param[in] length  length of that range */
+inline void evict_from_page_cache(int fd, uint64_t offset, uint64_t length) {
+#ifdef POSIX_FADV_DONTNEED
+  constexpr uint64_t EVICT_SIZE_BYTES = 2 * 1024 * 1024;
+  /* the file is a row of 2MB units; dividing a byte position by the
+  unit size gives the number of the unit it falls in. Evict every unit
+  the range touches, from the one holding its first byte to the one
+  holding its last byte, in full. */
+  const uint64_t first_unit = offset / EVICT_SIZE_BYTES;
+  const uint64_t last_unit = (offset + length - 1) / EVICT_SIZE_BYTES;
+  ::posix_fadvise(fd, first_unit * EVICT_SIZE_BYTES,
+                  (last_unit - first_unit + 1) * EVICT_SIZE_BYTES,
+                  POSIX_FADV_DONTNEED);
+#else
+  (void)fd;
+  (void)offset;
+  (void)length;
+#endif
+}
+
 /** Measure the storage behind a file.
 
 Every read below fetches whole 16KB pages at page-aligned offsets - the
@@ -122,26 +151,11 @@ inline std::optional<Probe_result> probe_storage(const char *path,
     return (std::nullopt);
   }
 
-#ifdef POSIX_FADV_DONTNEED
-  /* On a buffered server the sample regions may already sit in the OS
-  page cache (warmed by the server or a previous backup); measuring
-  them would time RAM, not the device. Drop the cached pages of just
-  the ~16MB this probe is about to read - surgical, so a live server
-  loses next to no cache warmth. The advice skips dirty pages, and the
-  RTT median tolerates a few residual hits; under O_DIRECT it is a
-  no-op. */
-  ::posix_fadvise(fd, (file_pages / 2) * PROBE_PAGE_BYTES,
-                  SEQ_SAMPLES * SEQ_CHUNK_BYTES, POSIX_FADV_DONTNEED);
-  for (int i = 1; i <= RTT_SAMPLES; i++) {
-    ::posix_fadvise(fd,
-                    ((file_pages / (RTT_SAMPLES + 1)) * i) * PROBE_PAGE_BYTES,
-                    PROBE_PAGE_BYTES, POSIX_FADV_DONTNEED);
-  }
-#endif
-
   /* contiguous pages from the middle of the file onward -> bandwidth
   (PROBE_MIN_FILE_BYTES guarantees the chunks fit after the midpoint) */
   Probe_result result;
+  evict_from_page_cache(fd, (file_pages / 2) * PROBE_PAGE_BYTES,
+                        SEQ_SAMPLES * SEQ_CHUNK_BYTES);
   uint64_t seq_page_no = file_pages / 2;
   uint64_t seq_bytes = 0;
   const auto seq_start = clock::now();
@@ -172,6 +186,7 @@ inline std::optional<Probe_result> probe_storage(const char *path,
   std::vector<uint64_t> rtt_samples;
   for (int i = 1; i <= RTT_SAMPLES; i++) {
     const uint64_t page_no = (file_pages / (RTT_SAMPLES + 1)) * i;
+    evict_from_page_cache(fd, page_no * PROBE_PAGE_BYTES, PROBE_PAGE_BYTES);
     const auto start = clock::now();
     if (::pread(fd, buf, PROBE_PAGE_BYTES, page_no * PROBE_PAGE_BYTES) !=
         static_cast<ssize_t>(PROBE_PAGE_BYTES)) {