PXB-3747: report backup size and uncompressed size

[satya-bodapati]

Summary

xtrabackup now embeds two byte-accurate size fields in xtrabackup_info
and mirrors them to the error log:

• backup_size (bytes) — final bytes the backup wrote to disk
  (or to the stream). Always reported.
• uncompressed_backup_size (bytes) — logical pre-compression
  bytes (data + redo + metadata + server-encrypted InnoDB files +
  RocksDB SSTs that skip the compressor). Reported only when
  --compress is used.

Both values are byte-perfect against the actual on-disk artifact for
every combination of --compress{,-lz4,-zstd} / --encrypt /
--stream=xbstream / --extra-lsndir / incremental / sparse.
Strict-equality asserts in t/backup_size_basic.sh and
t/backup_size_compress.sh cover 15+ scenarios end-to-end.

JIRA: https://perconadev.atlassian.net/browse/PXB-3747

How to use this feature

No new flags or options — the behavior is enabled for every backup.

1. Final backup size (backup_size) — always present in
   xtrabackup_info, e.g.:

   backup_size = 129456789
   

   and logged:

   Backup size: 123.45 MiB (129456789 bytes)
   

2. Uncompressed size under --compress (uncompressed_backup_size) —
   added only when --compress{,-lz4,-zstd} is used:

   uncompressed_backup_size = 479123456
   

   along with two extra log lines:

   Uncompressed backup size: 456.78 MiB (479123456 bytes)
   Compression ratio: 3.70x
   

   This helps operators estimate the disk space required to extract
   a compressed backup before running xbstream -x /
   xtrabackup --decompress, and makes backups taken at different
   dates comparable regardless of compressibility.

3. Where xtrabackup_info lives — the copy inside the backup
   directory follows the transport the user picked: it is compressed
   under --compress, encrypted under --encrypt, and wrapped in
   the xbstream under --stream=xbstream. To always obtain an
   unencrypted, uncompressed copy of xtrabackup_info, pass
   --extra-lsndir=<dir> — the file written there is plaintext
   regardless of --compress / --encrypt.

4. Error-log output — the two (or three, under --compress)
   lines above appear at the end of the backup, just before the
   "completed OK!" banner.

Commits

Split into two focused, bisect-friendly commits:

1. PXB-3747 : Report final on-disk backup_size in xtrabackup_info
   — adds backup_size with zero --compress awareness; includes
   the shared strict-size test infrastructure.
2. PXB-3747 : Report uncompressed_backup_size for --compress backups
   — adds uncompressed_backup_size, the "Uncompressed backup size"
   log line, and the "Compression ratio" log line, wired up only
   for --compress runs.

Commit 1 design — backup_size

The datasink pipeline has exactly one authoritative component: the
leaf. Every byte that lands in the backup artifact passes
through ds_local / ds_stdout / ds_fifo. Wrapper datasinks
(encrypt, xbstream, buffer, tmpfile, compress) only transform and
forward bytes.

    ds_data (head)
         |  ds_write()
         v
    [ wrappers: xbstream / encrypt / compress / ... ]
         |  pipe_ctxt
         v
    leaf (ds_local | ds_stdout | ds_fifo)   authoritative byte count
         |
         v
    my_write() / fwrite()

Rather than adding one-off get_foo() vtable slots per metric, the
patch introduces a tiny per-datasink metrics API: each datasink
can publish named {name, value} pairs via an optional
report_metrics slot, and callers read one off a specific node
with ds_find_metric. The three leaves publish an atomic
bytes_written counter bumped on every successful write.
get_final_backup_size() walks to the leaf and reads it;
backup_size = N is written to xtrabackup_info and logged.

Tablespace_map::serialize() and the transition-key dumper are
moved ahead of backup_finish() so their bytes are included in
the reported backup_size.

Commit 2 design — uncompressed_backup_size

Why two data pipelines exist

xtrabackup maintains two top-level data pipelines because not
every file benefits from compression:

• ds_data — normal InnoDB tablespaces, redo, metadata.
  Compressed when --compress is on.
• ds_uncompressed_data — server-encrypted InnoDB tablespaces
  (re-compressing ciphertext is wasteful) and RocksDB SST files
  (already compressed internally). Bypasses the compress wrapper.

Both pipelines share their encrypt / xbstream / leaf tail as an
init-time context-sharing optimization:

    ds_data              --> buffer --> compress --> encrypt --> leaf
                                                                  ^
                                                                  |
    ds_uncompressed_data -----------------------> encrypt --------+

Why the counter cannot live on any ds_ctxt_t

The shared tail makes "attach a counter to the ctxt" broken no
matter which ctxt is picked:

• Leaf ctxt. Under --stream=xbstream, ds_data, ds_redo,
  ds_meta, and ds_uncompressed_data all terminate at the same
  leaf. A leaf-ctxt counter would sum bytes from every pipeline,
  and for the compressed pipeline those bytes are post-compression
  — neither the logical total nor the on-disk total.
• Encrypt / xbstream / buffer ctxts. ds_data and
  ds_uncompressed_data share these too. Counting there mixes
  post-compression bytes (from ds_data) with raw bytes (from
  ds_uncompressed_data); the sum is meaningless.

Per-file ds_file_t handle is the right anchor

The counter attaches where a single logical write unambiguously
enters exactly one pipeline — on the per-file handle returned by
the top-level ds_open(). Each top-level open creates a fresh
ds_file_t, the caller decides whether that file's bytes count,
and ds_write / ds_write_sparse bump the counter with the
length supplied at the entry, before any wrapper transforms it:

    xtrabackup top-level open
         |  ds_open_track_uncomp(..., xb_global_track_uncomp_bytes())
         |                             |
         |                             +-- nullptr when !--compress
         v
    ds_file_t { uncomp_bytes -> xb_uncomp_bytes_counter }
         |  ds_write(len) / ds_write_sparse(packed_len)
         |     if (file->uncomp_bytes) uncomp_bytes->add_uncompressed(len)
         v
    [ compress / encrypt / buffer / xbstream wrappers ]
         |
         v
    leaf  (already counted as backup_size by commit 1)

    xb_uncomp_bytes_counter.get_uncompressed_backup_size()
         |
         v
    uncompressed_backup_size  (xtrabackup_info + error log)

Tracking is opt-in and scoped to the top-level opens on ds_data
/ ds_redo / ds_meta / ds_uncompressed_data, so the hot-path
cost is one atomic add per write on --compress runs; non-compress
runs skip even that. The same counter aggregates both pipelines,
so server-encrypted IBDs and RocksDB SSTs are counted exactly
once, at their logical size.

xb_global_track_uncomp_bytes() returns a real counter only when
--compress is active, so non-compress runs emit neither the
field nor the log lines.

Test plan

• t/backup_size_basic.sh — --target-dir, --stream=xbstream,
  --encrypt on both, incremental chains, sparse tablespaces, full
  restore round-trip. Every scenario asserts backup_size matches
  the on-disk / on-stream byte count exactly; uncompressed_backup_size
  must not be present.
• t/backup_size_compress.sh — --compress target-dir,
  --compress xbstream, --compress + --encrypt, incremental
  chains with compression, RocksDB, server-encrypted InnoDB,
  redo-log encryption, sparse + --compress, and --extra-lsndir.
  Every scenario asserts backup_size matches the compressed
  on-disk size and, after decompression, uncompressed_backup_size
  matches the decompressed tree size byte-for-byte.
• suites/reducedlock/backup_size_reduced.sh — reduced-lock
  • concurrent TRUNCATE edge case (zero-size tablespace).
• -s compression — 12/12 runnable tests pass; shared
  strict-size helpers live in inc/common.sh.

[azure-pipelines]

Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.

[azure-pipelines]

Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.

[azure-pipelines]

Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.

[satya-bodapati]

https://pxb.cd.percona.com/view/PXB%208.1/job/percona-xtrabackup-8.1-multijob/144/

[Copilot]

Pull request overview

This PR adds byte-accurate backup size reporting to xtrabackup by plumbing size counters through the datasink pipeline and emitting the results in xtrabackup_info and the error log. It also introduces end-to-end shell tests that assert strict equality between reported sizes and on-disk/on-stream artifacts across compressed/uncompressed, encrypted, streamed, incremental, and sparse scenarios.

Changes:

• Add backup_size reporting based on a leaf datasink byte counter (local/stdout/fifo) and log it at backup completion.
• Add uncompressed_backup_size reporting for --compress via per-file metrics tracked at top-level ds_open() entrypoints (pre-compression logical bytes).
• Add shared test helpers and new test suites validating strict byte-perfect invariants across many backup configurations.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
storage/innobase/xtrabackup/test/t/backup_size_compress.sh	New end-to-end coverage validating backup_size + uncompressed_backup_size under --compress across multiple scenarios.
storage/innobase/xtrabackup/test/t/backup_size_basic.sh	New end-to-end coverage validating backup_size for non-compress backups and asserting absence of uncompressed_backup_size.
storage/innobase/xtrabackup/test/suites/reducedlock/backup_size_reduced.sh	New reduced-lock TRUNCATE regression test for byte-perfect backup_size.
storage/innobase/xtrabackup/test/inc/common.sh	Adds shared helpers for parsing xtrabackup_info and performing strict size assertions.
storage/innobase/xtrabackup/src/xtrabackup.h	Declares size-reporting APIs and introduces xb_get_metrics() helper for conditional tracking under --compress.
storage/innobase/xtrabackup/src/xtrabackup.cc	Implements size getters, ensures tracked opens for key backup artifacts, and reorders operations so size sampling includes late-written files.
storage/innobase/xtrabackup/src/space_map.cc	Switches tablespace map output to tracked open for inclusion in uncompressed metrics when compressing.
storage/innobase/xtrabackup/src/redo_log.cc	Switches redo logfile creation to tracked open for uncompressed metrics.
storage/innobase/xtrabackup/src/keyring_plugins.cc	Switches key dump output to tracked open for uncompressed metrics.
storage/innobase/xtrabackup/src/ds_xbstream.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_tmpfile.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_stdout.cc	Adds leaf byte counting + get_bytes_written implementation for stdout leaf and ensures file initialization.
storage/innobase/xtrabackup/src/ds_local.cc	Adds leaf byte counting + get_bytes_written implementation for local leaf (including sparse trailing-byte fix accounting).
storage/innobase/xtrabackup/src/ds_fifo.cc	Adds leaf byte counting + get_bytes_written implementation for FIFO leaf.
storage/innobase/xtrabackup/src/ds_encrypt.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_decrypt.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_decompress_zstd.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_decompress_lz4.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_decompress.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_compress_zstd.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_compress_lz4.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_compress.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/ds_buffer.cc	Updates datasink vtable for new API slot and ensures ds_init_file() is called.
storage/innobase/xtrabackup/src/datasink.h	Extends datasink interface with get_bytes_written, introduces metrics plumbing on ds_file_t, and adds tracked open helpers.
storage/innobase/xtrabackup/src/datasink.cc	Implements tracked open + metrics accumulation, adds ds_leaf(), and adjusts ds_create initialization.
storage/innobase/xtrabackup/src/backup_mysql.cc	Embeds backup_size and conditionally appends uncompressed_backup_size into xtrabackup_info.
storage/innobase/xtrabackup/src/backup_copy.cc	Switches top-level opens to tracked opens and logs human-readable size summaries + compression ratio.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[satya-bodapati]

Pushed follow-up commit 7cc89789e5b — PXB-3747 : Addressing review comments. Will squash before merge.

What it addresses:

1. fs_support_punch_hole initialization — ds_ctxt_t::pipe_ctxt now defaults to nullptr in the struct, and the five my_malloc-based sinks (local, stdout, xbstream, buffer, tmpfile) use MY_FAE | MY_ZEROFILL. xbstream_init() advertises fs_support_punch_hole = true since its wire format round-trips sparse chunks natively. ds_create() no longer overrides the capability bit.
2. local_close() counts the 1-byte trailing-hole write into bytes_written, so sparse files under --target-dir no longer under-report by 1 byte per file.
3. "Finsh" → "Finish" in backup_copy.cc.
4. Reflowed the report_backup_size() doc comment so the "Compression ratio: inf" example reads as a single sentence.
5. sum_file_bytes() in inc/common.sh early-returns 0 when the argument is not a directory, silencing the stray find(1) stderr message.

Verified by rebuilding in Debug and running t/backup_size_basic.sh and t/backup_size_compress.sh — both pass.

[satya-bodapati]

Follow-up on c935a9d (force-pushed): dropped the redundant packed_len loop in ds_write_sparse(). The len parameter is already the packed (hole-excluded) payload size — callers pre-pack the buffer (dst_pos in write_ibd_buffer, chunk.length in xbstream.cc) and local_write_sparse walks the sparse_map writing exactly that many bytes in total. Now we just add_uncompressed(len). Verified with both t/backup_size_basic.sh and t/backup_size_compress.sh (byte-exact size assertions).

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jakub-nowakowski-percona]

Approved with optional remarks.

[satya-bodapati]

Follow-up commit 8d20d4b — PXB-3747 : Address Jakub's review comments (will squash before merge).

What it addresses from Jakub's review round:

• J2 (datasink.h pointer init) — every remaining pointer member in ds_ctxt_t (datasink, root, ptr) and all five in ds_file_t (ptr, path, datasink, ctxt, uncomp_bytes) is now default-initialised to nullptr in-class. Newly constructed handles no longer depend on the allocation site remembering MY_ZEROFILL / explicit field init.
• J3 + J5 (ds_metric::name and ds_find_metric doc) — ds_metric::name and ds_find_metric's name parameter are now std::string_view. The ownership comment and the "(strcmp comparison)" implementation-leak are gone. ds_find_metric uses m.name == name; strcmp / <cstring> removed from datasink.cc. Producers stay the same — string literals convert implicitly.
• J4 (report_metrics out param) — the vtable slot is now std::vector<ds_metric> &out. The three leaf implementations (ds_local, ds_stdout, ds_fifo) use out.push_back(...) accordingly; the single caller (ds_find_metric) passes a stack-local vector. Other vtables' nullptr /* report_metrics */ placeholders are unaffected.

Left as-is with the existing replies standing: J1 (ALWAYS_INLINE) and J6 (MY_ZEROFILL).

C1 (Compression ratio 3.70 vs 3.70x): the PR description already shows 3.70x; the Commit-2 message example will be fixed at squash time.

Verified: Debug rebuild clean; t/backup_size_basic.sh and t/backup_size_compress.sh both pass (byte-exact size assertions).

[satya-bodapati]

Squashed the two follow-up commits into the two base commits. PR now shows exactly the intended two-commit history:

• 428d631 PXB-3747 : Report final on-disk backup_size in xtrabackup_info
• 1d57a2c PXB-3747 : Report uncompressed_backup_size for --compress backups

What was folded:

• c935a9d ("Addressing review comments") split by hunk:
  • report_backup_size() doc reflow (backup_copy.cc), ds_write_sparse packed_len -> len opt (datasink.cc), and the "Finsh" -> "Finish" typo fix on backup_finish() -> Commit 2 (the typo fix's hunk context lives inside report_backup_size, which is Commit-2 territory).
  • pipe_ctxt = nullptr default, ds_create() cleanup, MY_ZEROFILL + xbstream fs_support_punch_hole = true + local_close trailing-byte accounting, sum_file_bytes -d guard -> Commit 1.
• 8d20d4b ("Address Jakub's review comments") split by hunk:
  • ds_file_t::uncomp_bytes = nullptr default -> Commit 2 (the field is introduced by Commit 2).
  • All other = nullptr defaults, ds_metric::name and ds_find_metric -> std::string_view, report_metrics out-param -> reference -> Commit 1.

Commit 2's "How to use this feature" section now shows Compression ratio: 3.70x (was 3.70), addressing Copilot's earlier flag.

Verification (byte-exact against the pre-fold tip):

$ git diff <new-tip> <pre-fold-safety-tag> -- storage/ test/ | wc -l
0

Re-ran both strict-size suites against the new history (Debug build, MySQL 8.4.8 (InnoDB 8.4.8) server):

• t/backup_size_basic.sh: passed.
• t/backup_size_compress.sh: passed (covers --compress, --compress + --encrypt, --compress + --stream=xbstream, --extra-lsndir, sparse, etc., all asserting byte-exact backup_size and uncompressed_backup_size).

Earlier per-thread review comments (Copilot's C1/C2; Jakub's J1/J2/J3/J4/J5/J6) reference commit IDs that no longer exist after the squash, so GitHub will mark them as outdated. The actual changes those comments were about are all in the two final commits.