Fix xbcloud put "out-of-order chunk" failure on sparse chunks (page-compressed tables)

[egezonberisha]

Note on process: GitHub issues are disabled on this repo and I was unable to create an account on perconadev.atlassian.net, so this PR doubles as the bug report. The same root cause was previously reported in PXB-2958 (closed Incomplete when the reporter went silent) — please feel free to reopen/link a JIRA ticket to this PR.

Problem

xtrabackup --backup --stream=xbstream | xbcloud put fails deterministically with

xbcloud: out-of-order chunk: real offset = 0x..., expected offset = 0x...

whenever the instance contains a tablespace using InnoDB page compression (CREATE TABLE ... COMPRESSION="zlib") whose .ibd produces a mix of sparse and non-sparse xbstream chunks. The stream itself is valid — piping the identical stream into xbstream -x extracts correctly — the failure is in xbcloud put's offset bookkeeping. Observed in production on 8.4.0-6; the affected code is identical on current 8.0, 8.4, and trunk.

Root cause

For page-compressed tablespaces, write_ibd_buffer() builds a sparse map from page contents and the backup emits XB_CHUNK_TYPE_SPARSE chunks (backup_copy.cc#L502-L545; the xbstream datasink always advertises sparse support, ds_xbstream.cc#L94-L96).

On the write side, the per-file stream offset advances by payload length plus the sum of the sparse-map holes (xbstream_write.cc#L258-L260), so the next chunk's offset field on the wire jumps over the holes.

xbstream -x (mode_extract) reads this back correctly — it advances its expected offset by chunk.length plus every sparse_map[i].skip (xbstream.cc#L625-L638).

xbcloud put (put_func) does not:

• it advances entry->offset += chunk.length unconditionally, ignoring the sparse map (xbcloud.cc#L912);
• it only performs the offset (and checksum) check for XB_CHUNK_TYPE_PAYLOAD chunks (xbcloud.cc#L863-L877).

So while a file streams as SPARSE chunks, entry->offset silently falls behind the real offset by the accumulated hole size. The moment the file produces a plain PAYLOAD chunk — the first --read-buffer-size (default 10 MiB) window containing zero compressed pages, e.g. a run of incompressible data — the check fires and xbcloud put aborts. The gap between "real" and "expected" in the error message equals the total punched-hole bytes up to that point.

Because the failure only occurs when a payload chunk follows sparse chunks of the same file, it is data-dependent: a table can back up fine for months and then fail every run once its data distribution shifts. This is why the error is easy to misdiagnose as network/S3 flakiness — see PXB-3572 and this forum thread (same signature, closed Not a Bug after being attributed to network issues; --curl-retriable-errors cannot fix it, and retries leave partial uploads in the bucket).

Fix

Mirror mode_extract's sparse handling in put_func():

1. validate checksum and offset for XB_CHUNK_TYPE_SPARSE chunks too (xb_stream_validate_checksum() already supports them — xb_stream_read_chunk() seeds checksum_part with the sparse map's CRC, xbstream_read.cc#L270-L288);
2. advance the expected offset by the sparse-map skips in addition to chunk.length;
3. free chunk.sparse_map before the per-iteration memset reset — chunk.raw_data's ownership moves to the upload buffer, but the sparse map (a separate allocation) does not, so it was leaked for every sparse chunk processed (possibly what PXB-3189 observed).

No stored-object format change: xbcloud put uploads raw chunk bytes, so backups uploaded with this fix restore with existing xbcloud get | xbstream -x, and previously-uploaded backups are unaffected.

The change mirrors existing, exercised logic from xbstream.cc; I have not run it against a full build, so please lean on CI/review accordingly.

How to reproduce

Any 8.0/8.4 server with the datadir on a punch-hole-capable filesystem (ext4/xfs, default innodb_page_size=16k), plus any S3 endpoint (local MinIO is fine):

CREATE DATABASE repro;
CREATE TABLE repro.t1 (id BIGINT AUTO_INCREMENT PRIMARY KEY, pad LONGBLOB) COMPRESSION="zlib";

SET SESSION cte_max_recursion_depth = 100000;

-- ~240 MiB of highly compressible pages -> punched holes -> SPARSE chunks
INSERT INTO repro.t1(pad)
  WITH RECURSIVE seq(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM seq WHERE n < 4000)
  SELECT REPEAT('a', 60000) FROM seq;

-- ~30 MiB of incompressible pages appended after them -> at least one full
-- 10 MiB read-buffer window with no compressed page -> a PAYLOAD chunk.
-- NB: must NOT use the default block_encryption_mode (aes-128-ecb): ECB of
-- repeated plaintext blocks is itself repetitive and compresses (a 16 KiB
-- page of it deflates to ~84 bytes), so the pages would stay page-compressed.
-- CBC with a random IV is incompressible.
SET SESSION block_encryption_mode = 'aes-256-cbc';
INSERT INTO repro.t1(pad)
  WITH RECURSIVE seq(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM seq WHERE n < 500)
  SELECT AES_ENCRYPT(REPEAT('a', 60000), CONCAT('k', n), RANDOM_BYTES(16)) FROM seq;

xtrabackup --backup --stream=xbstream --target-dir=/tmp/lsn \
  | xbcloud put --storage=s3 --s3-endpoint=http://127.0.0.1:9000 \
      --s3-access-key=... --s3-secret-key=... --s3-bucket=test \
      --parallel=4 repro-sparse
# -> xbcloud: out-of-order chunk: real offset = 0x..., expected offset = 0x...

# control: the identical stream is valid
xtrabackup --backup --stream=xbstream --target-dir=/tmp/lsn2 | xbstream -x -C /tmp/extract
# -> succeeds

Happy to contribute this reproducer as a regression test if you can point me at the preferred harness for xbcloud tests.

Workarounds for affected users (until fixed)

• Add --compress=zstd (or lz4) to xtrabackup: the compression datasink does not implement sparse writes, so the stream contains no SPARSE chunks; sparseness is restored at extract time via restore_sparseness(). Restore needs xbstream -x --decompress.
• Or stop using page compression on the affected tables (ALTER TABLE ... COMPRESSION='none' + OPTIMIZE TABLE).

🤖 Generated with Claude Code

[it-percona-cla]

All committers have signed the CLA.