Problem
hardlinkIfLocalPartExistsAndChecksumEqual (pkg/backup/download.go:978) is called per downloaded part and, on every call:
- globs the local disk for candidate part directories in other local backups, and
- re-reads and re-computes the CRC64 of
checksums.txt for each candidate.
That makes hardlink dedup O(parts x candidate paths) file reads/CRCs per download. On hosts that keep several local backups of large clusters (hundreds of thousands of parts), the repeated globbing and checksum recomputation adds substantial wall time and IO to every download --hardlink-exists-files run — the same checksums.txt files get re-hashed over and over.
Proposal
Build a checksum index once at download start:
- Scan local backup metadata JSONs (each part's
checksums_crc64 is already recorded there by create), and build an in-memory map (table, part-name, crc64) -> local path.
hardlinkIfLocalPartExistsAndChecksumEqual first consults the index: O(1) map lookup instead of glob + re-hash.
- If the index is nil (not built) or has no entry, fall back to the existing per-part glob/CRC path — behavior is identical, just faster in the common case.
Extracted from a production fork used to back up multi-TiB ClickHouse clusters.
I have a PR ready to submit.
Problem
hardlinkIfLocalPartExistsAndChecksumEqual(pkg/backup/download.go:978) is called per downloaded part and, on every call:checksums.txtfor each candidate.That makes hardlink dedup O(parts x candidate paths) file reads/CRCs per download. On hosts that keep several local backups of large clusters (hundreds of thousands of parts), the repeated globbing and checksum recomputation adds substantial wall time and IO to every
download --hardlink-exists-filesrun — the samechecksums.txtfiles get re-hashed over and over.Proposal
Build a checksum index once at download start:
checksums_crc64is already recorded there by create), and build an in-memory map(table, part-name, crc64) -> local path.hardlinkIfLocalPartExistsAndChecksumEqualfirst consults the index: O(1) map lookup instead of glob + re-hash.Extracted from a production fork used to back up multi-TiB ClickHouse clusters.
I have a PR ready to submit.