Skip to content

hardlinkIfLocalPartExistsAndChecksumEqual re-globs and re-hashes checksums.txt per part — build a checksum index once per download #1457

Description

@minguyen9988

Problem

hardlinkIfLocalPartExistsAndChecksumEqual (pkg/backup/download.go:978) is called per downloaded part and, on every call:

  1. globs the local disk for candidate part directories in other local backups, and
  2. re-reads and re-computes the CRC64 of checksums.txt for each candidate.

That makes hardlink dedup O(parts x candidate paths) file reads/CRCs per download. On hosts that keep several local backups of large clusters (hundreds of thousands of parts), the repeated globbing and checksum recomputation adds substantial wall time and IO to every download --hardlink-exists-files run — the same checksums.txt files get re-hashed over and over.

Proposal

Build a checksum index once at download start:

  • Scan local backup metadata JSONs (each part's checksums_crc64 is already recorded there by create), and build an in-memory map (table, part-name, crc64) -> local path.
  • hardlinkIfLocalPartExistsAndChecksumEqual first consults the index: O(1) map lookup instead of glob + re-hash.
  • If the index is nil (not built) or has no entry, fall back to the existing per-part glob/CRC path — behavior is identical, just faster in the common case.

Extracted from a production fork used to back up multi-TiB ClickHouse clusters.

I have a PR ready to submit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions