Skip to content

add --disk-limit protection with table-filtered download size estimation #1458

Description

@minguyen9988

Problem

The only disk-space protection on download/restore_remote is CheckDisksUsage (pkg/backup/backuper.go:613, added for #878), which compares free space against the whole backup size. Two gaps:

  1. No headroom guarantee. A download can legally fill a disk to 100%, taking ClickHouse down with it. Operators typically want "never push the data disk past N%".
  2. Whole-backup size vs filtered download. When downloading with --tables db.one_table from a multi-TiB backup, the space check still uses the full backup size, refusing downloads that would actually only fetch a few GiB.

Proposal

  • Add --disk-limit <pct> (1–100) to download and restore_remote CLI commands + the corresponding server API query parameter. 0/absent = disabled (default, current behavior unchanged).
  • CheckDiskLimit computes projected usage per data disk: (used + downloadSize) / total * 100, and refuses the download with a clear error naming the disk, projected pct, and limit.
  • estimateFilteredDownloadSize estimates the size actually downloaded honoring --tables/--partitions filtering (per-table metadata sizes for the tables that match), so single-table downloads from huge backups aren't blocked.
  • Refactor CheckDisksUsage into CheckDisksUsageFiltered(backup, disks, isResumeExists, downloadSize) so the free-space check also benefits from the filtered estimate (the old signature stays as a thin wrapper).

Extracted from a production fork used to back up multi-TiB ClickHouse clusters, where an unguarded restore once filled a data disk.

I have a PR ready to submit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions