Problem
UploadPath and DownloadPath in pkg/storage/general.go transfer the files of a part strictly sequentially:
DownloadPath (pkg/storage/general.go:699) walks the remote prefix and downloads one file at a time inside bd.Walk(...).UploadPath (pkg/storage/general.go:766) iterates for _, filename := range files and calls PutFile one file at a time.
upload_concurrency / download_concurrency parallelize across parts/tables, but each part's files still move one-by-one. For wide MergeTree parts (dozens to hundreds of small/medium column files) against high-latency object storage (S3/GCS), per-file round-trip latency dominates and a single part transfers far below the available bandwidth. On multi-TiB backups with many wide tables this leaves most of the network idle.
Proposal
Add a file_transfer_concurrency option (general section, env FILE_TRANSFER_CONCURRENCY) and parallel variants UploadPathParallel / DownloadPathParallel (plus a manifest-aware DownloadPathParallelWithManifest) that transfer the files of a single part with N concurrent workers using errgroup:
- Call sites in
pkg/backup/upload.go and pkg/backup/download.go pick the parallel variant when file_transfer_concurrency > 1, otherwise the existing sequential code path runs unchanged.
- Bandwidth throttling still honors
upload_max_bytes_per_second / download_max_bytes_per_second as an aggregate cap: one shared bwlimit.Limiter is created per path-transfer and shared by all workers, so parallelism does not multiply the configured max speed.
- Default
file_transfer_concurrency: 1 preserves current behavior exactly (opt-in).
We run this in a production fork backing up multi-TiB ClickHouse clusters; file-level parallelism is the single biggest wall-clock win we measured for wide tables (parts with many files) on object storage.
I have a PR ready to submit.
Problem
UploadPathandDownloadPathinpkg/storage/general.gotransfer the files of a part strictly sequentially:DownloadPath(pkg/storage/general.go:699) walks the remote prefix and downloads one file at a time insidebd.Walk(...).UploadPath(pkg/storage/general.go:766) iteratesfor _, filename := range filesand callsPutFileone file at a time.upload_concurrency/download_concurrencyparallelize across parts/tables, but each part's files still move one-by-one. For wide MergeTree parts (dozens to hundreds of small/medium column files) against high-latency object storage (S3/GCS), per-file round-trip latency dominates and a single part transfers far below the available bandwidth. On multi-TiB backups with many wide tables this leaves most of the network idle.Proposal
Add a
file_transfer_concurrencyoption (generalsection, envFILE_TRANSFER_CONCURRENCY) and parallel variantsUploadPathParallel/DownloadPathParallel(plus a manifest-awareDownloadPathParallelWithManifest) that transfer the files of a single part with N concurrent workers usingerrgroup:pkg/backup/upload.goandpkg/backup/download.gopick the parallel variant whenfile_transfer_concurrency > 1, otherwise the existing sequential code path runs unchanged.upload_max_bytes_per_second/download_max_bytes_per_secondas an aggregate cap: one sharedbwlimit.Limiteris created per path-transfer and shared by all workers, so parallelism does not multiply the configured max speed.file_transfer_concurrency: 1preserves current behavior exactly (opt-in).We run this in a production fork backing up multi-TiB ClickHouse clusters; file-level parallelism is the single biggest wall-clock win we measured for wide tables (parts with many files) on object storage.
I have a PR ready to submit.