Questions Regarding Backup Compression Size and Incremental Backups

[FeryET]

Hello.

I am testing the rest api for our backup pipeline, and I am a bit confused. So I ask the questions I have in the following lines. This is my config:

general:
  remote_storage: s3
  backups_to_keep_local: 3
  backups_to_keep_remote: 30
  log_level: info
  allow_empty_backups: false
  upload_by_part: true
  download_by_part: true
  use_resumable_state: true
  retries_on_failure: 5
  retries_pause: 1m
  watch_interval: 1h
  full_interval: 168h
  watch_backup_name_template: "shard{shard}-{type}-{time:RFC3339}"

clickhouse:
  username: <USERNAME>
  password: <PASSWORD>
  host: localhost
  port: 9000
  timeout: 5m
  sync_replicated_tables: true
  log_sql_queries: true
  debug: false
  config_dir:  "/etc/clickhouse-server"
  restart_command: "exec:docker restart clickhouse"
  ignore_not_exists_error_during_freeze: true
  check_replicas_before_attach: true
  default_replica_path: "/clickhouse/tables/{cluster}/{shard}/{database}/{table}" #?????????????????????????
  default_replica_name: "{replica}"
  max_connections: 0 #?

s3:
  access_key: <SECRET-KEY>
  secret_key: <ACCESS-KEY>
  bucket: <BUCKET>
  endpoint: <ENDPOINT>
  region: <REGION>
  force_path_style: true
  path: "<PATH>"
  compression_level: 5
  compression_format: zstd
  concurrency: 3
  max_parts_count: 1
  allow_multipart_download: true
  checksum_algorithm: "CRC32"
  debug: false

api:
  listen: "0.0.0.0:7171"
  enable_metrics: true
  enable_pprof: false
  username: "<API_USER>"
  password: "<API_PASSWORD>"
  secure: false
  allow_parallel: false
  create_integration_tables: true
  complete_resumable_after_restart: true
  watch_is_main_process: false

1. When does the backup server create a full backup instead of an incremental one? By which command I can control whether to create an incremental backup or a full backup?
2. How can I control the size of parts in the remote storage? The current backup configuration creates many parts in the remote storage, and the compression rate is low, despite setting compression level to 5 for zstd. It only compresses about 10% of overall size of the data. There are a lot of small files (around 2KBs) in the remote storage for each table, apparently there are the parts created. I need more info regarding how to tune this so we can optimize our network bandwidth usage.

[Slach]

When does the backup server create a full backup instead of an incremental one? By which command I can control whether to create an incremental backup or a full backup?

look
https://github.com/Altinity/clickhouse-backup/?tab=readme-ov-file#cli-command---create_remote
and
https://github.com/Altinity/clickhouse-backup/?tab=readme-ov-file#post-backupcreate

use
--diff-from-remote=full-backup-name for CLI and diff_from_remote get parameter for incremental backup

better use POST /backup/actions similar with CLI
or even use INSERT INTO system.backup_actions(command) VALUES('create_remote --diff-from-remote=full-backup-name')

look https://github.com/Altinity/clickhouse-backup/blob/master/Examples.md#how-to-use-clickhouse-backup-in-kubernetes for more detail examples

[Slach]

How can I control the size of parts in the remote storage? The current backup configuration creates many parts in the remote storage,

clickhouse-backup create the same parts on remote storage with the local parts

if you have many small parts on local storage, you will have many small parts on remote storage

think about the root reason is your INSERT ingestion pattern (read docs about async_insert for example) instead of trying to solve symptoms with clickhouse-backup