Intermittent error while trying to restore

[gavishdewangan]

I am getting this error sometimes when I try to restore from a remote backup:

one of restoreDataRegular go-routine return error: can't attach data parts for table 'table': table can't restore cause system.replicas entries already exists and replication in progress from another replica, log_pointer=12, log_max_index=11, absolute_delay=0, queue_size=0

As of now I am creating backup and then without dropping trying to restore, thought insertions are not happening to the database.

Is not dropping the database an issue before restore?

Also any other reason why we encounter this error?

What all possible reasons can be that we encounter this error while restore?

[Slach]

did you run restore command concurrenlty in multiple replicas?

could you follow proper commands sequence
https://github.com/Altinity/clickhouse-backup/blob/master/Examples.md#RESTORE

[Slach]

do you have only one tabe with ReplicatedMergeTree engine?

check engine_full

cluser-01 can contains tables which contains the same replication path with cluster-02 (after macros resolve)
this is could be root reason

are you sure you always use {cluster} and {shard} macros in ReplicatedMergeTree?

check
SELECT hostName(), * FROM cluster('all-sharded',system.server_settings) WHERE name='default_replica_path'

[gavishdewangan]

no we have a lot of tables, it is not possible to share all of them here so shared the relevant one

yes we always use {cluster} and {shard} macros, so I don't think it is not possible that cluster-01 and cluster-02 tables can have same replication path

[gavishdewangan]

   ┌─hostName()────────────────────────────────┬─name─────────────────┬─value─────────────────────────────┬─default───────────────────────────┬─changed─┬─description────────────────────────┬─type───┬─changeable_without_restart─┬─is_obsolete─┐
1. │ chi-clickhouse-server-cnxcluster-02-0-0-0 │ default_replica_path │ /clickhouse/tables/{uuid}/{shard} │ /clickhouse/tables/{uuid}/{shard} │       0 │ The path to the table in ZooKeeper │ String │ No                         │           0 │
   └───────────────────────────────────────────┴──────────────────────┴───────────────────────────────────┴───────────────────────────────────┴─────────┴────────────────────────────────────┴────────┴────────────────────────────┴─────────────┘
   ┌─hostName()────────────────────────────────┬─name─────────────────┬─value─────────────────────────────┬─default───────────────────────────┬─changed─┬─description────────────────────────┬─type───┬─changeable_without_restart─┬─is_obsolete─┐
2. │ chi-clickhouse-server-cnxcluster-01-0-0-0 │ default_replica_path │ /clickhouse/tables/{uuid}/{shard} │ /clickhouse/tables/{uuid}/{shard} │       0 │ The path to the table in ZooKeeper │ String │ No                         │           0 │
   └───────────────────────────────────────────┴──────────────────────┴───────────────────────────────────┴───────────────────────────────────┴─────────┴────────────────────────────────────┴────────┴────────────────────────────┴─────────────┘
   ┌─hostName()────────────────────────────────┬─name─────────────────┬─value─────────────────────────────┬─default───────────────────────────┬─changed─┬─description────────────────────────┬─type───┬─changeable_without_restart─┬─is_obsolete─┐
3. │ chi-clickhouse-server-cnxcluster-01-1-0-0 │ default_replica_path │ /clickhouse/tables/{uuid}/{shard} │ /clickhouse/tables/{uuid}/{shard} │       0 │ The path to the table in ZooKeeper │ String │ No                         │           0 │
   └───────────────────────────────────────────┴──────────────────────┴───────────────────────────────────┴───────────────────────────────────┴─────────┴────────────────────────────────────┴────────┴────────────────────────────┴─────────────┘
   ┌─hostName()────────────────────────────────┬─name─────────────────┬─value─────────────────────────────┬─default───────────────────────────┬─changed─┬─description────────────────────────┬─type───┬─changeable_without_restart─┬─is_obsolete─┐
4. │ chi-clickhouse-server-cnxcluster-02-0-1-0 │ default_replica_path │ /clickhouse/tables/{uuid}/{shard} │ /clickhouse/tables/{uuid}/{shard} │       0 │ The path to the table in ZooKeeper │ String │ No                         │           0 │
   └───────────────────────────────────────────┴──────────────────────┴───────────────────────────────────┴───────────────────────────────────┴─────────┴────────────────────────────────────┴────────┴────────────────────────────┴─────────────┘
   ┌─hostName()────────────────────────────────┬─name─────────────────┬─value─────────────────────────────┬─default───────────────────────────┬─changed─┬─description────────────────────────┬─type───┬─changeable_without_restart─┬─is_obsolete─┐
5. │ chi-clickhouse-server-cnxcluster-02-1-0-0 │ default_replica_path │ /clickhouse/tables/{uuid}/{shard} │ /clickhouse/tables/{uuid}/{shard} │       0 │ The path to the table in ZooKeeper │ String │ No                         │           0 │
   └───────────────────────────────────────────┴──────────────────────┴───────────────────────────────────┴───────────────────────────────────┴─────────┴────────────────────────────────────┴────────┴────────────────────────────┴─────────────┘
   ┌─hostName()────────────────────────────────┬─name─────────────────┬─value─────────────────────────────┬─default───────────────────────────┬─changed─┬─description────────────────────────┬─type───┬─changeable_without_restart─┬─is_obsolete─┐
6. │ chi-clickhouse-server-cnxcluster-02-1-1-0 │ default_replica_path │ /clickhouse/tables/{uuid}/{shard} │ /clickhouse/tables/{uuid}/{shard} │       0 │ The path to the table in ZooKeeper │ String │ No                         │           0 │
   └───────────────────────────────────────────┴──────────────────────┴───────────────────────────────────┴───────────────────────────────────┴─────────┴────────────────────────────────────┴────────┴────────────────────────────┴─────────────┘
   ┌─hostName()────────────────────────────────┬─name─────────────────┬─value─────────────────────────────┬─default───────────────────────────┬─changed─┬─description────────────────────────┬─type───┬─changeable_without_restart─┬─is_obsolete─┐
7. │ chi-clickhouse-server-cnxcluster-01-0-1-0 │ default_replica_path │ /clickhouse/tables/{uuid}/{shard} │ /clickhouse/tables/{uuid}/{shard} │       0 │ The path to the table in ZooKeeper │ String │ No                         │           0 │
   └───────────────────────────────────────────┴──────────────────────┴───────────────────────────────────┴───────────────────────────────────┴─────────┴────────────────────────────────────┴────────┴────────────────────────────┴─────────────┘
   ┌─hostName()────────────────────────────────┬─name─────────────────┬─value─────────────────────────────┬─default───────────────────────────┬─changed─┬─description────────────────────────┬─type───┬─changeable_without_restart─┬─is_obsolete─┐
8. │ chi-clickhouse-server-cnxcluster-01-1-1-0 │ default_replica_path │ /clickhouse/tables/{uuid}/{shard} │ /clickhouse/tables/{uuid}/{shard} │       0 │ The path to the table in ZooKeeper │ String │ No                         │           0 │
   └───────────────────────────────────────────┴──────────────────────┴───────────────────────────────────┴───────────────────────────────────┴─────────┴────────────────────────────────────┴────────┴────────────────────────────┴─────────────┘

[Slach]

could you share?
SELECT * FROM system.zookeeper WHERE path='/clickhouse/tables/{cluster}/{shard}/db/table' FORMAT Vertical

as temporary workaround use
restore_remote --rm --data --env=CLICKHOUSE_CHECK_REPLICAS_BEFORE_ATTACH=false

[gavishdewangan]

Row 1:
──────
name:  alter_partition_version
value: 
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 2:
──────
name:  metadata
value: metadata format version: 1
date column: 
sampling expression: 
index granularity: 8192
mode: 0
sign column: 
primary key: path
data format version: 1
partition key: 
sorting key: path, file_name
granularity bytes: 10485760
merge parameters format version: 2

path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 3:
──────
name:  temp
value: 
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 4:
──────
name:  table_shared_id
value: 9695d502-
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 5:
──────
name:  log
value: 
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 6:
──────
name:  leader_election
value: 
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 7:
──────
name:  columns
value: columns format version: 1
6 columns:
`start_time` DateTime64(3)
`end_time` DateTime64(3)
`path` String
`file_name` String
`table_name` String
`creation_date_time` DateTime64(3)

path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 8:
──────
name:  blocks
value: 
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 9:
───────
name:  async_blocks
value: 
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 10:
───────
name:  nonincrement_block_numbers
value: 
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 11:
───────
name:  replicas
value: last added replica: chi-clickhouse-server-cnxcluster-02-0-0
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 12:
───────
name:  quorum
value: 
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 13:
───────
name:  pinned_part_uuids
value: {"part_uuids":"[]"}
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 14:
───────
name:  block_numbers
value: 
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 15:
───────
name:  mutations
value: 
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 16:
───────
name:  part_moves_shard
value: 
path:  /clickhouse/tables/cnxcluster-02/0/db/table

Row 17:
───────
name:  lost_part_count
value: 1
path:  /clickhouse/tables/cnxcluster-02/0/db/table

17 rows in set. Elapsed: 0.005 sec.

[Slach]

SELECT * FROM system.zookeeper WHERE path='/clickhouse/tables/{cluster}/{shard}/db/table/block_numbers' FORMAT Vertical

[gavishdewangan]

`
Row 1:
──────
name: all
value:
path: /clickhouse/tables/cnxcluster-02/0/db/table/block_numbers

1 row in set. Elapsed: 0.004 sec.

`