Skip to content

Commit 7f72af5

Browse files
fix(clickhouse): drop orphaned shard table after mirror resync exchange (#4490)
## Problem After a mirror resync on a clustered ClickHouse destination, the old local shard table (e.g. `itunes_apps_shard`) is left behind as an orphan. During resync, `RenameTables` does: 1. `EXCHANGE TABLES itunes_apps AND itunes_apps_resync` — swaps the two Distributed tables 2. `DROP TABLE itunes_apps_resync` — drops the now-old Distributed shell But this only handles the Distributed layer. The local shard that the original `itunes_apps` pointed to (`itunes_apps_shard`) has no Distributed table referencing it anymore and is never dropped, leaving it as dead storage. ## Fix Before the `EXCHANGE`, resolve the shard table name from the original Distributed table's `engine_full` definition using the existing `getDistributedShardTable` helper. After the exchange and Distributed drop succeed, explicitly drop that shard table. If the shard lookup fails (e.g. non-Distributed engine, single-node deployment), we log a warning and continue — the swap itself is not blocked. ## Test plan - [ ] Run a mirror resync on a clustered ClickHouse destination - [ ] Confirm `itunes_apps_shard` (or equivalent old shard) is gone after resync completes - [ ] Confirm `itunes_apps` Distributed table still points to the new timestamped shard and data is intact
1 parent c21a707 commit 7f72af5

1 file changed

Lines changed: 21 additions & 0 deletions

File tree

  • flow/connectors/clickhouse

flow/connectors/clickhouse/cdc.go

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -331,6 +331,18 @@ func (c *ClickHouseConnector) RenameTables(
331331
}
332332

333333
if originalTableExists {
334+
// Before swapping, resolve the shard table that the original Distributed table currently points to.
335+
// After the EXCHANGE the Distributed table is dropped, leaving that shard orphaned unless we drop it explicitly.
336+
var originalShardTable string
337+
if c.Config.Cluster != "" {
338+
originalShardTable, err = c.getDistributedShardTable(ctx, renameRequest.NewName)
339+
if err != nil {
340+
c.logger.Warn("could not resolve shard table for original distributed table, old shard may be left behind",
341+
slog.String("table", renameRequest.NewName), slog.Any("error", err))
342+
originalShardTable = ""
343+
}
344+
}
345+
334346
// target table exists, so we can attempt to swap. In most cases, we will have Atomic engine,
335347
// which supports a special query to exchange two tables, allowing dependent (materialized) views and dictionaries on these tables
336348
c.logger.Info("attempting atomic exchange",
@@ -344,6 +356,15 @@ func (c *ClickHouseConnector) RenameTables(
344356
); err != nil {
345357
return nil, fmt.Errorf("unable to drop exchanged table %s: %w", renameRequest.CurrentName, err)
346358
}
359+
// Drop the original shard table that is now orphaned after the Distributed table was exchanged and dropped.
360+
if originalShardTable != "" {
361+
if err := c.execWithLogging(ctx,
362+
fmt.Sprintf(dropTableSQLWithCHSetting, peerdb_clickhouse.QuoteIdentifier(originalShardTable), onCluster),
363+
); err != nil {
364+
return nil, fmt.Errorf("unable to drop orphaned shard table %s: %w", originalShardTable, err)
365+
}
366+
c.logger.Info("dropped orphaned shard table after resync exchange", slog.String("shardTable", originalShardTable))
367+
}
347368
} else if ex, ok := err.(*clickhouse.Exception); !ok || chproto.Error(ex.Code) != chproto.ErrNotImplemented {
348369
// move on to the fallback code if unimplemented, in all other error codes / types return,
349370
// since we know/assume exchange would be the sensible action

0 commit comments

Comments
 (0)