Skip to content

bugfix to ensure resync cannot fail due to table size too large #4196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jgao54 merged 4 commits into from
Apr 20, 2026
Merged

bugfix to ensure resync cannot fail due to table size too large #4196

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f266e69
bugfix to ensure resync cannot fail due to table size too large
jgao54 Apr 18, 2026
7b23e3b
review
jgao54 Apr 20, 2026
3810857
unit test
jgao54 Apr 20, 2026
89abfbf
lint
jgao54 Apr 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion flow/connectors/clickhouse/normalize.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,9 +267,18 @@ func (c *ClickHouseConnector) generateCreateTableSQLForNormalizedTable(
}
}

chSettings := chinternal.NewCHSettings(chVersion)
if allowNullableKey {
stmtBuilder.WriteString(chinternal.NewCHSettingsString(chVersion, chinternal.SettingAllowNullableKey, "1"))
chSettings.Add(chinternal.SettingAllowNullableKey, "1")
}
if config.IsResync {
// CREATE OR REPLACE TABLE (used by resync) internally would create a NEW table with name
// <_tmp_replace_*>, atomic swap with the existing table, and then drop the existing table.
// Setting max_table_size_to_drop = 0 ensure the implicit drop can't fail, since it can be
// non-empty if a resync is triggered on top of another resync that is mid-snapshot.
chSettings.Add(chinternal.SettingMaxTableSizeToDrop, "0")
}
stmtBuilder.WriteString(chSettings.String())

if c.Config.Cluster != "" {
fmt.Fprintf(&stmtBuilderDistributed, " ENGINE = Distributed(%s,%s,%s",
Expand Down
Loading