Skip to content

fix(clickpipes): change excluded columns type from list to set to avoid ordering issue #560

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tpanetti wants to merge 4 commits into
base: main
Choose a base branch
from
Open

fix(clickpipes): change excluded columns type from list to set to avoid ordering issue #560

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c874ebd
fix(clickpipes): change excluded columns type from list to set to avo…
tpanetti Jun 1, 2026
13f5072
Merge branch 'main' into tp/clickpipes-cdc-excluded-column-ordering-fix
tpanetti Jun 1, 2026
1fa3e7a
Address PR comments
tpanetti Jun 1, 2026
9c5a159
update docs
tpanetti Jun 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions docs/resources/clickpipe.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@ Required:

Optional:

- `excluded_columns` (List of String) Columns to exclude from replication.
- `excluded_columns` (Set of String) Columns to exclude from replication.
- `sorting_keys` (List of String) Ordered list of columns to use as sorting key for the target table. Required when use_custom_sorting_key is true.
- `table_engine` (String) Table engine to use for the target table. (`MergeTree`, `ReplacingMergeTree`, `Null`)
- `use_custom_sorting_key` (Boolean) Whether to use a custom sorting key for the target table.
Expand Down Expand Up @@ -246,7 +246,7 @@ Optional:
- `certificate` (String, Sensitive) PEM encoded client certificate for mTLS authentication. Use with `MUTUAL_TLS` authentication.
- `connection_string` (String, Sensitive) The connection string for the Kafka source. Use with `azureeventhub` Kafka source type. Use with `PLAIN` authentication.
- `password` (String, Sensitive) The password for the Kafka source. Use `password_wo` instead to keep the value out of state.
- `password_wo` (String, Sensitive) Write-only password for the Kafka source. Not persisted to state. Pair with `password_wo_version` to trigger updates.
- `password_wo` (String, Sensitive, [Write-only](https://developer.hashicorp.com/terraform/language/resources/ephemeral#write-only-arguments)) Write-only password for the Kafka source. Not persisted to state. Pair with `password_wo_version` to trigger updates.
- `password_wo_version` (Number) Version trigger for `password_wo`. Increment to push a new password to the API.
- `private_key` (String, Sensitive) PEM encoded client private key for mTLS authentication. Use with `MUTUAL_TLS` authentication.
- `secret_key` (String, Sensitive) The secret key for the Kafka source. Use with `IAM_USER` authentication.
Expand Down Expand Up @@ -284,7 +284,7 @@ Required:
Optional:

- `password` (String, Sensitive) The password for the Schema Registry. Either `password` or `password_wo` must be provided.
- `password_wo` (String, Sensitive) Write-only password for the Schema Registry. Not persisted to state. Pair with `password_wo_version` to trigger updates.
- `password_wo` (String, Sensitive, [Write-only](https://developer.hashicorp.com/terraform/language/resources/ephemeral#write-only-arguments)) Write-only password for the Schema Registry. Not persisted to state. Pair with `password_wo_version` to trigger updates.
- `password_wo_version` (Number) Version trigger for `password_wo`. Increment to push a new password to the API.


Expand Down Expand Up @@ -376,7 +376,7 @@ Required:
Optional:

- `password` (String, Sensitive) The password for the MongoDB instance. Use `password_wo` instead to keep the value out of state.
- `password_wo` (String, Sensitive) Write-only password for the MongoDB instance. Not persisted to state. Pair with `password_wo_version` to trigger updates.
- `password_wo` (String, Sensitive, [Write-only](https://developer.hashicorp.com/terraform/language/resources/ephemeral#write-only-arguments)) Write-only password for the MongoDB instance. Not persisted to state. Pair with `password_wo_version` to trigger updates.
- `password_wo_version` (Number) Version trigger for `password_wo`. Increment to push a new password to the API.


Expand Down Expand Up @@ -412,7 +412,7 @@ Required:
Optional:

- `password` (String, Sensitive) The password for the MySQL instance. Use `password_wo` instead to keep the value out of state.
- `password_wo` (String, Sensitive) Write-only password for the MySQL instance. Not persisted to state. Pair with `password_wo_version` to trigger updates.
- `password_wo` (String, Sensitive, [Write-only](https://developer.hashicorp.com/terraform/language/resources/ephemeral#write-only-arguments)) Write-only password for the MySQL instance. Not persisted to state. Pair with `password_wo_version` to trigger updates.
- `password_wo_version` (Number) Version trigger for `password_wo`. Increment to push a new password to the API.


Expand Down Expand Up @@ -447,7 +447,7 @@ Required:

Optional:

- `excluded_columns` (List of String) Columns to exclude from replication.
- `excluded_columns` (Set of String) Columns to exclude from replication.
- `partition_key` (String) Custom partitioning column used for parallel snapshotting. Must be an indexed column of integer, date, datetime, or timestamp type.
- `sorting_keys` (List of String) Ordered list of columns to use as sorting key for the target table. Required when use_custom_sorting_key is true.
- `table_engine` (String) Table engine to use for the target table. (`MergeTree`, `ReplacingMergeTree`, `Null`)
Expand Down Expand Up @@ -518,7 +518,7 @@ Required:
Optional:

- `password` (String, Sensitive) The password for the Postgres instance. Use `password_wo` instead to keep the value out of state.
- `password_wo` (String, Sensitive) Write-only password for the Postgres instance. Not persisted to state. Pair with `password_wo_version` to trigger updates.
- `password_wo` (String, Sensitive, [Write-only](https://developer.hashicorp.com/terraform/language/resources/ephemeral#write-only-arguments)) Write-only password for the Postgres instance. Not persisted to state. Pair with `password_wo_version` to trigger updates.
- `password_wo_version` (Number) Version trigger for `password_wo`. Increment to push a new password to the API.


Expand Down Expand Up @@ -554,7 +554,7 @@ Required:

Optional:

- `excluded_columns` (List of String) Columns to exclude from replication.
- `excluded_columns` (Set of String) Columns to exclude from replication.
- `partition_key` (String) Custom partitioning column used for parallel snapshotting. Only beneficial for PostgreSQL 13 (no benefit for PG14+, which supports indexed ctid scans). Must be an indexed column of type: `smallint`, `integer`, `bigint`, `timestamp without time zone`, or `timestamp with time zone`. Unrelated to ClickHouse partitioning.
- `sorting_keys` (List of String) Ordered list of columns to use as sorting key for the target table. Required when use_custom_sorting_key is true.
- `table_engine` (String) Table engine to use for the target table. (`MergeTree`, `ReplacingMergeTree`, `Null`)
Expand Down
4 changes: 3 additions & 1 deletion docs/resources/service.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ resource "clickhouse_service" "service" {

### Optional

> **NOTE**: [Write-only arguments](https://developer.hashicorp.com/terraform/language/resources/ephemeral#write-only-arguments) are supported in Terraform 1.11 and later.

- `backup_configuration` (Attributes) Configuration of service backup settings. (see [below for nested schema](#nestedatt--backup_configuration))
- `backup_id` (String) ID of the backup to restore when creating new service. If specified, the service will be created as a restore operation
- `byoc_id` (String) BYOC ID related to the cloud provider account you want to create this service into.
Expand All @@ -75,7 +77,7 @@ resource "clickhouse_service" "service" {
- `num_replicas` (Number) Number of replicas for the service.
- `password` (String, Sensitive) Password for the default user. One of either `password`, `password_wo`, or `password_hash` must be specified.
- `password_hash` (String, Sensitive) SHA256 hash of password for the default user. One of either `password`, `password_wo`, or `password_hash` must be specified.
- `password_wo` (String, Sensitive) Password for the default user (write-only, not persisted to state). Use this instead of `password` to avoid storing the password hash in Terraform state.
- `password_wo` (String, Sensitive, [Write-only](https://developer.hashicorp.com/terraform/language/resources/ephemeral#write-only-arguments)) Password for the default user (write-only, not persisted to state). Use this instead of `password` to avoid storing the password hash in Terraform state.
- `password_wo_version` (Number) Version number for password_wo. Increment this to trigger a password update when using password_wo.
- `query_api_endpoints` (Attributes) Configuration of the query API endpoints feature. (see [below for nested schema](#nestedatt--query_api_endpoints))
- `readonly` (Boolean) Indicates if this service should be read only. Only allowed for secondary services, those which share data with another service (i.e. when `warehouse_id` field is set).
Expand Down
42 changes: 23 additions & 19 deletions pkg/resource/clickpipe.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import (
"github.com/hashicorp/terraform-plugin-framework/resource/schema/listplanmodifier"
"github.com/hashicorp/terraform-plugin-framework/resource/schema/objectplanmodifier"
"github.com/hashicorp/terraform-plugin-framework/resource/schema/planmodifier"
"github.com/hashicorp/terraform-plugin-framework/resource/schema/setdefault"
"github.com/hashicorp/terraform-plugin-framework/resource/schema/stringdefault"
"github.com/hashicorp/terraform-plugin-framework/resource/schema/stringplanmodifier"
"github.com/hashicorp/terraform-plugin-framework/schema/validator"
Expand Down Expand Up @@ -1041,7 +1042,7 @@ func (c *ClickPipeResource) Schema(_ context.Context, _ resource.SchemaRequest,
Description: "Target table name in ClickHouse.",
Required: true,
},
"excluded_columns": schema.ListAttribute{
"excluded_columns": schema.SetAttribute{
Description: "Columns to exclude from replication.",
Optional: true,
ElementType: types.StringType,
Expand Down Expand Up @@ -1321,7 +1322,7 @@ func (c *ClickPipeResource) Schema(_ context.Context, _ resource.SchemaRequest,
Description: "Target table name in ClickHouse.",
Required: true,
},
"excluded_columns": schema.ListAttribute{
"excluded_columns": schema.SetAttribute{
Description: "Columns to exclude from replication.",
Optional: true,
ElementType: types.StringType,
Expand Down Expand Up @@ -1460,11 +1461,11 @@ func (c *ClickPipeResource) Schema(_ context.Context, _ resource.SchemaRequest,
Description: "Target table name in ClickHouse.",
Required: true,
},
"excluded_columns": schema.ListAttribute{
"excluded_columns": schema.SetAttribute{
Description: "Columns to exclude from replication.",
Optional: true,
Computed: true,
Default: listdefault.StaticValue(types.ListNull(types.StringType)),
Default: setdefault.StaticValue(types.SetNull(types.StringType)),
ElementType: types.StringType,
},
"use_custom_sorting_key": schema.BoolAttribute{
Expand Down Expand Up @@ -4153,16 +4154,16 @@ func (c *ClickPipeResource) syncClickPipeState(ctx context.Context, state *model
}

if len(mapping.ExcludedColumns) > 0 {
excludedColsList := make([]attr.Value, len(mapping.ExcludedColumns))
excludedCols := make([]attr.Value, len(mapping.ExcludedColumns))
for j, col := range mapping.ExcludedColumns {
excludedColsList[j] = types.StringValue(col)
excludedCols[j] = types.StringValue(col)
}
tableMappingModel.ExcludedColumns, _ = types.ListValue(types.StringType, excludedColsList)
tableMappingModel.ExcludedColumns, _ = types.SetValue(types.StringType, excludedCols)
} else if hasStateMapping && !stateMapping.ExcludedColumns.IsNull() {
// Preserve empty list from state (vs null) to avoid plan diff
tableMappingModel.ExcludedColumns, _ = types.ListValue(types.StringType, []attr.Value{})
// Preserve empty set from state (vs null) to avoid plan diff
tableMappingModel.ExcludedColumns, _ = types.SetValue(types.StringType, []attr.Value{})
} else {
tableMappingModel.ExcludedColumns = types.ListNull(types.StringType)
tableMappingModel.ExcludedColumns = types.SetNull(types.StringType)
}

if mapping.UseCustomSortingKey != nil {
Expand Down Expand Up @@ -4377,15 +4378,15 @@ func (c *ClickPipeResource) syncClickPipeState(ctx context.Context, state *model
}

if len(mapping.ExcludedColumns) > 0 {
excludedColsList := make([]attr.Value, len(mapping.ExcludedColumns))
excludedCols := make([]attr.Value, len(mapping.ExcludedColumns))
for j, col := range mapping.ExcludedColumns {
excludedColsList[j] = types.StringValue(col)
excludedCols[j] = types.StringValue(col)
}
tableMappingModel.ExcludedColumns, _ = types.ListValue(types.StringType, excludedColsList)
tableMappingModel.ExcludedColumns, _ = types.SetValue(types.StringType, excludedCols)
} else if hasStateMapping && !stateMapping.ExcludedColumns.IsNull() {
tableMappingModel.ExcludedColumns, _ = types.ListValue(types.StringType, []attr.Value{})
tableMappingModel.ExcludedColumns, _ = types.SetValue(types.StringType, []attr.Value{})
} else {
tableMappingModel.ExcludedColumns = types.ListNull(types.StringType)
tableMappingModel.ExcludedColumns = types.SetNull(types.StringType)
}

if mapping.UseCustomSortingKey != nil {
Expand Down Expand Up @@ -4713,13 +4714,16 @@ func (c *ClickPipeResource) syncClickPipeState(ctx context.Context, state *model
}

if len(mapping.ExcludedColumns) > 0 {
excludedColsList := make([]attr.Value, len(mapping.ExcludedColumns))
excludedCols := make([]attr.Value, len(mapping.ExcludedColumns))
for j, col := range mapping.ExcludedColumns {
excludedColsList[j] = types.StringValue(col)
excludedCols[j] = types.StringValue(col)
}
tableMappingModel.ExcludedColumns, _ = types.ListValue(types.StringType, excludedColsList)
tableMappingModel.ExcludedColumns, _ = types.SetValue(types.StringType, excludedCols)
} else if stateMapping != nil && !stateMapping.ExcludedColumns.IsNull() {
// Preserve empty set from state (vs null) to avoid plan diff
tableMappingModel.ExcludedColumns, _ = types.SetValue(types.StringType, []attr.Value{})
} else {
tableMappingModel.ExcludedColumns = types.ListNull(types.StringType)
tableMappingModel.ExcludedColumns = types.SetNull(types.StringType)
}
Comment thread
tpanetti marked this conversation as resolved.

if mapping.UseCustomSortingKey != nil {
Expand Down
Loading
Loading