Skip to content

Commit 580bf01

Browse files
committed
update per changes to implementation
1 parent c985730 commit 580bf01

1 file changed

Lines changed: 23 additions & 15 deletions

File tree

  • site/docs/reference/Connectors/materialization-connectors

site/docs/reference/Connectors/materialization-connectors/ClickHouse.md

Lines changed: 23 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,9 @@
55
This connector materializes Estuary collections into tables in a ClickHouse database.
66

77
[ClickHouse](https://clickhouse.com/) is a column-oriented OLAP database designed for real-time analytics.
8-
This connector writes directly to ClickHouse using the native protocol.
8+
This connector writes batches directly to ClickHouse using the
9+
[Native protocol](https://clickhouse.com/docs/interfaces/tcp) and
10+
[Native format](https://clickhouse.com/docs/interfaces/formats/Native).
911

1012
Estuary also provides a [Dekaf-based integration](./Dekaf/clickhouse.md) for users who prefer to ingest via ClickPipes.
1113

@@ -14,7 +16,7 @@ Estuary also provides a [Dekaf-based integration](./Dekaf/clickhouse.md) for use
1416
To use this connector, you'll need:
1517

1618
* A ClickHouse database (self-hosted or ClickHouse Cloud) with a user that has permissions to create tables and write data.
17-
* The connector uses the ClickHouse native protocol (port 9000 by default, not the HTTP interface on port 8123).
19+
* The connector uses the ClickHouse native protocol. The default port is **9440** (TLS enabled, the default) or **9000** (TLS disabled). It does not use the HTTP interface on port 8123.
1820
* At least one Estuary collection.
1921

2022
:::tip
@@ -32,19 +34,22 @@ Use the below properties to configure a ClickHouse materialization, which will d
3234

3335
| Property | Title | Description | Type | Required/Default |
3436
|---|---|---|---|---|
35-
| **`/address`** | Address | Host and port of the database, in the form of `host[:port]`. Port 9000 is used as the default if no specific port is provided. | string | Required |
37+
| **`/address`** | Address | Host and port of the database, in the form of `host[:port]`. Port 9440 is used as the default when SSL is enabled (the default), or 9000 when SSL is disabled. | string | Required |
3638
| **`/credentials`** | Authentication | | object | Required |
3739
| **`/credentials/auth_type`** | Auth Type | Authentication type. Must be `user_password`. | string | Required |
3840
| **`/credentials/username`** | Username | Database username. | string | Required |
3941
| **`/credentials/password`** | Password | Database password. | string | Required |
4042
| **`/database`** | Database | Name of the ClickHouse database to materialize to. | string | Required |
41-
| `/hardDelete` | Hard Delete | If enabled, the connector inserts tombstone rows with `_is_deleted = 1` when source documents are deleted, causing them to be excluded from `FINAL` queries. By default, source deletions are ignored at the destination. | boolean | `false` |
43+
| `/hardDelete` | Hard Delete | If enabled, items deleted in the source will also be deleted from the destination. By default, deletions are tracked via `_meta/op` (soft-delete). | boolean | `false` |
44+
| `/advanced/sslmode` | SSL Mode | Controls the TLS connection behavior. Options: `disable`, `require`, `verify-full`. | string | `verify-full` |
45+
| `/advanced/no_flow_document` | Exclude Flow Document | When enabled, the root document column will not be required for standard updates. | boolean | `false` |
4246

4347
#### Bindings
4448

4549
| Property | Title | Description | Type | Required/Default |
4650
|---|---|---|---|---|
4751
| **`/table`** | Table | Name of the database table to materialize to. The connector will create the table if it doesn't already exist. | string | Required |
52+
| `/delta_updates` | Delta Update | Should updates to this table be done via delta updates. | boolean | `false` |
4853

4954
### Sample
5055

@@ -54,7 +59,7 @@ materializations:
5459
endpoint:
5560
connector:
5661
config:
57-
address: clickhouse.example.com:9000
62+
address: clickhouse.example.com:9440
5863
credentials:
5964
auth_type: user_password
6065
username: flow_user
@@ -69,24 +74,27 @@ materializations:
6974
7075
## ReplacingMergeTree and FINAL
7176
72-
The connector creates tables using the [ReplacingMergeTree engine](https://clickhouse.com/docs/engines/table-engines/mergetree-family/replacingmergetree). Updated records are actually inserted as duplicates; ClickHouse later deduplicates these as a background process.
77+
In standard (non-delta) mode, the connector creates tables using the [ReplacingMergeTree engine](https://clickhouse.com/docs/engines/table-engines/mergetree-family/replacingmergetree) with `flow_published_at` as the version column.
78+
Updated records are inserted as new rows; ClickHouse deduplicates them in a background process, keeping the row with the highest `flow_published_at` value for each key.
7379

74-
Your queries should use the `FINAL` directive to get deduplicated results, and include the predicate `_is_deleted = 0` to ignore deleted records.
80+
The connector also configures automatic background cleanup merges so that superseded rows and tombstones are eventually removed from disk.
81+
82+
Your queries should use the `FINAL` directive to get results with duplicate and tombstone rows removed:
7583

7684
```sql
77-
SELECT * FROM my_table FINAL WHERE _is_deleted = 0;
85+
SELECT * FROM my_table FINAL;
7886
```
7987

8088
## Hard deletes
8189

82-
All tables are created with `_version` (UInt64) and `_is_deleted` (UInt8) columns used internally by the `ReplacingMergeTree` engine.
83-
84-
If you set `hardDelete: true` in the endpoint configuration, the connector inserts a **tombstone row** when a source document is deleted. The tombstone has `_is_deleted = 1`, the same key columns as the original row, and zero values for all other columns. The `ReplacingMergeTree` engine then uses `_is_deleted` to hide these rows from `FINAL` queries, and eventually removes the tombstoned records from the table.
90+
When `hardDelete: true` is set in the endpoint configuration, the connector adds an `_is_deleted` (UInt8) column to each table.
91+
When a source document is deleted, the connector inserts a **tombstone row** with `_is_deleted = 1` and the same key columns as the original row.
92+
The `ReplacingMergeTree` engine uses `_is_deleted` to exclude these rows from `FINAL` queries, and automatic cleanup merges eventually remove the tombstoned records from disk.
8593

86-
## Soft deletes not supported
94+
## Soft deletes
8795

88-
Source deletions are effectively ignored at the destination.
96+
By default (when `hardDelete` is not enabled), source deletions are tracked in the destination via the `_meta/op` column, which indicates whether a row was created, updated, or deleted. The row itself remains in the table.
8997

90-
## Delta updates not supported
98+
## Delta updates
9199

92-
This connector does not support [delta updates](/concepts/materialization/#delta-updates). Only standard (merge) mode is supported.
100+
This connector supports [delta updates](/concepts/materialization/#delta-updates) on a per-binding basis. When `delta_updates` is enabled for a binding, the table uses the `MergeTree` engine instead of `ReplacingMergeTree`. Every store operation is appended as-is with no deduplication — rows accumulate and are never removed.

0 commit comments

Comments
 (0)