You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Estuary also provides a [Dekaf-based integration](./Dekaf/clickhouse.md) for users who prefer to ingest via ClickPipes.
11
13
@@ -14,7 +16,7 @@ Estuary also provides a [Dekaf-based integration](./Dekaf/clickhouse.md) for use
14
16
To use this connector, you'll need:
15
17
16
18
* A ClickHouse database (self-hosted or ClickHouse Cloud) with a user that has permissions to create tables and write data.
17
-
* The connector uses the ClickHouse native protocol (port 9000 by default, not the HTTP interface on port 8123).
19
+
* The connector uses the ClickHouse native protocol. The default port is **9440** (TLS enabled, the default) or **9000** (TLS disabled). It does not use the HTTP interface on port 8123.
18
20
* At least one Estuary collection.
19
21
20
22
:::tip
@@ -32,19 +34,22 @@ Use the below properties to configure a ClickHouse materialization, which will d
32
34
33
35
| Property | Title | Description | Type | Required/Default |
34
36
|---|---|---|---|---|
35
-
|**`/address`**| Address | Host and port of the database, in the form of `host[:port]`. Port 9000 is used as the default if no specific port is provided. | string | Required |
37
+
|**`/address`**| Address | Host and port of the database, in the form of `host[:port]`. Port 9440 is used as the default when SSL is enabled (the default), or 9000 when SSL is disabled. | string | Required |
|**`/database`**| Database | Name of the ClickHouse database to materialize to. | string | Required |
41
-
|`/hardDelete`| Hard Delete | If enabled, the connector inserts tombstone rows with `_is_deleted = 1` when source documents are deleted, causing them to be excluded from `FINAL` queries. By default, source deletions are ignored at the destination. | boolean |`false`|
43
+
|`/hardDelete`| Hard Delete | If enabled, items deleted in the source will also be deleted from the destination. By default, deletions are tracked via `_meta/op` (soft-delete). | boolean |`false`|
|`/advanced/no_flow_document`| Exclude Flow Document | When enabled, the root document column will not be required for standard updates. | boolean |`false`|
42
46
43
47
#### Bindings
44
48
45
49
| Property | Title | Description | Type | Required/Default |
46
50
|---|---|---|---|---|
47
51
|**`/table`**| Table | Name of the database table to materialize to. The connector will create the table if it doesn't already exist. | string | Required |
52
+
|`/delta_updates`| Delta Update | Should updates to this table be done via delta updates. | boolean |`false`|
48
53
49
54
### Sample
50
55
@@ -54,7 +59,7 @@ materializations:
54
59
endpoint:
55
60
connector:
56
61
config:
57
-
address: clickhouse.example.com:9000
62
+
address: clickhouse.example.com:9440
58
63
credentials:
59
64
auth_type: user_password
60
65
username: flow_user
@@ -69,24 +74,27 @@ materializations:
69
74
70
75
## ReplacingMergeTree and FINAL
71
76
72
-
The connector creates tables using the [ReplacingMergeTree engine](https://clickhouse.com/docs/engines/table-engines/mergetree-family/replacingmergetree). Updated records are actually inserted as duplicates; ClickHouse later deduplicates these as a background process.
77
+
In standard (non-delta) mode, the connector creates tables using the [ReplacingMergeTree engine](https://clickhouse.com/docs/engines/table-engines/mergetree-family/replacingmergetree) with `flow_published_at` as the version column.
78
+
Updated records are inserted as new rows; ClickHouse deduplicates them in a background process, keeping the row with the highest `flow_published_at` value for each key.
73
79
74
-
Your queries should use the `FINAL` directive to get deduplicated results, and include the predicate `_is_deleted = 0` to ignore deleted records.
80
+
The connector also configures automatic background cleanup merges so that superseded rows and tombstones are eventually removed from disk.
81
+
82
+
Your queries should use the `FINAL` directive to get results with duplicate and tombstone rows removed:
75
83
76
84
```sql
77
-
SELECT * FROM my_table FINAL WHERE _is_deleted = 0;
85
+
SELECT * FROM my_table FINAL;
78
86
```
79
87
80
88
## Hard deletes
81
89
82
-
All tables are created with `_version` (UInt64) and `_is_deleted` (UInt8) columns used internally by the `ReplacingMergeTree` engine.
83
-
84
-
If you set `hardDelete: true` in the endpoint configuration, the connector inserts a **tombstone row** when a source document is deleted. The tombstone has `_is_deleted = 1`, the same key columns as the original row, and zero values for all other columns. The `ReplacingMergeTree` engine then uses `_is_deleted` to hide these rows from `FINAL` queries, and eventually removes the tombstoned records from the table.
90
+
When `hardDelete: true` is set in the endpoint configuration, the connector adds an `_is_deleted` (UInt8) column to each table.
91
+
When a source document is deleted, the connector inserts a **tombstone row** with `_is_deleted = 1` and the same key columns as the original row.
92
+
The `ReplacingMergeTree` engine uses `_is_deleted` to exclude these rows from `FINAL` queries, and automatic cleanup merges eventually remove the tombstoned records from disk.
85
93
86
-
## Soft deletes not supported
94
+
## Soft deletes
87
95
88
-
Source deletions are effectively ignored at the destination.
96
+
By default (when `hardDelete` is not enabled), source deletions are tracked in the destination via the `_meta/op` column, which indicates whether a row was created, updated, or deleted. The row itself remains in the table.
89
97
90
-
## Delta updates not supported
98
+
## Delta updates
91
99
92
-
This connector does not support [delta updates](/concepts/materialization/#delta-updates). Only standard (merge) mode is supported.
100
+
This connector supports [delta updates](/concepts/materialization/#delta-updates) on a per-binding basis. When `delta_updates` is enabled for a binding, the table uses the `MergeTree` engine instead of `ReplacingMergeTree`. Every store operation is appended as-is with no deduplication — rows accumulate and are never removed.
0 commit comments