Skip to content

Commit 3c49c88

Browse files
authored
Merge pull request #6166 from ClickHouse/sak/fix/kafka-connect-sink-consumer-prefix-docs
[kafka-connect] standardize `consumer.override.` prefix in config examples
2 parents feb34e2 + 09c642f commit 3c49c88

1 file changed

Lines changed: 20 additions & 8 deletions

File tree

docs/integrations/data-ingestion/kafka/kafka-clickhouse-connect-sink.md

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,10 @@ The most basic configuration to get you started - it assumes you're running Kafk
201201
}
202202
```
203203

204+
:::note
205+
The above connector config requires that you enable client overrides in your worker configuration via `connector.client.config.override.policy=All`. See the [Kafka Connect documentation](https://docs.confluent.io/platform/current/connect/references/allconfigs.html#override-the-worker-configuration) for more information.
206+
:::
207+
204208
#### Basic configuration with multiple topics {#basic-configuration-with-multiple-topics}
205209

206210
The connector can consume data from multiple topics
@@ -571,18 +575,22 @@ For optimal performance with ClickHouse, aim for larger batches:
571575

572576
```properties
573577
# Increase the number of records per poll
574-
consumer.max.poll.records=5000
578+
consumer.override.max.poll.records=5000
575579

576580
# Increase the partition fetch size (5 MB)
577-
consumer.max.partition.fetch.bytes=5242880
581+
consumer.override.max.partition.fetch.bytes=5242880
578582

579583
# Optional: Increase minimum fetch size to wait for more data (1 MB)
580-
consumer.fetch.min.bytes=1048576
584+
consumer.override.fetch.min.bytes=1048576
581585

582586
# Optional: Reduce wait time if latency is critical
583-
consumer.fetch.max.wait.ms=300
587+
consumer.override.fetch.max.wait.ms=300
584588
```
585589

590+
:::note
591+
The above properties require that you enable client overrides in your worker configuration via `connector.client.config.override.policy=All`. See the [Kafka Connect documentation](https://docs.confluent.io/platform/current/connect/references/allconfigs.html#override-the-worker-configuration) for more information.
592+
:::
593+
586594
**Important**: Kafka Connect fetch settings represent compressed data, while ClickHouse receives uncompressed data. Balance these settings based on your compression ratio.
587595

588596
**Trade-offs**:
@@ -821,16 +829,20 @@ Here's a complete example optimized for high throughput:
821829
"exactlyOnce": "false",
822830
"ignorePartitionsWhenBatching": "true",
823831

824-
"consumer.max.poll.records": "10000",
825-
"consumer.max.partition.fetch.bytes": "5242880",
826-
"consumer.fetch.min.bytes": "1048576",
827-
"consumer.fetch.max.wait.ms": "500",
832+
"consumer.override.max.poll.records": "10000",
833+
"consumer.override.max.partition.fetch.bytes": "5242880",
834+
"consumer.override.fetch.min.bytes": "1048576",
835+
"consumer.override.fetch.max.wait.ms": "500",
828836

829837
"clickhouseSettings": "async_insert=1,wait_for_async_insert=1,async_insert_max_data_size=16777216,async_insert_busy_timeout_ms=1000,socket_timeout=300000"
830838
}
831839
}
832840
```
833841

842+
:::note
843+
The above connector config requires that you enable client overrides in your worker configuration via `connector.client.config.override.policy=All`. See the [Kafka Connect documentation](https://docs.confluent.io/platform/current/connect/references/allconfigs.html#override-the-worker-configuration) for more information.
844+
:::
845+
834846
**This configuration**:
835847
- Processes up to 10,000 records per poll
836848
- Batches across partitions for larger inserts

0 commit comments

Comments
 (0)