Skip to content

Commit 752e1ab

Browse files
perhammanton.voskresensky
perhamm
and
anton.voskresensky
authored
feat: Add Kafka SASL/SSL authentication support with ClickHouse integration (#45)
* add kafka auth * fix README * fix clickhouse -> kafka auth --------- Co-authored-by: anton.voskresensky <anton.voskresensky@flant.com>
1 parent 61f60a3 commit 752e1ab

9 files changed

Lines changed: 401 additions & 2 deletions

File tree

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,3 +119,5 @@ ios/Pods
119119
!.yarn/plugins
120120

121121
.turbo
122+
123+
certs

‎README.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,28 @@ If you have questions or need assistance, you can join our [Slack group](https:/
171171
}
172172
```
173173

174+
### Kafka authentication
175+
176+
Trench supports connecting to Kafka clusters that require SASL and/or SSL. Configure via the following environment variables (all optional):
177+
178+
- `KAFKA_SSL_ENABLED`: Enable SSL/TLS when connecting to brokers. Values: `true`/`false` (default: `false`).
179+
- `KAFKA_SSL_REJECT_UNAUTHORIZED`: Whether to verify broker certificates. Values: `true`/`false` (default: `true`). Set to `false` when using self-signed certs in development.
180+
- `KAFKA_SSL_CA`: CA certificate contents (PEM). Use when brokers use a custom CA.
181+
- `KAFKA_SSL_CERT`: Client certificate contents (PEM) if mutual TLS is required.
182+
- `KAFKA_SSL_KEY`: Client private key (PEM) if mutual TLS is required.
183+
- `KAFKA_SASL_MECHANISM`: One of `plain`, `scram-sha-256`, or `scram-sha-512`.
184+
- `KAFKA_SASL_USERNAME`: SASL username (required when `KAFKA_SASL_MECHANISM` is set).
185+
- `KAFKA_SASL_PASSWORD`: SASL password (required when `KAFKA_SASL_MECHANISM` is set).
186+
187+
Notes:
188+
- For SSL cert variables, provide the PEM content directly (including header/footer) or mount files and load into env before starting.
189+
- When using Bitnami Kafka images locally, the default `docker-compose.yml` uses PLAINTEXT; set the appropriate broker listeners and advertise SSL/SASL endpoints in your Kafka deployment if required.
190+
- Kafka authentication for ClickHouse should be configured in ClickHouse server configuration files, not in SQL migrations.
191+
192+
### ClickHouse Kafka Authentication
193+
194+
For ClickHouse to connect to authenticated Kafka clusters, you need to configure authentication in ClickHouse server configuration files.
195+
174196
### 2. Trench Cloud ☁️
175197

176198
If you don't want to selfhost, you can get started with Trench in a few minutes via:
@@ -188,6 +210,44 @@ If you don't want to selfhost, you can get started with Trench in a few minutes
188210
- [Documentation](https://docs.trench.dev/)
189211
- [Slack community](https://join.slack.com/t/trench-community/shared_invite/zt-2sjet5kh2-v31As3yC_zRIadk_AGn~3A)
190212

213+
### Kafka Authentication Examples
214+
215+
Trench supports Kafka authentication with SASL and SSL. Here are examples of how to configure both Node.js KafkaJS client and ClickHouse for authenticated Kafka connections.
216+
217+
**Quick Start:**
218+
- **SASL-only**: `docker compose -f docker-compose.sasl.yml up -d --build`
219+
- **SSL+SASL**: Generate certs first with `./scripts/generate-kafka-certs.sh`, then run the compose file
220+
221+
#### SASL-Only Authentication
222+
```bash
223+
# Run with SASL authentication (no SSL)
224+
docker compose -f docker-compose.sasl.yml up -d --build
225+
```
226+
227+
This setup uses:
228+
- **Node.js KafkaJS**: `KAFKA_SASL_MECHANISM=PLAIN`, `KAFKA_SASL_USERNAME=kafka_user`, `KAFKA_SASL_PASSWORD=kafka_password`
229+
- **ClickHouse**: Pre-configured with `clickhouse-kafka-auth-config-example/clickhouse-sasl.xml` for Kafka authentication
230+
- **Kafka**: SASL_PLAINTEXT listener on port 9095
231+
232+
#### SSL+SASL Authentication
233+
```bash
234+
# Step 1: Generate SSL certificates
235+
./scripts/generate-kafka-certs.sh
236+
237+
# Step 2: Set environment variables for SSL certificates
238+
export KAFKA_SSL_CA=$(cat ./certs/ca.pem)
239+
export KAFKA_SSL_CERT=$(cat ./certs/client.pem)
240+
export KAFKA_SSL_KEY=$(cat ./certs/client-key.pem)
241+
242+
# Step 3: Run with both SSL and SASL authentication
243+
docker compose -f docker-compose.ssl-sasl.yml up -d --build
244+
```
245+
246+
This setup uses:
247+
- **Node.js KafkaJS**: SSL+SASL authentication via environment variables
248+
- **ClickHouse**: Pre-configured with `clickhouse-kafka-auth-config-example/clickhouse-ssl-sasl.xml` for Kafka authentication
249+
- **Kafka**: SASL_SSL listener on port 9095 with SSL certificates
250+
191251
## 📚 Authors
192252

193253
Trench is a project built by [Frigade](https://frigade.com).

‎apps/trench/clickhouse-kafka-auth-config-example/clickhouse-sasl.xml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
<?xml version="1.0"?>
2+
<clickhouse>
3+
<kafka>
4+
<security_protocol>sasl_plaintext</security_protocol>
5+
<sasl_mechanism>PLAIN</sasl_mechanism>
6+
<sasl_username>kafka_user</sasl_username>
7+
<sasl_password>kafka_password</sasl_password>
8+
</kafka>
9+
</clickhouse>

‎apps/trench/clickhouse-kafka-auth-config-example/clickhouse-ssl-sasl.xml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
<?xml version="1.0"?>
2+
<clickhouse>
3+
<kafka>
4+
<security_protocol>sasl_ssl</security_protocol>
5+
<sasl_mechanism>PLAIN</sasl_mechanism>
6+
<sasl_username>kafka_user</sasl_username>
7+
<sasl_password>kafka_password</sasl_password>
8+
<enable_ssl_certificate_verification>false</enable_ssl_certificate_verification>
9+
</kafka>
10+
</clickhouse>

‎apps/trench/docker-compose.sasl.yml‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
services:
2+
api:
3+
build:
4+
dockerfile: Dockerfile
5+
context: .
6+
target: production-build
7+
args:
8+
API_DOMAIN: ${API_DOMAIN}
9+
env_file:
10+
- .env
11+
volumes:
12+
- ./:/app
13+
- /app/node_modules
14+
- /app/certs
15+
- /app/schemas
16+
- /app/dist
17+
command: node /app/dist/main.js -i -1
18+
ports:
19+
- '${API_PORT}:${API_PORT}'
20+
depends_on:
21+
kafka:
22+
condition: service_healthy
23+
clickhouse:
24+
condition: service_healthy
25+
networks:
26+
- app-network
27+
restart: unless-stopped
28+
environment:
29+
- KAFKA_BROKERS=kafka:9095
30+
- KAFKA_SSL_ENABLED=false
31+
- KAFKA_SASL_MECHANISM=PLAIN
32+
- KAFKA_SASL_USERNAME=kafka_user
33+
- KAFKA_SASL_PASSWORD=kafka_password
34+
clickhouse:
35+
image: clickhouse/clickhouse-server:24.8
36+
restart: always
37+
healthcheck:
38+
test: ["CMD", "bash", "-lc", "wget -qO- http://localhost:8123/ping | grep -q 'Ok'"]
39+
interval: 5s
40+
timeout: 3s
41+
retries: 30
42+
ports:
43+
- '8123:8123'
44+
- '9000:9000'
45+
volumes:
46+
- clickhouse-data:/var/lib/clickhouse
47+
- ./clickhouse-kafka-auth-config-example/clickhouse-sasl.xml:/etc/clickhouse-server/config.d/kafka.xml:ro
48+
environment:
49+
- CLICKHOUSE_USER=${CLICKHOUSE_USER}
50+
- CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD}
51+
ulimits:
52+
nofile:
53+
soft: 262144
54+
hard: 262144
55+
networks:
56+
- app-network
57+
kafka:
58+
image: bitnamilegacy/kafka:3.7.0
59+
ports:
60+
- 9092:9092
61+
- 9093:9093
62+
- 9094:9094
63+
- 9095:9095
64+
deploy:
65+
resources:
66+
limits:
67+
cpus: '4'
68+
memory: 4096M
69+
environment:
70+
- KAFKA_CFG_NODE_ID=0
71+
- KAFKA_CFG_PROCESS_ROLES=controller,broker
72+
- KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT
73+
- KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
74+
- KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR=1
75+
- KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
76+
- KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR=1
77+
- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,SASL_PLAINTEXT://:9095,EXTERNAL://:9094
78+
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,SASL_PLAINTEXT://kafka:9095,EXTERNAL://localhost:9094
79+
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,EXTERNAL:PLAINTEXT
80+
- KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
81+
- KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
82+
- KAFKA_CFG_SASL_ENABLED_MECHANISMS=PLAIN
83+
- KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=PLAIN
84+
- KAFKA_CFG_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka_user" password="kafka_password" user_kafka_user="kafka_password" user_admin="admin_password";
85+
- KAFKA_CFG_LISTENER_NAME_SASL_PLAINTEXT_SASL_ENABLED_MECHANISMS=PLAIN
86+
- KAFKA_CFG_LISTENER_NAME_SASL_PLAINTEXT_PLAIN_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka_user" password="kafka_password" user_kafka_user="kafka_password" user_admin="admin_password";
87+
volumes:
88+
- kafka-data:/bitnami/kafka
89+
healthcheck:
90+
test: ["CMD", "bash", "-lc", "/opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list || exit 1"]
91+
interval: 5s
92+
timeout: 3s
93+
retries: 60
94+
networks:
95+
- app-network
96+
volumes:
97+
clickhouse-data:
98+
kafka-data:
99+
networks:
100+
app-network:

‎apps/trench/docker-compose.ssl-sasl.yml‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
services:
2+
api:
3+
build:
4+
dockerfile: Dockerfile
5+
context: .
6+
target: production-build
7+
args:
8+
API_DOMAIN: ${API_DOMAIN}
9+
env_file:
10+
- .env
11+
volumes:
12+
- ./:/app
13+
- /app/node_modules
14+
- /app/certs
15+
- /app/schemas
16+
- /app/dist
17+
command: node /app/dist/main.js -i -1
18+
ports:
19+
- '${API_PORT}:${API_PORT}'
20+
depends_on:
21+
kafka:
22+
condition: service_healthy
23+
clickhouse:
24+
condition: service_healthy
25+
networks:
26+
- app-network
27+
restart: unless-stopped
28+
environment:
29+
- KAFKA_BROKERS=kafka:9095
30+
- KAFKA_SSL_ENABLED=true
31+
- KAFKA_SSL_REJECT_UNAUTHORIZED=false
32+
- KAFKA_SSL_CA=${KAFKA_SSL_CA}
33+
- KAFKA_SSL_CERT=${KAFKA_SSL_CERT}
34+
- KAFKA_SSL_KEY=${KAFKA_SSL_KEY}
35+
- KAFKA_SASL_MECHANISM=PLAIN
36+
- KAFKA_SASL_USERNAME=kafka_user
37+
- KAFKA_SASL_PASSWORD=kafka_password
38+
clickhouse:
39+
image: clickhouse/clickhouse-server:24.8
40+
restart: always
41+
healthcheck:
42+
test: ["CMD", "bash", "-lc", "clickhouse-client --query 'SELECT 1'"]
43+
interval: 5s
44+
timeout: 3s
45+
retries: 30
46+
ports:
47+
- '8123:8123'
48+
- '9000:9000'
49+
volumes:
50+
- clickhouse-data:/var/lib/clickhouse
51+
- ./clickhouse-kafka-auth-config-example/clickhouse-ssl-sasl.xml:/etc/clickhouse-server/config.d/kafka.xml:ro
52+
- ./certs:/opt/bitnami/kafka/config/certs:ro
53+
environment:
54+
- CLICKHOUSE_USER=${CLICKHOUSE_USER}
55+
- CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD}
56+
ulimits:
57+
nofile:
58+
soft: 262144
59+
hard: 262144
60+
networks:
61+
- app-network
62+
kafka:
63+
image: bitnamilegacy/kafka:3.7.0
64+
ports:
65+
- 9092:9092
66+
- 9093:9093
67+
- 9094:9094
68+
- 9095:9095
69+
deploy:
70+
resources:
71+
limits:
72+
cpus: '4'
73+
memory: 4096M
74+
environment:
75+
- KAFKA_CFG_NODE_ID=0
76+
- KAFKA_CFG_PROCESS_ROLES=controller,broker
77+
- KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT
78+
- KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
79+
- KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR=1
80+
- KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
81+
- KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR=1
82+
- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,SASL_SSL://:9095,EXTERNAL://:9094
83+
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,SASL_SSL://kafka:9095,EXTERNAL://localhost:9094
84+
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SASL_SSL:SASL_SSL,EXTERNAL:PLAINTEXT
85+
- KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
86+
- KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
87+
- KAFKA_CFG_SSL_KEYSTORE_LOCATION=/opt/bitnami/kafka/config/certs/kafka.keystore.jks
88+
- KAFKA_CFG_SSL_KEYSTORE_PASSWORD=server_keystore_password
89+
- KAFKA_CFG_SSL_KEY_PASSWORD=server_keystore_password
90+
- KAFKA_CFG_SSL_TRUSTSTORE_LOCATION=/opt/bitnami/kafka/config/certs/kafka.truststore.jks
91+
- KAFKA_CFG_SSL_TRUSTSTORE_PASSWORD=server_truststore_password
92+
- KAFKA_CFG_SSL_CLIENT_AUTH=required
93+
- KAFKA_CFG_SASL_ENABLED_MECHANISMS=PLAIN
94+
- KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=PLAIN
95+
- KAFKA_CFG_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka_user" password="kafka_password" user_kafka_user="kafka_password" user_admin="admin_password";
96+
- KAFKA_CFG_LISTENER_NAME_SASL_SSL_SASL_ENABLED_MECHANISMS=PLAIN
97+
- KAFKA_CFG_LISTENER_NAME_SASL_SSL_PLAIN_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka_user" password="kafka_password" user_kafka_user="kafka_password" user_admin="admin_password";
98+
volumes:
99+
- kafka-data:/bitnami/kafka
100+
- ./certs:/opt/bitnami/kafka/config/certs:ro
101+
healthcheck:
102+
test: ["CMD", "bash", "-lc", "/opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list || exit 1"]
103+
interval: 5s
104+
timeout: 3s
105+
retries: 60
106+
networks:
107+
- app-network
108+
volumes:
109+
clickhouse-data:
110+
kafka-data:
111+
networks:
112+
app-network:

‎apps/trench/docker-compose.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,10 @@ services:
2323
networks:
2424
- app-network
2525
restart: unless-stopped
26+
environment:
27+
- KAFKA_BROKERS=kafka:9092
2628
clickhouse:
27-
image: clickhouse/clickhouse-server
29+
image: clickhouse/clickhouse-server:24.8
2830
restart: always
2931
ports:
3032
- '8123:8123'
@@ -42,7 +44,7 @@ services:
4244
networks:
4345
- app-network
4446
kafka:
45-
image: bitnami/kafka:latest
47+
image: bitnamilegacy/kafka:3.7.0
4648
ports:
4749
- 9092:9092
4850
- 9093:9093

0 commit comments

Comments
 (0)