Skip to content

Commit 5e308f3

Browse files
vdesabouvdesabou
committed
👾⭐ Add fully managed InfluxDB 3 Sink Connector example #7370 
1 parent 37f8673 commit 5e308f3

File tree

7 files changed

+223
-1
lines changed

7 files changed

+223
-1
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -136,7 +136,7 @@ jobs:
136136
"🚀 connect/connect-jdbc-sap-hana-sink connect/connect-jdbc-sap-hana-source connect/connect-sap-hana-sink other/kafka-connect-jsonata",
137137

138138
# requiring ngrok
139-
"🚀2️⃣ ccloud/fm-influxdb2-sink ccloud/fm-influxdb2-source ccloud/fm-jdbc-oracle19-source ccloud/fm-jdbc-oracle19-sink ccloud/fm-cdc-oracle19-source ccloud/fm-cdc-xstream-oracle19-source ccloud/fm-rabbitmq-source ccloud/fm-zendesk-source ccloud/fm-splunk-sink ccloud/fm-rabbitmq-sink ccloud/fm-jdbc-mysql-sink ccloud/fm-sftp-source ccloud/fm-http-sink ccloud/fm-http-v2-sink ccloud/fm-http-source ccloud/fm-http-v2-source",
139+
"🚀2️⃣ ccloud/fm-influxdb2-sink ccloud/fm-influxdb3-sink ccloud/fm-influxdb2-source ccloud/fm-jdbc-oracle19-source ccloud/fm-jdbc-oracle19-sink ccloud/fm-cdc-oracle19-source ccloud/fm-cdc-xstream-oracle19-source ccloud/fm-rabbitmq-source ccloud/fm-zendesk-source ccloud/fm-splunk-sink ccloud/fm-rabbitmq-sink ccloud/fm-jdbc-mysql-sink ccloud/fm-sftp-source ccloud/fm-http-sink ccloud/fm-http-v2-sink ccloud/fm-http-source ccloud/fm-http-v2-source",
140140
"🚀3️⃣ ccloud/fm-solace-sink ccloud/fm-opensearch-sink ccloud/fm-sftp-sink ccloud/fm-redis-sink ccloud/fm-redis-kafka-sink ccloud/fm-redis-kafka-source ccloud/fm-active-mq-source ccloud/fm-clickhouse-sink ccloud/fm-debezium-mariadb-source ccloud/fm-mongodb-source ccloud/fm-mongodb-sink"
141141
]
142142
steps:

‎ccloud/fm-influxdb3-sink/README.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Fully Managed InfluxDB 3 Sink connector
2+
3+
4+
5+
## Objective
6+
7+
Quickly test [InfluxDB 3 Sink](https://docs.confluent.io/cloud/current/connectors/cc-influx-db3-sink.html) connector.
8+
9+
## Exposing docker container over internet
10+
11+
**🚨WARNING🚨** It is considered a security risk to run this example on your personal machine since you'll be exposing a TCP port over internet using [Ngrok](https://ngrok.com). It is strongly encouraged to run it on a AWS EC2 instance where you'll use [Confluent Static Egress IP Addresses](https://docs.confluent.io/cloud/current/networking/static-egress-ip-addresses.html#use-static-egress-ip-addresses-with-ccloud) (only available for public endpoints on AWS) to allow traffic from your Confluent Cloud cluster to your EC2 instance using EC2 Security Group.
12+
13+
Example in order to set EC2 Security Group with Confluent Static Egress IP Addresses and port 1414:
14+
15+
```bash
16+
group=$(aws ec2 describe-instances --instance-id <$ec2-instance-id> --output=json | jq '.Reservations[] | .Instances[] | {SecurityGroups: .SecurityGroups}' | jq -r '.SecurityGroups[] | .GroupName')
17+
aws ec2 authorize-security-group-ingress --group-name $group --protocol tcp --port 1414 --cidr 13.36.88.88/32
18+
aws ec2 authorize-security-group-ingress --group-name $group --protocol tcp --port 1414 --cidr 13.36.88.89/32
19+
etc...
20+
```
21+
22+
An [Ngrok](https://ngrok.com) auth token is necessary in order to expose the Docker Container port to internet, so that fully managed connector can reach it.
23+
24+
You can sign up at https://dashboard.ngrok.com/signup
25+
If you have already signed up, make sure your auth token is setup by exporting environment variable `NGROK_AUTH_TOKEN`
26+
27+
Your auth token is available on your dashboard: https://dashboard.ngrok.com/get-started/your-authtoken
28+
29+
Ngrok web interface available at http://localhost:4551
30+
31+
## Prerequisites
32+
33+
See [here](https://kafka-docker-playground.io/#/how-to-use?id=%f0%9f%8c%a4%ef%b8%8f-confluent-cloud-examples)
34+
35+
36+
37+
## How to run
38+
39+
Simply run:
40+
41+
```
42+
$ just use <playground run>
43+
```

‎ccloud/fm-influxdb3-sink/docker-compose.yml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
services:
3+
# https://docs.influxdata.com/influxdb3/core/get-started/setup/#object-store-examples
4+
influxdb3:
5+
image: influxdb:3-core
6+
hostname: influxdb3
7+
container_name: influxdb3
8+
ports:
9+
- "8181:8181"
10+
command:
11+
- influxdb3
12+
- serve
13+
- --node-id=node0
14+
- --object-store=file
15+
- --data-dir=/var/lib/influxdb3/data
16+
- --plugin-dir=/var/lib/influxdb3/plugins
17+
18+
# https://ngrok.com/docs/using-ngrok-with/docker/
19+
ngrok:
20+
image: ngrok/ngrok:latest
21+
hostname: ngrok
22+
container_name: ngrok
23+
ports:
24+
- 4040:4040
25+
restart: unless-stopped
26+
links:
27+
- influxdb3
28+
command:
29+
- "start"
30+
- "--all"
31+
- "--log=stdout"
32+
- "--config"
33+
- "/etc/ngrok.yml"
34+
volumes:
35+
- ../../ccloud/fm-influxdb3-sink/ngrok.yml:/etc/ngrok.yml
36+
environment:
37+
NGROK_AUTHTOKEN: $NGROK_AUTH_TOKEN

‎ccloud/fm-influxdb3-sink/fully-managed-influxdb3-sink.sh‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
#!/bin/bash
2+
set -e
3+
4+
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
5+
source ${DIR}/../../scripts/utils.sh
6+
7+
NGROK_AUTH_TOKEN=${NGROK_AUTH_TOKEN:-$1}
8+
9+
display_ngrok_warning
10+
11+
bootstrap_ccloud_environment
12+
13+
set +e
14+
playground topic delete --topic influx_measures
15+
sleep 3
16+
playground topic create --topic influx_measures --nb-partitions 1
17+
set -e
18+
19+
docker compose build
20+
docker compose down -v --remove-orphans
21+
docker compose up -d --quiet-pull
22+
23+
sleep 5
24+
25+
log "Waiting for ngrok to start"
26+
while true
27+
do
28+
container_id=$(docker ps -q -f name=ngrok)
29+
if [ -n "$container_id" ]
30+
then
31+
status=$(docker inspect --format '{{.State.Status}}' $container_id)
32+
if [ "$status" = "running" ]
33+
then
34+
log "Getting ngrok hostname and port"
35+
NGROK_URL=$(curl --silent http://127.0.0.1:4040/api/tunnels | jq -r '.tunnels[0].public_url')
36+
NGROK_HOSTNAME=$(echo $NGROK_URL | cut -d "/" -f3 | cut -d ":" -f 1)
37+
NGROK_PORT=$(echo $NGROK_URL | cut -d "/" -f3 | cut -d ":" -f 2)
38+
39+
if ! [[ $NGROK_PORT =~ ^[0-9]+$ ]]
40+
then
41+
log "NGROK_PORT is not a valid number, keep retrying..."
42+
continue
43+
else
44+
break
45+
fi
46+
fi
47+
fi
48+
log "Waiting for container ngrok to start..."
49+
sleep 5
50+
done
51+
52+
log "Sending messages to topic influx_measures"
53+
playground topic produce -t influx_measures --nb-messages 1 --forced-value '{"measurement": "influx_measures", "id": 999, "product": "foo", "quantity": 100, "price": 50}' << 'EOF'
54+
{
55+
"fields": [
56+
{
57+
"name": "measurement",
58+
"type": "string"
59+
},
60+
{
61+
"name": "id",
62+
"type": "int"
63+
},
64+
{
65+
"name": "product",
66+
"type": "string"
67+
},
68+
{
69+
"name": "quantity",
70+
"type": "int"
71+
},
72+
{
73+
"name": "price",
74+
"type": "float"
75+
}
76+
],
77+
"name": "myrecord",
78+
"type": "record"
79+
}
80+
EOF
81+
82+
connector_name="InfluxDB3Sink_$USER"
83+
set +e
84+
playground connector delete --connector $connector_name > /dev/null 2>&1
85+
set -e
86+
87+
log "Creating InfluxDB3 admin token"
88+
INFLUXDB_TOKEN=$(docker exec influxdb3 influxdb3 create token --admin | awk '/Token:/{print $NF}' | tr -d '\r\n')
89+
90+
log "Creating InfluxDB3 database test-db"
91+
curl -X POST "http://localhost:8181/api/v3/configure/database" \
92+
--header "Authorization: Bearer $INFLUXDB_TOKEN" \
93+
--header "Content-Type: application/json" \
94+
--data '{"db": "test-db"}'
95+
96+
log "Creating fully managed connector"
97+
playground connector create-or-update --connector $connector_name << EOF
98+
{
99+
"connector.class": "InfluxDB3Sink",
100+
"name": "$connector_name",
101+
"kafka.auth.mode": "KAFKA_API_KEY",
102+
"kafka.api.key": "$CLOUD_KEY",
103+
"kafka.api.secret": "$CLOUD_SECRET",
104+
"input.data.format": "AVRO",
105+
"influxdb.url": "http://$NGROK_HOSTNAME:$NGROK_PORT",
106+
"influxdb.token": "$INFLUXDB_TOKEN",
107+
"influxdb.database": "test-db",
108+
"topics": "influx_measures",
109+
"tasks.max" : "1"
110+
}
111+
EOF
112+
wait_for_ccloud_connector_up $connector_name 180
113+
114+
sleep 10
115+
116+
log "Verify that order is in influxdb3"
117+
docker exec influxdb3 influxdb3 query "SELECT * FROM influx_measures" --database test-db --token $INFLUXDB_TOKEN > output.txt
118+
cat output.txt
119+
grep "foo" output.txt
120+
rm output.txt
121+
122+
log "Do you want to delete the fully managed connector $connector_name ?"
123+
check_if_continue
124+
125+
playground connector delete --connector $connector_name

‎ccloud/fm-influxdb3-sink/ngrok.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
version: 2
2+
tunnels:
3+
influxdb3:
4+
addr: influxdb3:8181
5+
proto: tcp

‎ccloud/fm-influxdb3-sink/stop.sh‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/bin/bash
2+
3+
4+
5+
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
6+
source ${DIR}/../../scripts/utils.sh
7+
8+
docker compose down -v --remove-orphans
9+
10+
11+
maybe_delete_ccloud_environment

‎docs/content-template.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -237,6 +237,7 @@
237237
- <img src="https://github.com/vdesabou/kafka-docker-playground/raw/master/images/icons/ibm_mq.png" width="15"> [IBM MQ Source](https://github.com/vdesabou/kafka-docker-playground/tree/master/ccloud/fm-ibm-mq-source) :ccloud/fm-ibm-mq-source:
238238
- <img src="https://github.com/vdesabou/kafka-docker-playground/raw/master/images/icons/influxdb.svg" width="15"> [InfluxDB 2 Source](https://github.com/vdesabou/kafka-docker-playground/tree/master/ccloud/fm-influxdb2-source) :ccloud/fm-influxdb2-source:
239239
- <img src="https://github.com/vdesabou/kafka-docker-playground/raw/master/images/icons/influxdb.svg" width="15"> [InfluxDB 2 Sink](https://github.com/vdesabou/kafka-docker-playground/tree/master/ccloud/fm-influxdb2-sink) :ccloud/fm-influxdb2-sink:
240+
- <img src="https://github.com/vdesabou/kafka-docker-playground/raw/master/images/icons/influxdb.svg" width="15"> [InfluxDB 3 Sink](https://github.com/vdesabou/kafka-docker-playground/tree/master/ccloud/fm-influxdb3-sink) :ccloud/fm-influxdb3-sink:
240241
- <img src="https://github.com/vdesabou/kafka-docker-playground/raw/master/images/icons/mysql.png" width="15"> [JDBC MySQL Source](https://github.com/vdesabou/kafka-docker-playground/tree/master/ccloud/fm-jdbc-mysql-source) (also with 🔑 SSL) :ccloud/fm-jdbc-mysql-source:
241242
- <img src="https://github.com/vdesabou/kafka-docker-playground/raw/master/images/icons/mysql.png" width="15"> [JDBC MySQL Sink](https://github.com/vdesabou/kafka-docker-playground/tree/master/ccloud/fm-jdbc-mysql-sink) :ccloud/fm-jdbc-mysql-sink:
242243
- <img src="https://github.com/vdesabou/kafka-docker-playground/raw/master/images/icons/postgresql.png" width="15"> [JDBC PostGreSQL Sink](https://github.com/vdesabou/kafka-docker-playground/tree/master/ccloud/fm-jdbc-postgresql-sink) :ccloud/fm-jdbc-postgresql-sink:

0 commit comments

Comments
 (0)