Skip to content

Commit 1eb2101

Browse files
pfcoperezCopilotClaudemasterashu
authored
(feat) Local dev env: Tilt orchestrated data store initialization and connection coordinates propagation (#4101)
## Description This PR uses Tilt+Docker Compose to start ancillary services required for local e2e testing. <img width="1833" height="1023" alt="image" src="https://github.com/user-attachments/assets/310bd5ad-fcc2-4eac-bbbc-0cfba60c4d80" /> The idea is simple: Make tests read their input parameters from the same source of truth that generated the dev environment. The local `.env` file contains the ports, users and other parameters used to fill in the docker compose template defining the services. e.g: ``` MONGO_HOST=localhost MONGO_PORT=27017 MONGO_ADMIN_USERNAME=admin MONGO_ADMIN_PASSWORD=admin MONGO_USERNAME=csuser MONGO_PASSWORD=cspass CLICKHOUSE_HOST=localhost CLICKHOUSE_HTTP_PORT=8123 CLICKHOUSE_NATIVE_PORT=9000 PG_HOST=localhost PG_PORT=5436 PG_USER=postgres PG_PASSWORD=postgres PG_DATABASE=postgres ``` ## Notes for reviewers - This PR version extracts versions from GH Workflow test flow. I think we can re-use this infrastructure definition in the flow itself. If we do that, we can just use a different way of defining the test matrix images that works both locally and in CI without the need of brittle extraction scripts. Alternatively we can just hard-code versions in the local env too. - The dev env adds an additional PSQL service to use as data source but the existing e2e tests use the catalog PSQL instance. We can disable the extra resource. - Some PG tests fail because the catalog has not been provisioned completely. (e.g: `env -f ../.env go test -v -run TestGenericCH_PG/Test_Simple_Flow ./e2e/`) I am following up with a PR in the tests to use the additional PSQL instance which is intended to be a data source instead of the catalog. ## ~WIP~ - [x] Update README.md with instructions. - [x] Add MySQL/MariaDB f6ef96d - [x] Make provisioning scripts idempotent: 2d8a2cc ## Related work Follows: [#3901](#3901) Replaces: [#4095](#4095) (same changes, moving source repository) --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: anthropic-code-agent[bot] <242468646+Claude@users.noreply.github.com> Co-authored-by: pfcoperez <273379+pfcoperez@users.noreply.github.com> Co-authored-by: masterashu <ashutosh.chauhan@clickhouse.com>
1 parent b959339 commit 1eb2101

17 files changed

Lines changed: 555 additions & 12 deletions

‎.env.example‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
CI_MONGO_HOST=host.docker.internal
2+
CI_MONGO_PORT=11017
3+
CI_MONGO_ADMIN_URI=mongodb://host.docker.internal:11017
4+
CI_MONGO_ADMIN_USERNAME=admin
5+
CI_MONGO_ADMIN_PASSWORD=admin
6+
CI_MONGO_URI=mongodb://host.docker.internal:11017
7+
CI_MONGO_USERNAME=csuser
8+
CI_MONGO_PASSWORD=cspass
9+
10+
CI_CLICKHOUSE_HOST=host.docker.internal
11+
CI_CLICKHOUSE_HTTP_PORT=11123
12+
CI_CLICKHOUSE_NATIVE_PORT=11000
13+
14+
CI_MYSQL_HOST=host.docker.internal
15+
CI_MYSQL_GTID_PORT=3306
16+
CI_MYSQL_POS_PORT=3307
17+
CI_MARIADB_PORT=3308
18+
CI_MYSQL_ROOT_PASSWORD=cipass
19+
CI_MYSQL_VERSION=mysql-gtid
20+
21+
CI_MYSQL_PORT=3306 # Default is gtid
22+
23+
PG_HOST=host.docker.internal
24+
PG_PORT=5436
25+
PG_USER=postgres
26+
PG_PASSWORD=postgres
27+
PG_DATABASE=postgres
28+
29+
PEERDB_CATALOG_HOST=host.docker.internal
30+
PEERDB_CATALOG_PORT=9901
31+
PEERDB_CATALOG_USER=postgres
32+
PEERDB_CATALOG_PASSWORD=postgres
33+
PEERDB_CATALOG_DATABASE=postgres
34+
35+
AWS_REGION=us-east-1
36+
AWS_ACCESS_KEY_ID=_peerdb_minioadmin
37+
AWS_SECRET_ACCESS_KEY=_peerdb_minioadmin
38+
AWS_ENDPOINT_URL_S3=http://host.docker.internal:9001
39+
AWS_EC2_METADATA_DISABLED="true"
40+
41+
MINIO_ROOT_USER=_peerdb_minioadmin
42+
MINIO_ROOT_PASSWORD=_peerdb_minioadmin

‎.github/workflows/flow.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -488,6 +488,11 @@ jobs:
488488
PEERDB_CATALOG_USER: postgres
489489
PEERDB_CATALOG_PASSWORD: postgres
490490
PEERDB_CATALOG_DATABASE: postgres
491+
PG_HOST: localhost
492+
PG_PORT: 5432
493+
PG_USER: postgres
494+
PG_PASSWORD: postgres
495+
PG_DATABASE: postgres
491496
PEERDB_SWITCHBOARD_ENABLED: "true"
492497
PEERDB_QUEUE_FORCE_TOPIC_CREATION: "true"
493498
ELASTICSEARCH_TEST_ADDRESS: http://localhost:9200

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,3 +19,10 @@ go.work.sum
1919
ui/grpc_generated
2020
flow/generated
2121
nexus/pt/src/gen
22+
23+
# generated ancillary images file
24+
ancillary.env
25+
26+
# Ephemeral volumes for docker compose
27+
volumes/ch_*
28+

‎README.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,73 @@ You can use Postgres’ ecosystem to manage your ETL —
9191
9292
We have expanded our connector ecosystem to support multiple source connectors beyond Postgres, including MySQL and MongoDB. You can check the status of connectors [here](https://docs.peerdb.io/sql/commands/supported-connectors)
9393
94+
## Local End to End testing
95+
96+
You can run locally the same end-to-end tests that our CI uses to validate changes, enabling fast iteration cycles during development.
97+
98+
For example:
99+
100+
```bash
101+
cd flow
102+
go clean -cache
103+
env -f ../.env go test -v -run TestGenericCH_MySQL ./e2e/
104+
```
105+
106+
Or local debugging sessions.
107+
108+
These tests require both PeerDB services, source and destination stores to be running. We provide a local environment with all the necessary services and dependencies to run these tests.
109+
110+
This is done through [Tilt](https://tilt.dev/) orchestrated Docker compose.
111+
112+
To get the environment up you first need to specify the shared environment variables for both the test and the test environment in your local `.env` file. You can use the provided `.env.example` as a template: `cp .env.example .env `.
113+
114+
:memo: In the template, services URLs are set to `host.docker.internal`, which is the name for the default Docker gateway in Docker Desktop set-ups such as macOS and Windows. Using the default gateway address allows both test processes and services running inside Docker to access services on the host machine. In native Docker (Linux) this name is not resolved by default, you might replace it with the default gateway IP (e.g., `172.18.0.1`) or add a custom entry to your `/etc/hosts` file to resolve `host.docker.internal` to the appropriate IP address. e.g:
115+
116+
```bash
117+
echo "172.18.0.1 host.docker.internal" | sudo tee -a /etc/hosts
118+
```
119+
120+
Then you can just run:
121+
122+
```bash
123+
./tilt.sh
124+
```
125+
126+
And follow the status of the services and access logs through the Tilt UI at http://localhost:10352/. [Dozzle](https://dozzle.dev/) is also included at http://localhost:8118/, providing real-time container resource utilization metrics (CPU, memory) and log streaming for all running Docker containers.
127+
128+
<img width="1593" height="693" alt="image" src="https://github.com/user-attachments/assets/6c294dda-ca8f-45cc-b75c-11594118a641" />
129+
130+
Since `.env` is the environment configuration source of truth, it can be used directly to inject the required variables to the test execution processes. e.g:
131+
132+
```bash
133+
go clean -cache; env -f ../.env go test -v -run TestGenericCH_MySQL ./e2e/ # Some MySQL generic tests
134+
```
135+
136+
### Running tests from Tilt
137+
138+
The Tilt setup includes pre-configured test launcher resources under the `e2e` label. These resources do not start automatically; instead, you can trigger them on demand from the Tilt UI at http://localhost:10352/.
139+
140+
<img width="964" height="265" alt="image" src="https://github.com/user-attachments/assets/04ecad65-5d60-44c3-96ad-d285b2706545" />
141+
142+
Available test launchers:
143+
144+
- **e2e_postgres** -- Postgres to ClickHouse generic tests (`TestGenericCH_PG`)
145+
- **e2e_mysql** -- MySQL to ClickHouse generic tests (`TestGenericCH_MySQL`)
146+
- **e2e_mongodb** -- MongoDB to ClickHouse test suite (`TestMongoClickhouseSuite`)
147+
148+
Each launcher automatically depends on the required services and provisioning steps, so Tilt will ensure all prerequisites are running before executing the tests. To trigger a test, click the resource in the Tilt UI and press the trigger (play) button.
149+
150+
### Environment services versions
151+
152+
Data stores versions are extracted from `.github/workflows/flow.yml`, select the last row of the test matrix except for MySQL version which defaults to `9.5`.
153+
This automatic extraction relies on the `yq` CLI; install the Go-based [`mikefarah/yq`](https://github.com/mikefarah/yq) version 4 or later so that local environment generation works correctly.
154+
You can specify different versions in the local `.env` file to override them as follows:
155+
156+
```bash
157+
MYSQL_VERSION=8.0
158+
MONGODB_VERSION=4.4
159+
CLICKHOUSE_VERSION=21.8
160+
```
94161

95162
## Support
96163

‎Tiltfile‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,3 +58,112 @@ dc_resource('flow-worker', resource_deps=['proto-gen'], labels=['PeerDB'])
5858
dc_resource('flow-snapshot-worker', resource_deps=['proto-gen'], labels=['PeerDB'])
5959
dc_resource('peerdb', resource_deps=['proto-gen'], labels=['PeerDB'])
6060
dc_resource('minio', labels=['PeerDB'])
61+
62+
63+
# Ancillary services
64+
65+
local_resource(
66+
'provision-mongodb',
67+
cmd='./local_provision_scripts/mongodb.sh',
68+
labels=['Ancillary', 'Provisioning'],
69+
resource_deps=['mongodb']
70+
)
71+
72+
local_resource(
73+
'provision-clickhouse',
74+
cmd='./local_provision_scripts/clickhouse.sh',
75+
labels=['Ancillary', 'Provisioning'],
76+
resource_deps=['clickhouse']
77+
)
78+
79+
local_resource(
80+
'provision-mysql-gtid',
81+
cmd='./local_provision_scripts/mysql.sh peerdb-mysql-gtid 3306',
82+
labels=['Ancillary', 'Provisioning'],
83+
resource_deps=['mysql-gtid']
84+
)
85+
86+
local_resource(
87+
'provision-mysql-pos',
88+
cmd='./local_provision_scripts/mysql.sh peerdb-mysql-pos 3307',
89+
labels=['Ancillary', 'Provisioning'],
90+
resource_deps=['mysql-pos']
91+
)
92+
93+
local_resource(
94+
'provision-mariadb',
95+
cmd='./local_provision_scripts/mysql.sh peerdb-mariadb 3308',
96+
labels=['Ancillary', 'Provisioning'],
97+
resource_deps=['mariadb']
98+
)
99+
100+
local_resource(
101+
'provision-postgres',
102+
cmd='./local_provision_scripts/postgres.sh',
103+
labels=['Ancillary', 'Provisioning'],
104+
resource_deps=['postgres']
105+
)
106+
107+
# This is not defined as a resource as we need the file to be present
108+
# when `docker_compose` loads the configuration (next line).
109+
local('./generate-test-environment.sh')
110+
111+
tiltfile_dir = config.main_path.replace('/Tiltfile', '')
112+
docker_compose('./ancillary-docker-compose.yml', env_file=tiltfile_dir + '/ancillary.env')
113+
114+
# ClickHouse is the only datastore that is automatically started as the common destination for all pipes.
115+
dc_resource('clickhouse', labels=['Ancillary', 'DataStore'], links=[
116+
link('http://localhost:11123', 'ClickHouse HTTP'),
117+
link('http://localhost:11000', 'ClickHouse TCP'),
118+
])
119+
120+
# Source data storages for tests, they are not automatically started to save resources.
121+
# Their provisioning step resources are autostarted but blocked on the manual activation
122+
# of their corresponding datastore.
123+
# This way, users can choose which ones to start and when, depending on the tests they want to run.
124+
125+
dc_resource('mongodb', labels=['Ancillary', 'DataStore'], links=[
126+
link('http://localhost:11017', 'MongoDB'),
127+
], auto_init=False)
128+
129+
dc_resource('mysql-gtid', labels=['Ancillary', 'DataStore'], links=[
130+
link('http://localhost:3306', 'MySQL GTID'),
131+
], auto_init=False)
132+
133+
dc_resource('mysql-pos', labels=['Ancillary', 'DataStore'], links=[
134+
link('http://localhost:3307', 'MySQL File-Pos'),
135+
], auto_init=False)
136+
137+
dc_resource('mariadb', labels=['Ancillary', 'DataStore'], links=[
138+
link('http://localhost:3308', 'MariaDB'),
139+
], auto_init=False)
140+
141+
dc_resource('postgres', labels=['Ancillary', 'DataStore'], links=[
142+
link('http://localhost:5432', 'PostgreSQL'),
143+
], auto_init=False)
144+
145+
# Monitoring and utility tools
146+
147+
dc_resource('dozzle', labels=['Ancillary', 'Monitoring'], links=[
148+
link('http://localhost:8118', 'Dozzle Container Monitor'),
149+
])
150+
151+
# Tests launchers
152+
153+
def e2e_test(name, test_run, extra_deps=[]):
154+
local_resource(
155+
'e2e_' + name,
156+
cmd='cd flow && go clean -cache && env -f ../.env go test -v -run %s ./e2e/' % test_run,
157+
labels=['e2e', 'Test'],
158+
auto_init=False,
159+
resource_deps=['flow-api', 'flow-worker', 'catalog'] + extra_deps,
160+
)
161+
162+
# Postgres to ClickHouse generic tests
163+
e2e_test('postgres', 'TestGenericCH_PG', ['provision-postgres'])
164+
165+
# MySQL to ClickHouse generic tests
166+
e2e_test('mysql', 'TestGenericCH_MySQL', ['provision-mysql-gtid', 'provision-mysql-pos', 'provision-mariadb'])
167+
168+
# MongoDB to ClickHouse test suite
169+
e2e_test('mongodb', 'TestMongoClickhouseSuite', ['provision-mongodb'])

‎ancillary-docker-compose.yml‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
name: peerdb-ancillary-services
2+
3+
services:
4+
mongodb:
5+
container_name: peerdb-mongodb
6+
image: ${MONGODB_IMAGE}
7+
entrypoint: ["bash", "-c", "openssl rand -base64 756 > /data/mongo.key && chmod 400 /data/mongo.key && exec mongod --replSet rs0 --oplogMinRetentionHours 24 --bind_ip_all --keyFile /data/mongo.key"]
8+
restart: unless-stopped
9+
environment:
10+
MONGO_INITDB_ROOT_USERNAME: ${CI_MONGO_ADMIN_USERNAME}
11+
MONGO_INITDB_ROOT_PASSWORD: ${CI_MONGO_ADMIN_PASSWORD}
12+
13+
ports:
14+
- "${CI_MONGO_PORT}:27017"
15+
extra_hosts:
16+
- "host.docker.internal:host-gateway"
17+
healthcheck:
18+
test: ["CMD", "mongosh", "--eval", "db.runCommand({ ping: 1 })"]
19+
interval: 2s
20+
timeout: 10s
21+
retries: 5
22+
23+
clickhouse:
24+
container_name: peerdb-clickhouse
25+
image: ${CLICKHOUSE_IMAGE}
26+
restart: unless-stopped
27+
ports:
28+
- "${CI_CLICKHOUSE_HTTP_PORT}:8123"
29+
- "${CI_CLICKHOUSE_NATIVE_PORT}:9000"
30+
volumes:
31+
- ./volumes/ch_data:/var/lib/clickhouse/
32+
- ./volumes/ch_logs:/var/log/clickhouse-server/
33+
- ./volumes/ch_users.d:/etc/clickhouse-server/users.d/
34+
ulimits:
35+
nofile:
36+
soft: 262144
37+
hard: 262144
38+
extra_hosts:
39+
- "host.docker.internal:host-gateway"
40+
healthcheck:
41+
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8123/ping"]
42+
interval: 2s
43+
timeout: 10s
44+
retries: 5
45+
46+
mysql-gtid:
47+
container_name: peerdb-mysql-gtid
48+
image: mysql:9.5
49+
restart: unless-stopped
50+
ports:
51+
- "${CI_MYSQL_GTID_PORT}:3306"
52+
environment:
53+
MYSQL_ROOT_PASSWORD: ${CI_MYSQL_ROOT_PASSWORD}
54+
extra_hosts:
55+
- "host.docker.internal:host-gateway"
56+
healthcheck:
57+
test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
58+
interval: 2s
59+
timeout: 10s
60+
retries: 5
61+
62+
mysql-pos:
63+
container_name: peerdb-mysql-pos
64+
image: mysql:5.7
65+
command: ["--log_bin=mysql-bin", "--server-id=1", "--bind-address=::"]
66+
restart: unless-stopped
67+
ports:
68+
- "${CI_MYSQL_POS_PORT}:3306"
69+
environment:
70+
MYSQL_ROOT_PASSWORD: ${CI_MYSQL_ROOT_PASSWORD}
71+
extra_hosts:
72+
- "host.docker.internal:host-gateway"
73+
healthcheck:
74+
test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
75+
interval: 2s
76+
timeout: 10s
77+
retries: 5
78+
79+
mariadb:
80+
container_name: peerdb-mariadb
81+
image: mariadb:lts-ubi9@sha256:55a81b2d791d2ff8ad33fef413d9e45e0ac57a951127e0cfc69a8e59f922ba6e
82+
command: ["--log-bin=maria"]
83+
restart: unless-stopped
84+
ports:
85+
- "${CI_MARIADB_PORT}:3306"
86+
environment:
87+
MARIADB_ROOT_PASSWORD: ${CI_MYSQL_ROOT_PASSWORD}
88+
extra_hosts:
89+
- "host.docker.internal:host-gateway"
90+
healthcheck:
91+
test: ["CMD", "mariadb-admin", "ping", "-h", "localhost"]
92+
interval: 2s
93+
timeout: 10s
94+
retries: 5
95+
96+
postgres:
97+
container_name: peerdb-postgres
98+
image: ${POSTGRES_IMAGE}
99+
restart: unless-stopped
100+
ports:
101+
- "${PG_PORT}:5432"
102+
environment:
103+
PGUSER: ${PG_USER}
104+
POSTGRES_PASSWORD: ${PG_PASSWORD}
105+
POSTGRES_DB: ${PG_DATABASE}
106+
POSTGRES_INITDB_ARGS: --locale=C.UTF-8
107+
extra_hosts:
108+
- "host.docker.internal:host-gateway"
109+
healthcheck:
110+
test: ["CMD", "pg_isready"]
111+
interval: 2s
112+
timeout: 10s
113+
retries: 5
114+
115+
dozzle:
116+
container_name: dozzle-compose-monitor
117+
image: amir20/dozzle:latest
118+
restart: unless-stopped
119+
ports:
120+
- "8118:8080"
121+
volumes:
122+
- /var/run/docker.sock:/var/run/docker.sock
123+
124+
125+
networks:
126+
default:
127+
name: peerdb_network

0 commit comments

Comments
 (0)