Skip to content

Commit a8ea9e9

Browse files
authored
feat(clickhouse): add the ClickHouse destination package (#46)
1 parent 9f0c1d0 commit a8ea9e9

48 files changed

Lines changed: 7282 additions & 13 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,3 +186,53 @@ jobs:
186186

187187
- name: Build
188188
run: pnpm --filter "@workos/sync-mysql..." build
189+
190+
clickhouse:
191+
name: ClickHouse Node ${{ matrix.node }}
192+
runs-on: ubuntu-latest
193+
strategy:
194+
fail-fast: false
195+
matrix:
196+
node: [22, 24]
197+
env:
198+
# There is no embedded ClickHouse, so the whole suite runs against the
199+
# container; without this variable the database-backed tests skip.
200+
TEST_CLICKHOUSE_URL: http://default@127.0.0.1:8123
201+
steps:
202+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
203+
with:
204+
persist-credentials: false
205+
206+
# Not a service container: those start before the checkout, and the
207+
# server needs the config in test/clickhouse/ — an embedded Keeper for
208+
# the KeeperMap bookkeeping tables and `keeper_map_path_prefix`.
209+
- name: Start ClickHouse
210+
run: |
211+
docker run -d --name clickhouse -p 127.0.0.1:8123:8123 \
212+
-v "$PWD/packages/sync-clickhouse/test/clickhouse/config.xml:/etc/clickhouse-server/config.d/sync.xml:ro" \
213+
-v "$PWD/packages/sync-clickhouse/test/clickhouse/users.xml:/etc/clickhouse-server/users.d/sync.xml:ro" \
214+
-e CLICKHOUSE_SKIP_USER_SETUP=1 \
215+
--ulimit nofile=262144:262144 \
216+
clickhouse/clickhouse-server:26.10
217+
for _ in $(seq 1 60); do
218+
if curl -fsS http://127.0.0.1:8123/ping >/dev/null 2>&1; then exit 0; fi
219+
sleep 1
220+
done
221+
echo "ClickHouse did not become ready" >&2
222+
docker logs clickhouse >&2
223+
exit 1
224+
225+
- uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
226+
227+
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
228+
with:
229+
node-version: ${{ matrix.node }}
230+
231+
- name: Install
232+
run: pnpm install --frozen-lockfile
233+
234+
- name: Test
235+
run: pnpm --filter @workos/sync-clickhouse test:database
236+
237+
- name: Build
238+
run: pnpm --filter "@workos/sync-clickhouse..." build

‎.github/workflows/docker.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ jobs:
2424
- postgres
2525
- mongodb
2626
- mysql
27+
- clickhouse
2728
runner: [ubuntu-latest, ubuntu-24.04-arm]
2829
steps:
2930
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

‎.github/workflows/release-please.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ jobs:
3939
sync-mysql--tag_name: ${{ steps.release.outputs['packages/sync-mysql--tag_name'] }}
4040
sync-mongodb--release_created: ${{ steps.release.outputs['packages/sync-mongodb--release_created'] }}
4141
sync-mongodb--tag_name: ${{ steps.release.outputs['packages/sync-mongodb--tag_name'] }}
42+
sync-clickhouse--release_created: ${{ steps.release.outputs['packages/sync-clickhouse--release_created'] }}
43+
sync-clickhouse--tag_name: ${{ steps.release.outputs['packages/sync-clickhouse--tag_name'] }}
4244
steps:
4345
# Release PRs and tags come from the SDK Automation Bot, as in workos-node.
4446
# Events raised with GITHUB_TOKEN start no workflow runs, so a release PR
@@ -108,3 +110,15 @@ jobs:
108110
tag_name: ${{ needs.release-please.outputs.sync-mongodb--tag_name }}
109111
secrets:
110112
PUBLIC_SOCKET_FIREWALL_TOKEN: ${{ secrets.PUBLIC_SOCKET_FIREWALL_TOKEN }}
113+
114+
publish-sync-clickhouse:
115+
needs: [release-please, publish-sync-engine]
116+
if: >-
117+
${{ !cancelled()
118+
&& needs.publish-sync-engine.result != 'failure'
119+
&& needs.release-please.outputs.sync-clickhouse--release_created == 'true' }}
120+
uses: ./.github/workflows/release.yml
121+
with:
122+
tag_name: ${{ needs.release-please.outputs.sync-clickhouse--tag_name }}
123+
secrets:
124+
PUBLIC_SOCKET_FIREWALL_TOKEN: ${{ secrets.PUBLIC_SOCKET_FIREWALL_TOKEN }}

‎.release-please-manifest.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,6 @@
22
"packages/sync-engine": "0.1.0",
33
"packages/sync-postgres": "0.1.0",
44
"packages/sync-mongodb": "0.0.0",
5-
"packages/sync-mysql": "0.0.0"
5+
"packages/sync-mysql": "0.0.0",
6+
"packages/sync-clickhouse": "0.0.1"
67
}

‎README.md‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -31,15 +31,16 @@ base tables keep them as tombstones for when you want to ask who left.
3131
One engine, one package per database. Each ships a CLI with the same
3232
commands and a container image under the same name:
3333

34-
| Destination | Package | Mirror |
35-
| ----------- | ------------------------------------------------- | ----------------------------------------------------------------------- |
36-
| Postgres | [`@workos/sync-postgres`](packages/sync-postgres) | A `workos` schema of tables with generated columns and `_live` views |
37-
| MongoDB | [`@workos/sync-mongodb`](packages/sync-mongodb) | `workos_*` collections in your database, with indexes and `_live` views |
38-
| MySQL | [`@workos/sync-mysql`](packages/sync-mysql) | A `workos` database of tables with generated columns and `_live` views |
34+
| Destination | Package | Mirror |
35+
| ----------- | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
36+
| Postgres | [`@workos/sync-postgres`](packages/sync-postgres) | A `workos` schema of tables with generated columns and `_live` views |
37+
| MongoDB | [`@workos/sync-mongodb`](packages/sync-mongodb) | `workos_*` collections in your database, with indexes and `_live` views |
38+
| MySQL | [`@workos/sync-mysql`](packages/sync-mysql) | A `workos` database of tables with generated columns and `_live` views |
39+
| ClickHouse | [`@workos/sync-clickhouse`](packages/sync-clickhouse) | A `workos` database of `ReplacingMergeTree` tables with `_live` views, plus append-only tables for the sign-in log and events; for history that outlives the 90-day retention |
3940

4041
The rest of this page shows Postgres; every command and option below has a
41-
MongoDB and a MySQL counterpart, and each package's README is the reference
42-
for it.
42+
MongoDB, a MySQL and a ClickHouse counterpart, and each package's README is
43+
the reference for it.
4344

4445
## Try it in 30 seconds
4546

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 WorkOS
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

0 commit comments

Comments
 (0)