Skip to content

Commit 1908e15

Browse files
authored
docs: Add the ClickHouse Cloud guide (#59)
1 parent 172869a commit 1908e15

5 files changed

Lines changed: 209 additions & 4 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,7 @@ Although any accessible database is supported, we also have guides for integrati
123123
- [Neon](docs/integrations/neon.md)
124124
- [PlanetScale](docs/integrations/planetscale.md), Postgres and Vitess
125125
- [MongoDB Atlas](docs/integrations/mongodb-atlas.md)
126+
- [ClickHouse Cloud](docs/integrations/clickhouse-cloud.md)
126127

127128
## How it works
128129

‎docs/getting-started.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,13 @@ so you can query users, organizations, memberships and directory data with SQL
55
and join them to your own tables.
66

77
This guide takes you from an empty database to a mirror that stays current.
8+
It shows Postgres; the commands in it have MongoDB, MySQL and ClickHouse
9+
counterparts, with options that differ where the database does (MongoDB, for
10+
one, has no `init --print-sql`), and each package's README is the reference
11+
for it:
12+
[`@workos/sync-mongodb`](../packages/sync-mongodb/README.md),
13+
[`@workos/sync-mysql`](../packages/sync-mysql/README.md) and
14+
[`@workos/sync-clickhouse`](../packages/sync-clickhouse/README.md).
815

916
## Before you start
1017

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
# ClickHouse Cloud
2+
3+
[ClickHouse Cloud](https://clickhouse.com/cloud) is ClickHouse's hosted
4+
service. The engine's ClickHouse destination reaches it over the HTTP
5+
interface with the official client, so this guide is about the choices the
6+
console puts in front of you: the service tier, the user, the IP access
7+
list, which endpoint to copy, and what idling and a second replica mean for
8+
the sync.
9+
10+
## Create the service
11+
12+
Create a service in the console, in an organization set aside for the
13+
mirror or the one your analytics already use: the engine creates its own
14+
`workos` database and touches nothing else. Choose a cloud and a region
15+
close to where the sync will run, and a tier:
16+
17+
| Tier | Suits |
18+
| -------------- | ----------------------------------------------------------------------------------------------------------------------- |
19+
| **Basic** | Evaluating the sync. One replica of a fixed size. |
20+
| **Scale** | A production mirror. Two or more replicas, autoscaling, backups you configure. The tier this guide was written against. |
21+
| **Enterprise** | Scale, plus scheduled upgrades, the slow release channel and compliance features. |
22+
23+
A new organization starts on a trial with a credit allowance that runs out
24+
after 30 days at the time of writing; the console's banner counts the days
25+
down, and the service keeps running past them only once payment details are
26+
in. Any tier runs a current ClickHouse (26.6 on the service used here), well
27+
past the 23.11 the package needs, and every service has the Keeper and the
28+
`keeper_map_path_prefix` that the engine's lease, cursor and write-lock
29+
tables depend on. There is nothing to enable: `init` creates its
30+
[KeeperMap](../../packages/sync-clickhouse/README.md#requirements) tables
31+
like any other.
32+
33+
The `default` user's password is shown once, in the **Connect** dialog
34+
when the service is created, and the dialog says so afterwards. If it is
35+
gone, **Settings → Reset password** issues a new one and invalidates the
36+
old one for every client that had it.
37+
38+
## Database user
39+
40+
The `default` user is the service's administrator and is what this guide
41+
was verified with; it is more than the sync needs and fine for an
42+
evaluation. For production, create a user in the SQL console with the
43+
grants the
44+
[package README](../../packages/sync-clickhouse/README.md#command-line)
45+
lists: `CREATE DATABASE` the first time, or an existing database it may
46+
create tables in, plus `CREATE TABLE`, `CREATE VIEW`, `DROP TABLE`, `ALTER`,
47+
`INSERT` and `SELECT` on that database. `ALTER` covers the lightweight
48+
`DELETE` that `redact` runs and the `ADD COLUMN` and `ADD INDEX` a later
49+
release may need. The bookkeeping tables are KeeperMap tables, created with
50+
the same `CREATE TABLE` grant. A dedicated user has not been tried against
51+
Cloud; the suite below is how to confirm one.
52+
53+
## Network access
54+
55+
**Settings → Security → IP access list** is where the service decides who
56+
may connect. A refused address looks like a connection timeout, not an
57+
error about access, so add the address the sync connects from before the
58+
first `init`, and again when it moves. The console warns that a service
59+
open to anywhere can be woken by internet scanners, and a woken service
60+
bills compute, which is a second reason to list addresses rather than open
61+
it. A host with a changing address is better served by a private endpoint,
62+
also under **Security**.
63+
64+
Only ports 8443 (HTTPS) and 9440 (the native protocol) are exposed. A
65+
network that allows outbound traffic on 443 alone cannot reach the service
66+
and shows the same timeout as a missing access-list entry. Tell the two
67+
apart from a shell:
68+
69+
```sh
70+
curl https://HOST.clickhouse.cloud:8443/ping # prints "Ok." when the port is reachable
71+
```
72+
73+
## Choose a connection
74+
75+
**Connect** on the service, with **HTTPS** selected, shows a `curl`
76+
command whose last line is the endpoint:
77+
78+
```
79+
https://abc123.us-east1.gcp.clickhouse.cloud:8443
80+
```
81+
82+
The engine takes the same endpoint with the credentials in front:
83+
84+
```
85+
https://default:PASSWORD@abc123.us-east1.gcp.clickhouse.cloud:8443
86+
```
87+
88+
Percent-encode the password if it has characters that mean something in a
89+
URL; the engine decodes it. A path in the URL is ignored: the mirror lives
90+
in the `workos` database the engine creates, or whatever
91+
`WORKOS_SYNC_SCHEMA` or `--schema` names. Only `http://` and `https://` are
92+
accepted; the native protocol on 9440, which `clickhouse client` uses, is
93+
not an option for the engine.
94+
95+
Every connection the engine opens carries the settings a multi-replica
96+
service needs and nothing on the service has to change for them: inserts
97+
are asynchronous but waited for, mutations return once every replica has
98+
them, and reads are sequentially consistent, so the write path never reads
99+
a replica the last insert has not reached.
100+
101+
## TLS
102+
103+
The endpoint is HTTPS, and Cloud's certificates chain to public roots that
104+
Node already trusts, so `WORKOS_SYNC_CLICKHOUSE_TLS_CA` stays unset and
105+
there is no `sslCa` to pass in embedded code. There is no plaintext port to
106+
fall back to.
107+
108+
## Idling
109+
110+
A service idles after a stretch without queries, 15 minutes by default
111+
(**Settings → Scaling → Idling**), and wakes on the next request, which
112+
waits while it does. Two consequences for the sync:
113+
114+
- **`listen` keeps the service awake.** It polls every few seconds, so a
115+
service with a listener on it never idles and bills compute the whole
116+
time. That is the normal cost of a tail. `tick` on a schedule lets the
117+
service idle between runs at the price of a wake per run.
118+
- **A cold first command waits for the wake.** The engine allows two
119+
minutes per request; a `status` or `init` against an idle service spends
120+
the first seconds of that waiting, which is not a failure.
121+
122+
## Verify the endpoint
123+
124+
From a checkout of this repository, after `pnpm install` and `pnpm build`:
125+
126+
```sh
127+
TEST_CLICKHOUSE_URL='https://default:PASSWORD@abc123.us-east1.gcp.clickhouse.cloud:8443' \
128+
pnpm --filter @workos/sync-clickhouse test:database
129+
```
130+
131+
With that variable set, the suite runs against your service instead of a
132+
local server. Each test creates a `workos_test_<hex>` database and drops it
133+
afterwards, which is why the user running the suite needs `CREATE DATABASE`
134+
and `DROP DATABASE`; the KeeperMap tables keep their Keeper paths under the
135+
database's name, so parallel tests never share a lease. WorkOS responses
136+
are local fixtures, so no key is involved. Expect it to take around ten
137+
minutes: every statement is an HTTPS round trip to the region.
138+
139+
Against a two-replica Scale service in November 2026 the suite passed
140+
except for one test. That test applies the script `init --print-sql`
141+
prints statement by statement, each as its own HTTP request, and on a
142+
service with more than one replica a `CREATE VIEW` reached a replica before
143+
the `CREATE TABLE` it selects from had been replicated to it. The engine's
144+
own `init`, which the rest of the suite runs many times against the same
145+
service, did not hit it. If you apply the printed script by hand and a
146+
statement fails this way, run the script again: every statement in it is
147+
idempotent, and the second pass completes what the first could not.
148+
149+
## Run the sync from your own host
150+
151+
```sh
152+
export DATABASE_URL='https://default:PASSWORD@abc123.us-east1.gcp.clickhouse.cloud:8443'
153+
export WORKOS_API_KEY=sk_...
154+
155+
npx workos-sync-clickhouse init
156+
npx workos-sync-clickhouse backfill
157+
npx workos-sync-clickhouse listen
158+
```
159+
160+
The [container image](../../README.md#against-a-real-environment) and the
161+
[library](../../packages/sync-clickhouse/README.md#embed-it) take the same
162+
URL. Two things to plan for:
163+
164+
- **Start the tail early.** The sign-in history in `auth_attempts` is the
165+
reason to mirror into ClickHouse, and it begins the day `listen` first
166+
runs: WorkOS keeps events for 90 days and has no endpoint that lists past
167+
sign-ins, so `backfill` cannot recover them.
168+
- **The service will not idle while `listen` runs**, as above. Size the
169+
service for a process that is always connected, and raise
170+
`WORKOS_SYNC_MAX_EVENTS_PER_TICK` so each poll is a few larger inserts
171+
rather than many small ones.
172+
173+
## Query it
174+
175+
The questions in [Querying the mirror](../queries.md) apply, in
176+
ClickHouse's terms: that guide is written for Postgres, so its JSON
177+
operators become `JSONExtractString(_raw, 'field')`, its date arithmetic
178+
becomes ClickHouse's, and a reader's grant is `GRANT SELECT ON workos.* TO
179+
...` rather than a schema grant. The tables and views are the same. The
180+
service's **SQL console** lists the `workos` database next to `default`
181+
once `init` has run; the `_live` views read with `FINAL` and hide
182+
tombstones. The base tables keep the newest version of each row, tombstones
183+
included, so "who left" is a query on `workos.users` where `_deleted_at` is
184+
set; older versions survive only until the next merge, and
185+
`workos.events` is where history is kept. From your own machine,
186+
`clickhouse client` takes the host with `--secure` and the password, and
187+
`curl` takes the endpoint from **Connect** with a query as its body:
188+
189+
```sql
190+
SELECT _env, email FROM workos.users_live;
191+
```
192+
193+
The [`@workos/sync-clickhouse` README](../../packages/sync-clickhouse/README.md)
194+
opens with the query this destination exists for, failed sign-ins per
195+
organization per hour joined across the mirror tables, and documents the
196+
schema those queries read.

‎docs/integrations/mongodb-atlas.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -109,8 +109,8 @@ npx workos-sync-mongodb listen
109109
```
110110

111111
The [container image](../../README.md#against-a-real-environment) and the
112-
[library](../getting-started.md#embed-it) take the same URL. Two things
113-
to plan for:
112+
[library](../../packages/sync-mongodb/README.md#embed-it) take the same
113+
URL. Two things to plan for:
114114

115115
- **A free cluster must stay in use.** Atlas pauses an M0 that has had no
116116
connections for a long stretch and resumes it from the console. A

‎packages/sync-clickhouse/README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -196,8 +196,9 @@ merge time and under `FINAL`; that is the staleness gate, enforced by the
196196
table whatever order writes land in. A deletion's tombstone carries the deletion time as its version, so it outranks every earlier version of the row. Each mirror table has a generated
197197
`<table>_live` view — `workos.users_live`, `workos.organizations_live`, and so
198198
on — that reads with `FINAL` and hides tombstoned rows. **Query the views.**
199-
The base table shows every version a row ever had, plus its tombstones
200-
(`_deleted_at`), for when you want to ask who left.
199+
The base table keeps the newest version of each row, tombstones
200+
(`_deleted_at`) included, for when you want to ask who left; older versions
201+
survive only until the next merge. `workos.events` is where history is kept.
201202

202203
A field added upstream can never break ingest — it is queryable as
203204
`JSONExtractString(_raw, 'field')` immediately, and promoting it to a typed

0 commit comments

Comments
 (0)