|
| 1 | +# ClickHouse Cloud |
| 2 | + |
| 3 | +[ClickHouse Cloud](https://clickhouse.com/cloud) is ClickHouse's hosted |
| 4 | +service. The engine's ClickHouse destination reaches it over the HTTP |
| 5 | +interface with the official client, so this guide is about the choices the |
| 6 | +console puts in front of you: the service tier, the user, the IP access |
| 7 | +list, which endpoint to copy, and what idling and a second replica mean for |
| 8 | +the sync. |
| 9 | + |
| 10 | +## Create the service |
| 11 | + |
| 12 | +Create a service in the console, in an organization set aside for the |
| 13 | +mirror or the one your analytics already use: the engine creates its own |
| 14 | +`workos` database and touches nothing else. Choose a cloud and a region |
| 15 | +close to where the sync will run, and a tier: |
| 16 | + |
| 17 | +| Tier | Suits | |
| 18 | +| -------------- | ----------------------------------------------------------------------------------------------------------------------- | |
| 19 | +| **Basic** | Evaluating the sync. One replica of a fixed size. | |
| 20 | +| **Scale** | A production mirror. Two or more replicas, autoscaling, backups you configure. The tier this guide was written against. | |
| 21 | +| **Enterprise** | Scale, plus scheduled upgrades, the slow release channel and compliance features. | |
| 22 | + |
| 23 | +A new organization starts on a trial with a credit allowance that runs out |
| 24 | +after 30 days at the time of writing; the console's banner counts the days |
| 25 | +down, and the service keeps running past them only once payment details are |
| 26 | +in. Any tier runs a current ClickHouse (26.6 on the service used here), well |
| 27 | +past the 23.11 the package needs, and every service has the Keeper and the |
| 28 | +`keeper_map_path_prefix` that the engine's lease, cursor and write-lock |
| 29 | +tables depend on. There is nothing to enable: `init` creates its |
| 30 | +[KeeperMap](../../packages/sync-clickhouse/README.md#requirements) tables |
| 31 | +like any other. |
| 32 | + |
| 33 | +The `default` user's password is shown once, in the **Connect** dialog |
| 34 | +when the service is created, and the dialog says so afterwards. If it is |
| 35 | +gone, **Settings → Reset password** issues a new one and invalidates the |
| 36 | +old one for every client that had it. |
| 37 | + |
| 38 | +## Database user |
| 39 | + |
| 40 | +The `default` user is the service's administrator and is what this guide |
| 41 | +was verified with; it is more than the sync needs and fine for an |
| 42 | +evaluation. For production, create a user in the SQL console with the |
| 43 | +grants the |
| 44 | +[package README](../../packages/sync-clickhouse/README.md#command-line) |
| 45 | +lists: `CREATE DATABASE` the first time, or an existing database it may |
| 46 | +create tables in, plus `CREATE TABLE`, `CREATE VIEW`, `DROP TABLE`, `ALTER`, |
| 47 | +`INSERT` and `SELECT` on that database. `ALTER` covers the lightweight |
| 48 | +`DELETE` that `redact` runs and the `ADD COLUMN` and `ADD INDEX` a later |
| 49 | +release may need. The bookkeeping tables are KeeperMap tables, created with |
| 50 | +the same `CREATE TABLE` grant. A dedicated user has not been tried against |
| 51 | +Cloud; the suite below is how to confirm one. |
| 52 | + |
| 53 | +## Network access |
| 54 | + |
| 55 | +**Settings → Security → IP access list** is where the service decides who |
| 56 | +may connect. A refused address looks like a connection timeout, not an |
| 57 | +error about access, so add the address the sync connects from before the |
| 58 | +first `init`, and again when it moves. The console warns that a service |
| 59 | +open to anywhere can be woken by internet scanners, and a woken service |
| 60 | +bills compute, which is a second reason to list addresses rather than open |
| 61 | +it. A host with a changing address is better served by a private endpoint, |
| 62 | +also under **Security**. |
| 63 | + |
| 64 | +Only ports 8443 (HTTPS) and 9440 (the native protocol) are exposed. A |
| 65 | +network that allows outbound traffic on 443 alone cannot reach the service |
| 66 | +and shows the same timeout as a missing access-list entry. Tell the two |
| 67 | +apart from a shell: |
| 68 | + |
| 69 | +```sh |
| 70 | +curl https://HOST.clickhouse.cloud:8443/ping # prints "Ok." when the port is reachable |
| 71 | +``` |
| 72 | + |
| 73 | +## Choose a connection |
| 74 | + |
| 75 | +**Connect** on the service, with **HTTPS** selected, shows a `curl` |
| 76 | +command whose last line is the endpoint: |
| 77 | + |
| 78 | +``` |
| 79 | +https://abc123.us-east1.gcp.clickhouse.cloud:8443 |
| 80 | +``` |
| 81 | + |
| 82 | +The engine takes the same endpoint with the credentials in front: |
| 83 | + |
| 84 | +``` |
| 85 | +https://default:PASSWORD@abc123.us-east1.gcp.clickhouse.cloud:8443 |
| 86 | +``` |
| 87 | + |
| 88 | +Percent-encode the password if it has characters that mean something in a |
| 89 | +URL; the engine decodes it. A path in the URL is ignored: the mirror lives |
| 90 | +in the `workos` database the engine creates, or whatever |
| 91 | +`WORKOS_SYNC_SCHEMA` or `--schema` names. Only `http://` and `https://` are |
| 92 | +accepted; the native protocol on 9440, which `clickhouse client` uses, is |
| 93 | +not an option for the engine. |
| 94 | + |
| 95 | +Every connection the engine opens carries the settings a multi-replica |
| 96 | +service needs and nothing on the service has to change for them: inserts |
| 97 | +are asynchronous but waited for, mutations return once every replica has |
| 98 | +them, and reads are sequentially consistent, so the write path never reads |
| 99 | +a replica the last insert has not reached. |
| 100 | + |
| 101 | +## TLS |
| 102 | + |
| 103 | +The endpoint is HTTPS, and Cloud's certificates chain to public roots that |
| 104 | +Node already trusts, so `WORKOS_SYNC_CLICKHOUSE_TLS_CA` stays unset and |
| 105 | +there is no `sslCa` to pass in embedded code. There is no plaintext port to |
| 106 | +fall back to. |
| 107 | + |
| 108 | +## Idling |
| 109 | + |
| 110 | +A service idles after a stretch without queries, 15 minutes by default |
| 111 | +(**Settings → Scaling → Idling**), and wakes on the next request, which |
| 112 | +waits while it does. Two consequences for the sync: |
| 113 | + |
| 114 | +- **`listen` keeps the service awake.** It polls every few seconds, so a |
| 115 | + service with a listener on it never idles and bills compute the whole |
| 116 | + time. That is the normal cost of a tail. `tick` on a schedule lets the |
| 117 | + service idle between runs at the price of a wake per run. |
| 118 | +- **A cold first command waits for the wake.** The engine allows two |
| 119 | + minutes per request; a `status` or `init` against an idle service spends |
| 120 | + the first seconds of that waiting, which is not a failure. |
| 121 | + |
| 122 | +## Verify the endpoint |
| 123 | + |
| 124 | +From a checkout of this repository, after `pnpm install` and `pnpm build`: |
| 125 | + |
| 126 | +```sh |
| 127 | +TEST_CLICKHOUSE_URL='https://default:PASSWORD@abc123.us-east1.gcp.clickhouse.cloud:8443' \ |
| 128 | + pnpm --filter @workos/sync-clickhouse test:database |
| 129 | +``` |
| 130 | + |
| 131 | +With that variable set, the suite runs against your service instead of a |
| 132 | +local server. Each test creates a `workos_test_<hex>` database and drops it |
| 133 | +afterwards, which is why the user running the suite needs `CREATE DATABASE` |
| 134 | +and `DROP DATABASE`; the KeeperMap tables keep their Keeper paths under the |
| 135 | +database's name, so parallel tests never share a lease. WorkOS responses |
| 136 | +are local fixtures, so no key is involved. Expect it to take around ten |
| 137 | +minutes: every statement is an HTTPS round trip to the region. |
| 138 | + |
| 139 | +Against a two-replica Scale service in November 2026 the suite passed |
| 140 | +except for one test. That test applies the script `init --print-sql` |
| 141 | +prints statement by statement, each as its own HTTP request, and on a |
| 142 | +service with more than one replica a `CREATE VIEW` reached a replica before |
| 143 | +the `CREATE TABLE` it selects from had been replicated to it. The engine's |
| 144 | +own `init`, which the rest of the suite runs many times against the same |
| 145 | +service, did not hit it. If you apply the printed script by hand and a |
| 146 | +statement fails this way, run the script again: every statement in it is |
| 147 | +idempotent, and the second pass completes what the first could not. |
| 148 | + |
| 149 | +## Run the sync from your own host |
| 150 | + |
| 151 | +```sh |
| 152 | +export DATABASE_URL='https://default:PASSWORD@abc123.us-east1.gcp.clickhouse.cloud:8443' |
| 153 | +export WORKOS_API_KEY=sk_... |
| 154 | + |
| 155 | +npx workos-sync-clickhouse init |
| 156 | +npx workos-sync-clickhouse backfill |
| 157 | +npx workos-sync-clickhouse listen |
| 158 | +``` |
| 159 | + |
| 160 | +The [container image](../../README.md#against-a-real-environment) and the |
| 161 | +[library](../../packages/sync-clickhouse/README.md#embed-it) take the same |
| 162 | +URL. Two things to plan for: |
| 163 | + |
| 164 | +- **Start the tail early.** The sign-in history in `auth_attempts` is the |
| 165 | + reason to mirror into ClickHouse, and it begins the day `listen` first |
| 166 | + runs: WorkOS keeps events for 90 days and has no endpoint that lists past |
| 167 | + sign-ins, so `backfill` cannot recover them. |
| 168 | +- **The service will not idle while `listen` runs**, as above. Size the |
| 169 | + service for a process that is always connected, and raise |
| 170 | + `WORKOS_SYNC_MAX_EVENTS_PER_TICK` so each poll is a few larger inserts |
| 171 | + rather than many small ones. |
| 172 | + |
| 173 | +## Query it |
| 174 | + |
| 175 | +The questions in [Querying the mirror](../queries.md) apply, in |
| 176 | +ClickHouse's terms: that guide is written for Postgres, so its JSON |
| 177 | +operators become `JSONExtractString(_raw, 'field')`, its date arithmetic |
| 178 | +becomes ClickHouse's, and a reader's grant is `GRANT SELECT ON workos.* TO |
| 179 | +...` rather than a schema grant. The tables and views are the same. The |
| 180 | +service's **SQL console** lists the `workos` database next to `default` |
| 181 | +once `init` has run; the `_live` views read with `FINAL` and hide |
| 182 | +tombstones. The base tables keep the newest version of each row, tombstones |
| 183 | +included, so "who left" is a query on `workos.users` where `_deleted_at` is |
| 184 | +set; older versions survive only until the next merge, and |
| 185 | +`workos.events` is where history is kept. From your own machine, |
| 186 | +`clickhouse client` takes the host with `--secure` and the password, and |
| 187 | +`curl` takes the endpoint from **Connect** with a query as its body: |
| 188 | + |
| 189 | +```sql |
| 190 | +SELECT _env, email FROM workos.users_live; |
| 191 | +``` |
| 192 | + |
| 193 | +The [`@workos/sync-clickhouse` README](../../packages/sync-clickhouse/README.md) |
| 194 | +opens with the query this destination exists for, failed sign-ins per |
| 195 | +organization per hour joined across the mirror tables, and documents the |
| 196 | +schema those queries read. |
0 commit comments