|
| 1 | +# Neon |
| 2 | + |
| 3 | +[Neon](https://neon.com) is a hosted Postgres with branching, autoscaling and |
| 4 | +scale to zero. The engine's `pg` destination reaches it over an ordinary |
| 5 | +PostgreSQL connection, so this guide is about the choices Neon puts in front |
| 6 | +of you, and about the one way of running the sync that only Neon offers: as a |
| 7 | +Function on the branch itself, next to the data. |
| 8 | + |
| 9 | +| Run the sync | Suits | |
| 10 | +| ------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | |
| 11 | +| From a host you operate, pointed at Neon | A server, container or cron you already run; `listen` as a long-lived process | |
| 12 | +| On Neon, as a Function from the registry template | No host to run: schedule triggers call `tick`, `backfill` and `reconcile` on the branch, next to the data | |
| 13 | + |
| 14 | +Both start with a database. The first also needs a connection string you |
| 15 | +choose; a Function has one injected. |
| 16 | + |
| 17 | +## Create the database |
| 18 | + |
| 19 | +Create a Neon project for the mirror, or a branch of an existing one. The |
| 20 | +engine creates its own `workos` schema and touches nothing else, so a database |
| 21 | +your app already uses is fine; a branch is the cheap way to evaluate without |
| 22 | +involving it. See [Branches](#branches) for what a branch carries with it. |
| 23 | + |
| 24 | +The project's default role owns the default database, which covers the |
| 25 | +`CREATE` and `TEMPORARY` privileges the engine needs. A role you create |
| 26 | +yourself needs those two grants on the database and must own the `workos` |
| 27 | +schema's objects, so run `init` as that role. |
| 28 | + |
| 29 | +To run the sync as a [Neon Function](#run-the-sync-as-a-neon-function), put |
| 30 | +the project in a region where Functions are available. At the time of writing |
| 31 | +those are `aws-us-east-1`, `aws-us-east-2`, `aws-eu-central-1` and |
| 32 | +`aws-ap-southeast-1`; the |
| 33 | +[Functions overview](https://neon.com/docs/compute/functions/overview) has the |
| 34 | +current list. |
| 35 | + |
| 36 | +## Choose a connection |
| 37 | + |
| 38 | +Neon gives two connection strings for the same database. Both work with the |
| 39 | +engine. |
| 40 | + |
| 41 | +| Connection | Host | Notes | |
| 42 | +| ---------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | |
| 43 | +| Direct | `ep-<id>.<region>.aws.neon.tech` | A simple starting point. | |
| 44 | +| Pooled | `ep-<id>-pooler.<region>.aws.neon.tech` | PgBouncer in transaction mode. Has passed the engine's compatibility suite against a live Neon endpoint. What a Function receives as `DATABASE_URL`. | |
| 45 | + |
| 46 | +Copy the URI from the **Connect** dialog in the console, or print it from a |
| 47 | +linked checkout: |
| 48 | + |
| 49 | +```sh |
| 50 | +neon connection-string --ssl verify-full # direct |
| 51 | +neon connection-string --ssl verify-full --pooled # pooled |
| 52 | +``` |
| 53 | + |
| 54 | +The **Connection pooling** switch in that dialog only chooses which URI is |
| 55 | +shown; it is not a project setting. Keep every parameter the URI comes with, |
| 56 | +such as `channel_binding=require`, use the database password rather than a |
| 57 | +Neon API key, and percent-encode any password character that has meaning in a |
| 58 | +URL: |
| 59 | + |
| 60 | +``` |
| 61 | +postgresql://USER:PASSWORD@ep-<id>-pooler.<region>.aws.neon.tech/neondb?sslmode=verify-full&channel_binding=require |
| 62 | +``` |
| 63 | + |
| 64 | +Transaction pooling is enough because of how the engine uses the database: |
| 65 | +short transactions, transaction-scoped migration locks, row-based writer |
| 66 | +leases and temporary tables created `ON COMMIT DROP` inside a transaction. |
| 67 | +[Hosted Postgres](../docs/hosted-postgres.md#choose-a-connection) has the reasoning. |
| 68 | + |
| 69 | +## TLS |
| 70 | + |
| 71 | +Neon's server certificates chain to ISRG Root X1, the Let's Encrypt root, |
| 72 | +which Node's bundled root store already trusts. `sslmode=verify-full` |
| 73 | +therefore works with no certificate download, and `WORKOS_SYNC_PG_SSL_CA` |
| 74 | +stays unset. In embedded code there is likewise no `sslCa` to pass. |
| 75 | + |
| 76 | +## Verify the endpoint |
| 77 | + |
| 78 | +From a checkout of this repository, after `pnpm install` and `pnpm build`: |
| 79 | + |
| 80 | +```sh |
| 81 | +TEST_DATABASE_URL='postgresql://USER:PASSWORD@ep-<id>-pooler.<region>.aws.neon.tech/neondb?sslmode=verify-full' \ |
| 82 | + pnpm --filter @workos/sync-postgres test:database |
| 83 | +``` |
| 84 | + |
| 85 | +The suite creates a uniquely named schema per test and drops it afterward, |
| 86 | +exercising every stream definition, generated columns, live views, |
| 87 | +transactions, reconciliation, writer leases, checkpoints and tombstones |
| 88 | +against the real endpoint. WorkOS responses are local fixtures, so it makes |
| 89 | +no WorkOS calls and needs no key. Run it once for each URI you intend to use: |
| 90 | +it proves the credentials, network path, TLS and permissions that a local run |
| 91 | +cannot. [Hosted Postgres](../docs/hosted-postgres.md#verify-the-actual-endpoint) has |
| 92 | +more on what it covers. |
| 93 | + |
| 94 | +## Run the sync from your own host |
| 95 | + |
| 96 | +Point the command at Neon as you would at any Postgres: |
| 97 | + |
| 98 | +```sh |
| 99 | +export DATABASE_URL='postgresql://USER:PASSWORD@ep-<id>-pooler.<region>.aws.neon.tech/neondb?sslmode=verify-full&channel_binding=require' |
| 100 | +export WORKOS_API_KEY=sk_... |
| 101 | + |
| 102 | +npx workos-sync-postgres init |
| 103 | +npx workos-sync-postgres backfill |
| 104 | +npx workos-sync-postgres listen |
| 105 | +``` |
| 106 | + |
| 107 | +The [container image](../README.md#against-a-real-environment) and the |
| 108 | +[library](../docs/getting-started.md#embed-it) take the same URL. Two things about |
| 109 | +Neon's compute to plan for: |
| 110 | + |
| 111 | +- **The compute stays awake while the sync runs.** Neon suspends an idle |
| 112 | + compute after five minutes by default and resumes it on the next |
| 113 | + connection. A running sync never leaves that gap: `listen` renews its |
| 114 | + writer lease every 20 seconds, and a `tick` every minute takes and releases |
| 115 | + the lease each time. Budget for the compute to run continuously at its |
| 116 | + minimum size for as long as the sync does. Where a few minutes of lag is |
| 117 | + acceptable, a `tick` on a longer interval than the suspend timeout lets the |
| 118 | + compute sleep between polls, at the cost of a cold start of a few hundred |
| 119 | + milliseconds on each one, which the sync simply waits out. |
| 120 | +- **An IP allow list must include the sync's host.** If the project restricts |
| 121 | + connections by address, add the address the sync connects from before the |
| 122 | + first `init`. |
| 123 | + |
| 124 | +## Run the sync as a Neon Function |
| 125 | + |
| 126 | +Neon can host the sync itself. The |
| 127 | +[Mirror WorkOS data into Postgres](https://neon.com/functions/templates/workos-sync) |
| 128 | +template in the Neon Functions registry packages the engine as one function |
| 129 | +on your branch. Schedule triggers call its `tick`, `backfill` and `reconcile` |
| 130 | +routes, a `/webhook` route lets a WorkOS webhook make it poll right away, and |
| 131 | +`GET /status` reports the mirror's health. The branch's pooled `DATABASE_URL` |
| 132 | +is injected; you supply three variables. The |
| 133 | +[template source](https://github.com/neondatabase/examples/tree/main/registry/workos-sync) |
| 134 | +is the full walkthrough; this is the shape of it. |
| 135 | + |
| 136 | +| Route | Called by | Purpose | |
| 137 | +| ----------------------- | --------------------------------- | --------------------------------------------------------------------------------- | |
| 138 | +| `POST /tick` | schedule trigger, every minute | Poll the Events API once | |
| 139 | +| `POST /backfill` | schedule trigger, every 5 minutes | One slice of the initial load; a no-op once every stream is complete | |
| 140 | +| `POST /reconcile` | schedule trigger, hourly | Continue the reconcile pass in progress and repair drift | |
| 141 | +| `POST /reconcile/fresh` | schedule trigger, weekly | Start a new reconcile pass | |
| 142 | +| `POST /webhook` | WorkOS | Verify the signature, then poll now | |
| 143 | +| `GET /status` | you, with a bearer token | Per-stream counts, poll age, lag, and any stream reconcile abandoned as oversized | |
| 144 | + |
| 145 | +### Scaffold and configure |
| 146 | + |
| 147 | +```sh |
| 148 | +npm install -g neon@latest # 8.11.0 or later |
| 149 | +neon login |
| 150 | +neon link |
| 151 | +neon functions new workos-sync --slug workossync |
| 152 | +``` |
| 153 | + |
| 154 | +The CLI writes `functions/workossync/index.ts`, declares the function in |
| 155 | +`neon.ts`, and adds the three WorkOS variables to `.env.example`. Put their |
| 156 | +values in `.env.local`, next to the Neon values `neon link` pulled: |
| 157 | + |
| 158 | +| Variable | Value | |
| 159 | +| -------------------------- | ------------------------------------------------------------------------------------------------------------------- | |
| 160 | +| `WORKOS_API_KEY` | A secret key (`sk_...`) for the environment to mirror, from **API Keys** in the WorkOS Dashboard | |
| 161 | +| `WORKOS_WEBHOOK_SECRET` | The signing secret of the webhook endpoint below; any placeholder until it exists | |
| 162 | +| `WORKOS_SYNC_STATUS_TOKEN` | A long random string, for example `openssl rand -hex 32`. It guards `GET /status`, because function URLs are public | |
| 163 | + |
| 164 | +### Schedule and deploy |
| 165 | + |
| 166 | +Triggers are not part of the template, so add them beside `functions` in |
| 167 | +`neon.ts`. Trigger names must be unique on the branch. |
| 168 | + |
| 169 | +```ts |
| 170 | +triggers: { |
| 171 | + "workos-tick": { |
| 172 | + type: "schedule", |
| 173 | + function: "workossync", |
| 174 | + cron: "* * * * *", |
| 175 | + functionPath: "/tick", |
| 176 | + }, |
| 177 | + "workos-backfill": { |
| 178 | + type: "schedule", |
| 179 | + function: "workossync", |
| 180 | + cron: "*/5 * * * *", |
| 181 | + functionPath: "/backfill", |
| 182 | + }, |
| 183 | + "workos-reconcile": { |
| 184 | + type: "schedule", |
| 185 | + function: "workossync", |
| 186 | + cron: "30 * * * *", |
| 187 | + functionPath: "/reconcile", |
| 188 | + }, |
| 189 | + "workos-reconcile-fresh": { |
| 190 | + type: "schedule", |
| 191 | + function: "workossync", |
| 192 | + cron: "0 3 * * 0", |
| 193 | + functionPath: "/reconcile/fresh", |
| 194 | + }, |
| 195 | +}, |
| 196 | +``` |
| 197 | + |
| 198 | +```sh |
| 199 | +neon deploy --env .env.local |
| 200 | +neon functions get workossync # prints the invocation_url |
| 201 | +``` |
| 202 | + |
| 203 | +The first `/backfill` delivery creates the schema and starts the load; |
| 204 | +`/tick` runs every minute from then on. Then, in the WorkOS Dashboard under |
| 205 | +**Webhooks**, add an endpoint at the function's `invocation_url` plus |
| 206 | +`/webhook`, select any events, copy its signing secret into `.env.local` as |
| 207 | +`WORKOS_WEBHOOK_SECRET`, and deploy again. The event selection does not |
| 208 | +matter: a delivery is only a nudge to poll, and the state still arrives |
| 209 | +through the events stream. |
| 210 | + |
| 211 | +```sh |
| 212 | +curl -H "Authorization: Bearer $WORKOS_SYNC_STATUS_TOKEN" <invocation_url>/status |
| 213 | +``` |
| 214 | + |
| 215 | +### What is different from running it yourself |
| 216 | + |
| 217 | +- **There is no `listen`.** The mirror is kept current by a tick per minute, |
| 218 | + or within seconds of a change when the webhook nudges it. Every route is a |
| 219 | + bounded, resumable slice of the kind [Serverless hosts](../docs/serverless.md) |
| 220 | + describes. `MAX_PAGES` and `MAX_EVENTS_PER_TICK` at the top of the module |
| 221 | + are the budgets, and each response reports `pages` so you can size them |
| 222 | + against the 15-minute limit. |
| 223 | +- **A reconcile cannot pause inside a stream.** A stream whose listing alone |
| 224 | + needs more than `MAX_PAGES` pages could never finish in a slot that size, |
| 225 | + so the engine abandons it on the page after the budget, rolls back its |
| 226 | + diff with nothing repaired, and reports it `oversized`, in that response |
| 227 | + and under `oversized` in `GET /status`. The hourly slices skip it from |
| 228 | + then on, and the weekly `/reconcile/fresh` tries it again, so left alone |
| 229 | + it costs one budget of pages a week and is never reconciled. Leave it out |
| 230 | + of both reconcile routes, and reconcile it on its own schedule from a host |
| 231 | + without a time limit: |
| 232 | + |
| 233 | + ```ts |
| 234 | + // In both reconcile routes of functions/workossync/index.ts. |
| 235 | + return Response.json( |
| 236 | + await getEngine().reconcile({ |
| 237 | + repair: true, |
| 238 | + resume, |
| 239 | + maxPages: MAX_PAGES, |
| 240 | + excludeStreams: ["directory_users"], |
| 241 | + }), |
| 242 | + ); |
| 243 | + ``` |
| 244 | + |
| 245 | + ```sh |
| 246 | + # Weekly, with the function's DATABASE_URL and WORKOS_ENVIRONMENT_NAME set to its ENVIRONMENT. |
| 247 | + npx workos-sync-postgres reconcile --repair --streams directory_users |
| 248 | + ``` |
| 249 | + |
| 250 | + `directory_users` is the usual case, at four requests per second per |
| 251 | + directory. The `pages` field in each response shows a stream approaching |
| 252 | + the budget before it crosses it, and |
| 253 | + [Serverless hosts](../docs/serverless.md#a-stream-larger-than-the-slot) |
| 254 | + has the same recipe for the CLI. |
| 255 | + |
| 256 | +- **The function is branch-scoped.** Each branch that deploys it polls WorkOS |
| 257 | + separately into its own database, all within the one key's rate limit. |
| 258 | + Deploy it on the branches that need a live mirror; a branch for a preview |
| 259 | + or a test can keep the copy it inherited instead, see [Branches](#branches). |
| 260 | +- **The compute stays awake here too.** The minute tick keeps the branch's |
| 261 | + compute from suspending, as described above. |
| 262 | +- **Logs are branch logs.** |
| 263 | + `neon logs query --branch <branch> --source function --since 1h` shows the |
| 264 | + function's output, including the details behind any `500`. |
| 265 | + |
| 266 | +## Branches |
| 267 | + |
| 268 | +A Neon branch is a copy-on-write clone of its parent, and that includes the |
| 269 | +mirror and its bookkeeping: the cursor, backfill checkpoints, the writer |
| 270 | +lease and the schema fingerprint. In practice: |
| 271 | + |
| 272 | +- **A child branch is a complete mirror as of the moment it was created**, |
| 273 | + with no backfill. For a preview or a test suite that only reads the mirror, |
| 274 | + that may be all it needs. |
| 275 | +- **A sync started on the child resumes from the copied cursor** and catches |
| 276 | + up from the branch point instead of re-listing everything, provided it |
| 277 | + mirrors the same WorkOS environment under the same `_env` label. The copied |
| 278 | + lease row can still be unexpired for up to a minute after branching: |
| 279 | + `listen` waits for it and `tick` reports it as held, as with any other |
| 280 | + holder. |
| 281 | +- **Resetting a branch from its parent** replaces the mirror and its cursor |
| 282 | + together, so the sync on that branch carries on from the parent's position |
| 283 | + with nothing to re-run. |
| 284 | +- **Two branches syncing the same environment are two syncs on one key.** |
| 285 | + Each polls and backfills on its own; see |
| 286 | + [rate limits](../docs/operations.md#rate-limits) for the budget they share. |
| 287 | + |
| 288 | +## Query it |
| 289 | + |
| 290 | +Everything in [Querying the mirror](../docs/queries.md) applies. Use the SQL editor |
| 291 | +in the Neon console, or open `psql` on the same URI: |
| 292 | + |
| 293 | +```sh |
| 294 | +neon connection-string --psql |
| 295 | +``` |
| 296 | + |
| 297 | +If the branch has the Data API enabled, it exposes only `public` by default. |
| 298 | +Leave `workos` out of its exposed schemas: the mirror holds raw event payloads |
| 299 | +and deleted rows, and access from browser clients needs its own grants and |
| 300 | +row-level security design. Neon Auth and the Data API keep their own schemas |
| 301 | +and roles, which the engine never touches. |
0 commit comments