Skip to content

Commit 5df9add

Browse files
authored
docs: Add Neon integration guide (#44)
1 parent 598f82a commit 5df9add

2 files changed

Lines changed: 307 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,12 @@ ships with the package and so always describes the version you installed. The
113113
`pageSize`, `logger`) are in the
114114
[`@workos/sync-engine` README](packages/sync-engine/README.md).
115115

116+
## Integration
117+
118+
Although any accessible database is supported, we also have guides for integrating with the following platforms:
119+
120+
- [Neon](integrations/neon.md)
121+
116122
## How it works
117123

118124
| | What it does | Why it exists |

‎integrations/neon.md‎

Lines changed: 301 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,301 @@
1+
# Neon
2+
3+
[Neon](https://neon.com) is a hosted Postgres with branching, autoscaling and
4+
scale to zero. The engine's `pg` destination reaches it over an ordinary
5+
PostgreSQL connection, so this guide is about the choices Neon puts in front
6+
of you, and about the one way of running the sync that only Neon offers: as a
7+
Function on the branch itself, next to the data.
8+
9+
| Run the sync | Suits |
10+
| ------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
11+
| From a host you operate, pointed at Neon | A server, container or cron you already run; `listen` as a long-lived process |
12+
| On Neon, as a Function from the registry template | No host to run: schedule triggers call `tick`, `backfill` and `reconcile` on the branch, next to the data |
13+
14+
Both start with a database. The first also needs a connection string you
15+
choose; a Function has one injected.
16+
17+
## Create the database
18+
19+
Create a Neon project for the mirror, or a branch of an existing one. The
20+
engine creates its own `workos` schema and touches nothing else, so a database
21+
your app already uses is fine; a branch is the cheap way to evaluate without
22+
involving it. See [Branches](#branches) for what a branch carries with it.
23+
24+
The project's default role owns the default database, which covers the
25+
`CREATE` and `TEMPORARY` privileges the engine needs. A role you create
26+
yourself needs those two grants on the database and must own the `workos`
27+
schema's objects, so run `init` as that role.
28+
29+
To run the sync as a [Neon Function](#run-the-sync-as-a-neon-function), put
30+
the project in a region where Functions are available. At the time of writing
31+
those are `aws-us-east-1`, `aws-us-east-2`, `aws-eu-central-1` and
32+
`aws-ap-southeast-1`; the
33+
[Functions overview](https://neon.com/docs/compute/functions/overview) has the
34+
current list.
35+
36+
## Choose a connection
37+
38+
Neon gives two connection strings for the same database. Both work with the
39+
engine.
40+
41+
| Connection | Host | Notes |
42+
| ---------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
43+
| Direct | `ep-<id>.<region>.aws.neon.tech` | A simple starting point. |
44+
| Pooled | `ep-<id>-pooler.<region>.aws.neon.tech` | PgBouncer in transaction mode. Has passed the engine's compatibility suite against a live Neon endpoint. What a Function receives as `DATABASE_URL`. |
45+
46+
Copy the URI from the **Connect** dialog in the console, or print it from a
47+
linked checkout:
48+
49+
```sh
50+
neon connection-string --ssl verify-full # direct
51+
neon connection-string --ssl verify-full --pooled # pooled
52+
```
53+
54+
The **Connection pooling** switch in that dialog only chooses which URI is
55+
shown; it is not a project setting. Keep every parameter the URI comes with,
56+
such as `channel_binding=require`, use the database password rather than a
57+
Neon API key, and percent-encode any password character that has meaning in a
58+
URL:
59+
60+
```
61+
postgresql://USER:PASSWORD@ep-<id>-pooler.<region>.aws.neon.tech/neondb?sslmode=verify-full&channel_binding=require
62+
```
63+
64+
Transaction pooling is enough because of how the engine uses the database:
65+
short transactions, transaction-scoped migration locks, row-based writer
66+
leases and temporary tables created `ON COMMIT DROP` inside a transaction.
67+
[Hosted Postgres](../docs/hosted-postgres.md#choose-a-connection) has the reasoning.
68+
69+
## TLS
70+
71+
Neon's server certificates chain to ISRG Root X1, the Let's Encrypt root,
72+
which Node's bundled root store already trusts. `sslmode=verify-full`
73+
therefore works with no certificate download, and `WORKOS_SYNC_PG_SSL_CA`
74+
stays unset. In embedded code there is likewise no `sslCa` to pass.
75+
76+
## Verify the endpoint
77+
78+
From a checkout of this repository, after `pnpm install` and `pnpm build`:
79+
80+
```sh
81+
TEST_DATABASE_URL='postgresql://USER:PASSWORD@ep-<id>-pooler.<region>.aws.neon.tech/neondb?sslmode=verify-full' \
82+
pnpm --filter @workos/sync-postgres test:database
83+
```
84+
85+
The suite creates a uniquely named schema per test and drops it afterward,
86+
exercising every stream definition, generated columns, live views,
87+
transactions, reconciliation, writer leases, checkpoints and tombstones
88+
against the real endpoint. WorkOS responses are local fixtures, so it makes
89+
no WorkOS calls and needs no key. Run it once for each URI you intend to use:
90+
it proves the credentials, network path, TLS and permissions that a local run
91+
cannot. [Hosted Postgres](../docs/hosted-postgres.md#verify-the-actual-endpoint) has
92+
more on what it covers.
93+
94+
## Run the sync from your own host
95+
96+
Point the command at Neon as you would at any Postgres:
97+
98+
```sh
99+
export DATABASE_URL='postgresql://USER:PASSWORD@ep-<id>-pooler.<region>.aws.neon.tech/neondb?sslmode=verify-full&channel_binding=require'
100+
export WORKOS_API_KEY=sk_...
101+
102+
npx workos-sync-postgres init
103+
npx workos-sync-postgres backfill
104+
npx workos-sync-postgres listen
105+
```
106+
107+
The [container image](../README.md#against-a-real-environment) and the
108+
[library](../docs/getting-started.md#embed-it) take the same URL. Two things about
109+
Neon's compute to plan for:
110+
111+
- **The compute stays awake while the sync runs.** Neon suspends an idle
112+
compute after five minutes by default and resumes it on the next
113+
connection. A running sync never leaves that gap: `listen` renews its
114+
writer lease every 20 seconds, and a `tick` every minute takes and releases
115+
the lease each time. Budget for the compute to run continuously at its
116+
minimum size for as long as the sync does. Where a few minutes of lag is
117+
acceptable, a `tick` on a longer interval than the suspend timeout lets the
118+
compute sleep between polls, at the cost of a cold start of a few hundred
119+
milliseconds on each one, which the sync simply waits out.
120+
- **An IP allow list must include the sync's host.** If the project restricts
121+
connections by address, add the address the sync connects from before the
122+
first `init`.
123+
124+
## Run the sync as a Neon Function
125+
126+
Neon can host the sync itself. The
127+
[Mirror WorkOS data into Postgres](https://neon.com/functions/templates/workos-sync)
128+
template in the Neon Functions registry packages the engine as one function
129+
on your branch. Schedule triggers call its `tick`, `backfill` and `reconcile`
130+
routes, a `/webhook` route lets a WorkOS webhook make it poll right away, and
131+
`GET /status` reports the mirror's health. The branch's pooled `DATABASE_URL`
132+
is injected; you supply three variables. The
133+
[template source](https://github.com/neondatabase/examples/tree/main/registry/workos-sync)
134+
is the full walkthrough; this is the shape of it.
135+
136+
| Route | Called by | Purpose |
137+
| ----------------------- | --------------------------------- | --------------------------------------------------------------------------------- |
138+
| `POST /tick` | schedule trigger, every minute | Poll the Events API once |
139+
| `POST /backfill` | schedule trigger, every 5 minutes | One slice of the initial load; a no-op once every stream is complete |
140+
| `POST /reconcile` | schedule trigger, hourly | Continue the reconcile pass in progress and repair drift |
141+
| `POST /reconcile/fresh` | schedule trigger, weekly | Start a new reconcile pass |
142+
| `POST /webhook` | WorkOS | Verify the signature, then poll now |
143+
| `GET /status` | you, with a bearer token | Per-stream counts, poll age, lag, and any stream reconcile abandoned as oversized |
144+
145+
### Scaffold and configure
146+
147+
```sh
148+
npm install -g neon@latest # 8.11.0 or later
149+
neon login
150+
neon link
151+
neon functions new workos-sync --slug workossync
152+
```
153+
154+
The CLI writes `functions/workossync/index.ts`, declares the function in
155+
`neon.ts`, and adds the three WorkOS variables to `.env.example`. Put their
156+
values in `.env.local`, next to the Neon values `neon link` pulled:
157+
158+
| Variable | Value |
159+
| -------------------------- | ------------------------------------------------------------------------------------------------------------------- |
160+
| `WORKOS_API_KEY` | A secret key (`sk_...`) for the environment to mirror, from **API Keys** in the WorkOS Dashboard |
161+
| `WORKOS_WEBHOOK_SECRET` | The signing secret of the webhook endpoint below; any placeholder until it exists |
162+
| `WORKOS_SYNC_STATUS_TOKEN` | A long random string, for example `openssl rand -hex 32`. It guards `GET /status`, because function URLs are public |
163+
164+
### Schedule and deploy
165+
166+
Triggers are not part of the template, so add them beside `functions` in
167+
`neon.ts`. Trigger names must be unique on the branch.
168+
169+
```ts
170+
triggers: {
171+
"workos-tick": {
172+
type: "schedule",
173+
function: "workossync",
174+
cron: "* * * * *",
175+
functionPath: "/tick",
176+
},
177+
"workos-backfill": {
178+
type: "schedule",
179+
function: "workossync",
180+
cron: "*/5 * * * *",
181+
functionPath: "/backfill",
182+
},
183+
"workos-reconcile": {
184+
type: "schedule",
185+
function: "workossync",
186+
cron: "30 * * * *",
187+
functionPath: "/reconcile",
188+
},
189+
"workos-reconcile-fresh": {
190+
type: "schedule",
191+
function: "workossync",
192+
cron: "0 3 * * 0",
193+
functionPath: "/reconcile/fresh",
194+
},
195+
},
196+
```
197+
198+
```sh
199+
neon deploy --env .env.local
200+
neon functions get workossync # prints the invocation_url
201+
```
202+
203+
The first `/backfill` delivery creates the schema and starts the load;
204+
`/tick` runs every minute from then on. Then, in the WorkOS Dashboard under
205+
**Webhooks**, add an endpoint at the function's `invocation_url` plus
206+
`/webhook`, select any events, copy its signing secret into `.env.local` as
207+
`WORKOS_WEBHOOK_SECRET`, and deploy again. The event selection does not
208+
matter: a delivery is only a nudge to poll, and the state still arrives
209+
through the events stream.
210+
211+
```sh
212+
curl -H "Authorization: Bearer $WORKOS_SYNC_STATUS_TOKEN" <invocation_url>/status
213+
```
214+
215+
### What is different from running it yourself
216+
217+
- **There is no `listen`.** The mirror is kept current by a tick per minute,
218+
or within seconds of a change when the webhook nudges it. Every route is a
219+
bounded, resumable slice of the kind [Serverless hosts](../docs/serverless.md)
220+
describes. `MAX_PAGES` and `MAX_EVENTS_PER_TICK` at the top of the module
221+
are the budgets, and each response reports `pages` so you can size them
222+
against the 15-minute limit.
223+
- **A reconcile cannot pause inside a stream.** A stream whose listing alone
224+
needs more than `MAX_PAGES` pages could never finish in a slot that size,
225+
so the engine abandons it on the page after the budget, rolls back its
226+
diff with nothing repaired, and reports it `oversized`, in that response
227+
and under `oversized` in `GET /status`. The hourly slices skip it from
228+
then on, and the weekly `/reconcile/fresh` tries it again, so left alone
229+
it costs one budget of pages a week and is never reconciled. Leave it out
230+
of both reconcile routes, and reconcile it on its own schedule from a host
231+
without a time limit:
232+
233+
```ts
234+
// In both reconcile routes of functions/workossync/index.ts.
235+
return Response.json(
236+
await getEngine().reconcile({
237+
repair: true,
238+
resume,
239+
maxPages: MAX_PAGES,
240+
excludeStreams: ["directory_users"],
241+
}),
242+
);
243+
```
244+
245+
```sh
246+
# Weekly, with the function's DATABASE_URL and WORKOS_ENVIRONMENT_NAME set to its ENVIRONMENT.
247+
npx workos-sync-postgres reconcile --repair --streams directory_users
248+
```
249+
250+
`directory_users` is the usual case, at four requests per second per
251+
directory. The `pages` field in each response shows a stream approaching
252+
the budget before it crosses it, and
253+
[Serverless hosts](../docs/serverless.md#a-stream-larger-than-the-slot)
254+
has the same recipe for the CLI.
255+
256+
- **The function is branch-scoped.** Each branch that deploys it polls WorkOS
257+
separately into its own database, all within the one key's rate limit.
258+
Deploy it on the branches that need a live mirror; a branch for a preview
259+
or a test can keep the copy it inherited instead, see [Branches](#branches).
260+
- **The compute stays awake here too.** The minute tick keeps the branch's
261+
compute from suspending, as described above.
262+
- **Logs are branch logs.**
263+
`neon logs query --branch <branch> --source function --since 1h` shows the
264+
function's output, including the details behind any `500`.
265+
266+
## Branches
267+
268+
A Neon branch is a copy-on-write clone of its parent, and that includes the
269+
mirror and its bookkeeping: the cursor, backfill checkpoints, the writer
270+
lease and the schema fingerprint. In practice:
271+
272+
- **A child branch is a complete mirror as of the moment it was created**,
273+
with no backfill. For a preview or a test suite that only reads the mirror,
274+
that may be all it needs.
275+
- **A sync started on the child resumes from the copied cursor** and catches
276+
up from the branch point instead of re-listing everything, provided it
277+
mirrors the same WorkOS environment under the same `_env` label. The copied
278+
lease row can still be unexpired for up to a minute after branching:
279+
`listen` waits for it and `tick` reports it as held, as with any other
280+
holder.
281+
- **Resetting a branch from its parent** replaces the mirror and its cursor
282+
together, so the sync on that branch carries on from the parent's position
283+
with nothing to re-run.
284+
- **Two branches syncing the same environment are two syncs on one key.**
285+
Each polls and backfills on its own; see
286+
[rate limits](../docs/operations.md#rate-limits) for the budget they share.
287+
288+
## Query it
289+
290+
Everything in [Querying the mirror](../docs/queries.md) applies. Use the SQL editor
291+
in the Neon console, or open `psql` on the same URI:
292+
293+
```sh
294+
neon connection-string --psql
295+
```
296+
297+
If the branch has the Data API enabled, it exposes only `public` by default.
298+
Leave `workos` out of its exposed schemas: the mirror holds raw event payloads
299+
and deleted rows, and access from browser clients needs its own grants and
300+
row-level security design. Neon Auth and the Data API keep their own schemas
301+
and roles, which the engine never touches.

0 commit comments

Comments
 (0)