Skip to content

Commit b04dd8f

Browse files
committed
cleanup
1 parent 3959847 commit b04dd8f

2 files changed

Lines changed: 63 additions & 76 deletions

File tree

‎packages/sync-engine/README.md‎

Lines changed: 27 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,6 @@ choosing. It knows WorkOS — the object catalog, event routing, the REST
55
backfill, the Events API tail, reconciliation — and nothing about any
66
particular database. Writes go through a small `Destination` interface.
77

8-
> **Prototype.** A working proof of concept — not yet a supported product.
9-
> The API may change between minor versions.
10-
118
**You probably want a destination package instead.** Each one depends on this
129
package and re-exports it, so it is the only thing to install:
1310

@@ -59,20 +56,20 @@ The lifecycle is deliberately overlapping: the cursor is captured _before_
5956
backfill starts, so anything that changes mid-backfill is replayed afterwards
6057
and the staleness gate decides which version wins.
6158

62-
| Option | Default | |
63-
| ------------------ | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
64-
| `destination` | — | A `Destination` implementation. Required. |
65-
| `environments` | — | `{ name, apiKey, apiUrl? }[]`. `name` is written to every row and must never change. |
66-
| `streams` | `defaultStreams()` | Stream names to sync: every stream with events. Poll-only streams are opt-in. `events` is always included. |
67-
| `extraColumns` | — | Extra typed columns to project out of `_raw`, keyed by stream. Add-only. |
68-
| `eventTypes` | derived from `streams` | The `GET /events` filter. Widen it to archive event families with no mirror table. |
69-
| `hydrate` | users and organizations `on_create` | Re-fetch objects on event receipt: `off`, `on_create`, `always`, or a per-stream map. Only streams with a get-by-id endpoint can be hydrated. |
70-
| `pollIntervalMs` | `5000` | Idle wait between polls. |
71-
| `lagWindowMs` | `5000` | Never advance the cursor past events younger than this. |
72-
| `pageSize` | `100` | Page size for list and event calls. |
73-
| `maxEventsPerTick` | one page | Events to apply per tick, across as many pages as needed. Raise it for destinations where every statement is expensive. |
59+
| Option | Default | |
60+
| ------------------ | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
61+
| `destination` | — | A `Destination` implementation. Required. |
62+
| `environments` | — | `{ name, apiKey, apiUrl? }[]`. `name` is written to every row and must never change. |
63+
| `streams` | `defaultStreams()` | Stream names to sync: every stream with events. Poll-only streams are opt-in. `events` is always included. |
64+
| `extraColumns` | — | Extra typed columns to project out of `_raw`, keyed by stream. Add-only. |
65+
| `eventTypes` | derived from `streams` | The `GET /events` filter. Widen it to archive event families with no mirror table. |
66+
| `hydrate` | users and organizations `on_create` | Re-fetch objects on event receipt: `off`, `on_create`, `always`, or a per-stream map. Only streams with a get-by-id endpoint can be hydrated. |
67+
| `pollIntervalMs` | `5000` | Idle wait between polls. |
68+
| `lagWindowMs` | `5000` | Never advance the cursor past events younger than this. |
69+
| `pageSize` | `100` | Page size for list and event calls. |
70+
| `maxEventsPerTick` | one page | Events to apply per tick, across as many pages as needed. Raise it for destinations where every statement is expensive. |
7471
| `leaseTtlMs` | `60000` | How long the events-tail lease outlives a crashed process; at least 1. Renewed every third of this while polling; a clean stop releases it immediately. |
75-
| `logger` | JSON to stdout | `{ debug, info, warn, error }`. |
72+
| `logger` | JSON to stdout | `{ debug, info, warn, error }`. |
7673

7774
Methods: `init()`, `backfill({ resume })`, `tick()`, `listen()`, `stop()`,
7875
`reconcile({ repair, streams })`, `refresh()`, `status()`, `health()`,
@@ -188,13 +185,22 @@ class MyDestination implements Destination {
188185

189186
getState<T>(env: string, stream: string): Promise<T | null>;
190187
setState(env: string, stream: string, state: unknown): Promise<void>;
191-
setStateIfHeld(env: string, stream: string, state: unknown, holder: string, expected: unknown): Promise<boolean>;
188+
setStateIfHeld(
189+
env: string,
190+
stream: string,
191+
state: unknown,
192+
holder: string,
193+
expected: unknown,
194+
): Promise<boolean>;
192195

193-
acquireLease(env: string, holder: string, ttlMs: number): Promise<LeaseResult>;
196+
acquireLease(
197+
env: string,
198+
holder: string,
199+
ttlMs: number,
200+
): Promise<LeaseResult>;
194201
releaseLease(env: string, holder: string): Promise<void>;
195202
getLease(env: string): Promise<Lease | null>;
196203

197-
198204
reject(
199205
env: string,
200206
rows: { event: WorkOSEvent; error: string }[],
@@ -206,8 +212,7 @@ class MyDestination implements Destination {
206212
}
207213
```
208214

209-
The engine relies on seven properties of your implementation. Each one exists
210-
because of a documented failure in a comparable sync engine.
215+
The engine relies on seven properties of your implementation.
211216

212217
1. **Staleness gate, `>=` not `>`.** `upsert` rejects a row whose `updatedAt`
213218
is older than the stored one and accepts an equal one. With a strict `<`,
@@ -241,7 +246,7 @@ because of a documented failure in a comparable sync engine.
241246
process that stalled past its lease, or whose position a backfill
242247
re-captured meanwhile, is refused by the database rather than by a check
243248
made a moment earlier. In SQL it is an `INSERT ... SELECT ... WHERE EXISTS
244-
(lease) ... ON CONFLICT DO UPDATE ... WHERE`; also in `db/lease.ts`.
249+
(lease) ... ON CONFLICT DO UPDATE ... WHERE`; also in `db/lease.ts`.
245250

246251
`WriteStats` (`created` / `updated` / `skipped` / `deleted`) is a health metric
247252
the engine logs but never branches on; a database that cannot tell an insert
@@ -276,16 +281,6 @@ and the engine will retry instead of ever quarantining on them. Without it,
276281
every error is presumed deterministic and only the two-sightings rule stands
277282
between a flaky connection and a quarantine.
278283

279-
The reference implementation is
280-
[`PostgresDestination`](https://github.com/workos/workos-pg-sync/blob/main/packages/sync-postgres/src/destination.ts)
281-
in `@workos/sync-postgres` (about 200 lines).
282-
283-
## Documentation
284-
285-
- [How it works](https://github.com/workos/workos-pg-sync/blob/main/docs/how-it-works.md)
286-
- [Consistency](https://github.com/workos/workos-pg-sync/blob/main/docs/consistency.md)
287-
- [Operations](https://github.com/workos/workos-pg-sync/blob/main/docs/operations.md)
288-
289284
## License
290285

291286
MIT

0 commit comments

Comments
 (0)