Skip to content

Commit 598f82a

Browse files
authored
feat: Abandon oversized reconcile streams, add excludeStreams (#45)
1 parent 01fa153 commit 598f82a

12 files changed

Lines changed: 520 additions & 81 deletions

File tree

‎docs/operations.md‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ npx workos-sync-postgres status
3030
"expiresAt": "2026-10-22T18:05:11.902Z"
3131
},
3232
"rejected": 0,
33+
"oversized": [],
3334
"streams": [
3435
{
3536
"stream": "users",
@@ -59,6 +60,10 @@ Two of these numbers look similar but mean different things:
5960
deterministically. The tail kept going, so poll age looks healthy, but the
6061
objects those events describe are missing or stale until you replay them.
6162
**Alert on this too.** See [Rejected events](#rejected-events).
63+
- **`oversized`** names the streams the last `reconcile` abandoned because
64+
their listing alone needs more pages than its `--max-pages`. They stay
65+
unreconciled until a run with a larger budget, or none, diffs them; see
66+
[Backfill without a long-lived process](#backfill-without-a-long-lived-process).
6267

6368
`lease` names the process currently tailing this environment, as
6469
`hostname:pid:suffix`, and when its claim lapses if it stops renewing. It is
@@ -484,11 +489,23 @@ A diff is taken against a stream's whole listing, so a listing cannot stop
484489
partway: the objects on the pages a budget cut off would all read as missed
485490
deletes, and `--repair` would tombstone them. The budget decides whether the
486491
next stream starts, not where a listing stops. A stream that starts runs to
487-
its last page, so a slice overshoots by at most one stream's listing; a stream
488-
whose own listing does not fit the host's limit has to run somewhere without
489-
one. Each result carries `pages` and `complete`, a stream that fails is
490-
reported with `complete: false` rather than left out, and `--resume` skips
491-
the streams already diffed.
492+
its last page, so a slice overshoots by at most `n` pages: a stream whose own
493+
listing passes `n` could never finish in a slot that size, so the slice
494+
abandons it on the page after the budget, rolls back its diff with nothing
495+
repaired, and reports it `oversized`. Later slices with `--resume` skip it,
496+
and `status` lists it under `oversized`, until a run with a larger budget or
497+
none diffs it; a run without `--resume` tries it again. Each result carries
498+
`pages` and `complete`, a stream that fails is reported with `complete: false`
499+
rather than left out, and `--resume` skips the streams already diffed.
500+
501+
Give an oversized stream a run of its own somewhere without a limit, and leave
502+
it out of the bounded slices with `--exclude-streams`, so neither the resumed
503+
slices nor a fresh one spend their budget on it:
504+
505+
```bash
506+
npx workos-sync-postgres reconcile --resume --max-pages 50 --exclude-streams directory_users
507+
npx workos-sync-postgres reconcile --repair --streams directory_users # from a host without a limit
508+
```
492509

493510
`refresh` takes the same two flags and shares backfill's checkpoints. For the
494511
schedules that tie these commands together on a host with a time limit, see

‎docs/serverless.md‎

Lines changed: 36 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,12 @@ long-lived process.
88

99
## What fits in a slot
1010

11-
| Command | Bounded by | Continues with |
12-
| --------------------------- | -------------------------------------- | ----------------------------------------- |
13-
| `tick` | `maxEventsPerTick` events, one poll | the cursor, automatically |
14-
| `backfill --max-pages <n>` | `n` pages across every stream | `--resume` |
15-
| `refresh --max-pages <n>` | `n` pages across the top-level streams | `--resume`; shares backfill's checkpoints |
16-
| `reconcile --max-pages <n>` | `n` pages, stopping between streams | `--resume` |
11+
| Command | Bounded by | Continues with |
12+
| --------------------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------- |
13+
| `tick` | `maxEventsPerTick` events, one poll | the cursor, automatically |
14+
| `backfill --max-pages <n>` | `n` pages across every stream | `--resume` |
15+
| `refresh --max-pages <n>` | `n` pages across the top-level streams | `--resume`; shares backfill's checkpoints |
16+
| `reconcile --max-pages <n>` | `n` pages, stopping between streams; a stream needing more than `n` alone is abandoned as oversized | `--resume` |
1717

1818
`tick` takes the writer lease and renews it while it runs. A second invocation
1919
that overlaps it exits 3, or throws `LeaseHeldError` in the library, and should
@@ -46,7 +46,9 @@ resumed ones stay scheduled and cost a few state reads.
4646
- **`reconcile`** starts fresh once a week and is finished by the hourly
4747
resumed slices, which do nothing once every stream is complete. Give both
4848
the same flags; `--repair` is applied per stream inside each slice. Without
49-
`--repair`, drift exits 2, which is the signal to alert on.
49+
`--repair`, drift exits 2, which is the signal to alert on. A stream too
50+
large for the slot needs a schedule of its own; see
51+
[A stream larger than the slot](#a-stream-larger-than-the-slot).
5052
- **`refresh`** fits the same shape as `reconcile` when you use it.
5153

5254
Size the budget from the output. A slice's run time is its `pages` times the
@@ -140,19 +142,37 @@ changes nothing. Verify on the raw bytes, before any JSON parsing.
140142
There is no `listen`, so there is no health server. `status` reads the
141143
database and works from any invocation: `pollAgeSeconds` is the number to
142144
alert on for a stalled sync, `rejected` for events the tail has quarantined,
143-
and the per-stream rows give counts, tombstones, staleness and lag. Serve it
144-
from a probe route, or run it on a schedule and alert on the result.
145+
`oversized` for streams a reconcile abandoned as too large for the slot, and
146+
the per-stream rows give counts, tombstones, staleness and lag. Serve it from
147+
a probe route, or run it on a schedule and alert on the result.
145148

146149
## A stream larger than the slot
147150

148151
`reconcile` cannot pause inside a stream, because a diff is taken against the
149-
stream's whole listing. A stream whose listing alone outlasts the slot runs to
150-
its last page regardless of the budget, and the slice overshoots. The `pages`
151-
field shows it coming: when one stream's pages times the time per page
152-
approaches the slot, run that stream's reconcile somewhere without a limit,
153-
`engine.reconcile({ streams: [...] })` from the library or a one-off
154-
`reconcile` from a container, and let the scheduled slices cover the rest.
155-
`backfill` and `refresh` have no such floor; they checkpoint after every page.
152+
stream's whole listing. A stream whose listing alone needs more than
153+
`--max-pages` pages could never finish in a slot that size, so the slice
154+
abandons it on the page after the budget, rolls back what it staged with
155+
nothing repaired, and reports it `oversized`. Later slices with `--resume`
156+
skip it rather than spend the budget on it again, `status` lists it under
157+
`oversized`, and it stays unreconciled until a run with a larger budget, or
158+
none, diffs it. A fresh slice tries it again, so left in the schedule it
159+
costs one budget's worth of pages a week and is never reconciled.
160+
161+
Give it a schedule of its own somewhere without a limit, and leave it out of
162+
both scheduled slices with `--exclude-streams`:
163+
164+
```cron
165+
30 * * * * workos-sync-postgres reconcile --resume --repair --max-pages 50 --exclude-streams directory_users
166+
0 3 * * 0 workos-sync-postgres reconcile --repair --max-pages 50 --exclude-streams directory_users
167+
0 4 * * 0 workos-sync-postgres reconcile --repair --streams directory_users # a host without a limit
168+
```
169+
170+
From the library, pass `excludeStreams` to the bounded calls and
171+
`engine.reconcile({ repair: true, streams: ['directory_users'] })` to the
172+
unbounded one. `directory_users` is the usual case, at four requests per
173+
second per directory; the `pages` field shows any stream approaching the
174+
budget before it crosses it. `backfill` and `refresh` have no such floor;
175+
they checkpoint after every page.
156176

157177
## Connections
158178

‎packages/sync-engine/README.md‎

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ and the staleness gate decides which version wins.
7676
| `logger` | JSON to stdout | `{ debug, info, warn, error }`. |
7777

7878
Methods: `init()`, `backfill({ resume, maxPages })`, `tick()`, `listen()`,
79-
`stop()`, `reconcile({ repair, streams, resume, maxPages })`,
79+
`stop()`, `reconcile({ repair, streams, excludeStreams, resume, maxPages })`,
8080
`refresh({ resume, maxPages })`, `status()`,
8181
`health()`, `redact(userId)`, `rejected({ ids })`, `replayRejected({ ids })`,
8282
`processWebhook()`, `close()`. Every method except `health()` applies the
@@ -100,11 +100,17 @@ cannot stop partway: every object on the pages a budget cut off would read as
100100
a missed delete, and `repair` would tombstone objects that exist upstream. The
101101
budget therefore decides whether the next stream starts, not where a listing
102102
stops; a stream that starts runs to its last page, and a call overshoots by at
103-
most one stream's listing. Each `DiffResult` carries `pages` and `complete`,
104-
a stream that fails is reported with `complete: false` rather than left out,
105-
and `resume` skips the streams already diffed. A stream whose own listing does
106-
not fit the host's limit has to run, selected with `streams`, somewhere
107-
without one.
103+
most `maxPages`. That bound holds because a stream whose own listing passes
104+
`maxPages` could never finish inside a slot that size: it is abandoned on the
105+
page after the budget, its diff is rolled back with nothing repaired, and it
106+
is recorded as oversized. A call with `resume` skips it, and `status()` lists
107+
it under `oversized`, until a call with a larger `maxPages`, or none, diffs
108+
it; a call without `resume` tries it again. Each `DiffResult` carries `pages`,
109+
`complete` and, for such a stream, `oversized: true`; a stream that fails is
110+
reported with `complete: false` rather than left out, and `resume` skips the
111+
streams already diffed. Give an oversized stream a run of its own somewhere
112+
without a limit, selected with `streams`, and leave it out of the bounded
113+
calls with `excludeStreams`.
108114

109115
`refresh()` takes the same `maxPages` and `resume` and shares backfill's
110116
checkpoints, since re-listing a top-level stream is the same work whichever
@@ -135,8 +141,9 @@ a tail.
135141
the last successful poll; alert on this), `cursorAgeDays` (age of the newest
136142
event applied; distance from the 90-day retention cliff, not liveness),
137143
`lease` (who is tailing it now, or `null`), `rejected` (events quarantined by
138-
the tail; alert on this too), and per-stream row counts, tombstones,
139-
staleness and lag.
144+
the tail; alert on this too), `oversized` (streams the last `reconcile()`
145+
abandoned because their listing alone passes its `maxPages`), and per-stream
146+
row counts, tombstones, staleness and lag.
140147

141148
`health()` is the same liveness picture without the database: a synchronous
142149
snapshot of in-memory state (`starting` / `listening` / `stopped`, any fatal

0 commit comments

Comments
 (0)