@@ -8,12 +8,12 @@ long-lived process.
88
99## What fits in a slot
1010
11- | Command | Bounded by | Continues with |
12- | --------------------------- | -------------------------------------- | ----------------------------------------- |
13- | ` tick ` | ` maxEventsPerTick ` events, one poll | the cursor, automatically |
14- | ` backfill --max-pages <n> ` | ` n ` pages across every stream | ` --resume ` |
15- | ` refresh --max-pages <n> ` | ` n ` pages across the top-level streams | ` --resume ` ; shares backfill's checkpoints |
16- | ` reconcile --max-pages <n> ` | ` n ` pages, stopping between streams | ` --resume ` |
11+ | Command | Bounded by | Continues with |
12+ | --------------------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------- |
13+ | ` tick ` | ` maxEventsPerTick ` events, one poll | the cursor, automatically |
14+ | ` backfill --max-pages <n> ` | ` n ` pages across every stream | ` --resume ` |
15+ | ` refresh --max-pages <n> ` | ` n ` pages across the top-level streams | ` --resume ` ; shares backfill's checkpoints |
16+ | ` reconcile --max-pages <n> ` | ` n ` pages, stopping between streams; a stream needing more than ` n ` alone is abandoned as oversized | ` --resume ` |
1717
1818` tick ` takes the writer lease and renews it while it runs. A second invocation
1919that overlaps it exits 3, or throws ` LeaseHeldError ` in the library, and should
@@ -46,7 +46,9 @@ resumed ones stay scheduled and cost a few state reads.
4646- ** ` reconcile ` ** starts fresh once a week and is finished by the hourly
4747 resumed slices, which do nothing once every stream is complete. Give both
4848 the same flags; ` --repair ` is applied per stream inside each slice. Without
49- ` --repair ` , drift exits 2, which is the signal to alert on.
49+ ` --repair ` , drift exits 2, which is the signal to alert on. A stream too
50+ large for the slot needs a schedule of its own; see
51+ [ A stream larger than the slot] ( #a-stream-larger-than-the-slot ) .
5052- ** ` refresh ` ** fits the same shape as ` reconcile ` when you use it.
5153
5254Size the budget from the output. A slice's run time is its ` pages ` times the
@@ -140,19 +142,37 @@ changes nothing. Verify on the raw bytes, before any JSON parsing.
140142There is no ` listen ` , so there is no health server. ` status ` reads the
141143database and works from any invocation: ` pollAgeSeconds ` is the number to
142144alert on for a stalled sync, ` rejected ` for events the tail has quarantined,
143- and the per-stream rows give counts, tombstones, staleness and lag. Serve it
144- from a probe route, or run it on a schedule and alert on the result.
145+ ` oversized ` for streams a reconcile abandoned as too large for the slot, and
146+ the per-stream rows give counts, tombstones, staleness and lag. Serve it from
147+ a probe route, or run it on a schedule and alert on the result.
145148
146149## A stream larger than the slot
147150
148151` reconcile ` cannot pause inside a stream, because a diff is taken against the
149- stream's whole listing. A stream whose listing alone outlasts the slot runs to
150- its last page regardless of the budget, and the slice overshoots. The ` pages `
151- field shows it coming: when one stream's pages times the time per page
152- approaches the slot, run that stream's reconcile somewhere without a limit,
153- ` engine.reconcile({ streams: [...] }) ` from the library or a one-off
154- ` reconcile ` from a container, and let the scheduled slices cover the rest.
155- ` backfill ` and ` refresh ` have no such floor; they checkpoint after every page.
152+ stream's whole listing. A stream whose listing alone needs more than
153+ ` --max-pages ` pages could never finish in a slot that size, so the slice
154+ abandons it on the page after the budget, rolls back what it staged with
155+ nothing repaired, and reports it ` oversized ` . Later slices with ` --resume `
156+ skip it rather than spend the budget on it again, ` status ` lists it under
157+ ` oversized ` , and it stays unreconciled until a run with a larger budget, or
158+ none, diffs it. A fresh slice tries it again, so left in the schedule it
159+ costs one budget's worth of pages a week and is never reconciled.
160+
161+ Give it a schedule of its own somewhere without a limit, and leave it out of
162+ both scheduled slices with ` --exclude-streams ` :
163+
164+ ``` cron
165+ 30 * * * * workos-sync-postgres reconcile --resume --repair --max-pages 50 --exclude-streams directory_users
166+ 0 3 * * 0 workos-sync-postgres reconcile --repair --max-pages 50 --exclude-streams directory_users
167+ 0 4 * * 0 workos-sync-postgres reconcile --repair --streams directory_users # a host without a limit
168+ ```
169+
170+ From the library, pass ` excludeStreams ` to the bounded calls and
171+ ` engine.reconcile({ repair: true, streams: ['directory_users'] }) ` to the
172+ unbounded one. ` directory_users ` is the usual case, at four requests per
173+ second per directory; the ` pages ` field shows any stream approaching the
174+ budget before it crosses it. ` backfill ` and ` refresh ` have no such floor;
175+ they checkpoint after every page.
156176
157177## Connections
158178
0 commit comments