Skip to content

Commit 55176cb

Browse files
authored
feat: Serve /healthz and Prometheus metrics from listen (#8)
1 parent c855c99 commit 55176cb

15 files changed

Lines changed: 1210 additions & 12 deletions

File tree

‎.github/workflows/docker.yml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,3 +109,19 @@ jobs:
109109
run: |
110110
docker run --rm "$IMAGE" | grep -q '^Usage'
111111
docker run --rm "$IMAGE" streams | jq -e 'any(.[]; .stream == "users")'
112+
113+
# The HEALTHCHECK has to find the listener on the port `listen` resolved,
114+
# whether that came from the variable or from a `--health-port` flag on
115+
# the entrypoint's command line. A stand-in server plays the listener,
116+
# so no database or credentials are involved.
117+
- name: Health check finds the listener's port
118+
env:
119+
IMAGE: sync-${{ matrix.destination }}:ci
120+
SERVER: require('node:http').createServer((_, res) => res.end()).listen(9000)
121+
run: |
122+
flag=$(docker run -d --entrypoint node "$IMAGE" -e "$SERVER" -- listen --health-port 9000)
123+
var=$(docker run -d --entrypoint node -e WORKOS_SYNC_HEALTH_PORT=9000 "$IMAGE" -e "$SERVER" -- listen)
124+
sleep 2
125+
docker exec "$flag" node healthcheck.mjs
126+
docker exec "$var" node healthcheck.mjs
127+
docker rm -f "$flag" "$var" >/dev/null

‎Dockerfile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,21 @@ RUN addgroup --system --gid 1001 nodejs && adduser --system --uid 1001 --ingroup
7676
COPY --from=build /deploy/package.json ./
7777
COPY --from=build /deploy/node_modules ./node_modules
7878
COPY --from=build /deploy/dist ./dist
79+
COPY docker/healthcheck.mjs ./healthcheck.mjs
7980
USER workos
8081
ENV NODE_ENV=production
82+
# `listen` serves GET /healthz and /metrics on this port, from in-memory state,
83+
# so an orchestrator can tell a stalled tail from a live one without a
84+
# database connection. Only `listen` opens the port; one-shot commands (init,
85+
# backfill, tick, ...) ignore it and exit long before the check below would
86+
# run, so the HEALTHCHECK is only meaningful for a long-running `listen`.
87+
ENV WORKOS_SYNC_HEALTH_PORT=8080
88+
EXPOSE 8080
89+
# The check resolves the port the way `listen` does, so `listen --health-port
90+
# <n>` keeps the container healthy just as `-e WORKOS_SYNC_HEALTH_PORT=<n>`
91+
# does. healthcheck.mjs explains how it finds the flag.
92+
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
93+
CMD ["node", "healthcheck.mjs"]
8194
# --use-env-proxy makes fetch honor HTTP_PROXY, HTTPS_PROXY and NO_PROXY, which
8295
# is how the sync reaches the WorkOS API from inside a locked-down network.
8396
ENTRYPOINT ["node", "--use-env-proxy", "dist/cli.js"]

‎README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,9 @@ the same setting, the flag wins.
100100
| `WORKOS_SYNC_POLL_MS` | `5000` | Idle wait between polls, in ms, when the last poll returned nothing. |
101101
| `WORKOS_SYNC_LAG_MS` | `5000` | Hold back events younger than this, in ms, before advancing the cursor. |
102102
| `WORKOS_SYNC_MAX_EVENTS_PER_TICK` | one page (100) | Events to apply per poll, fetched across as many pages as needed. |
103+
| `WORKOS_SYNC_HEALTH_PORT` | off (`8080` in the container image) | `listen` only: serve `GET /healthz` and `GET /metrics` on this port, from in-memory state, with no database query. |
104+
| `WORKOS_SYNC_HEALTH_HOST` | `0.0.0.0` | Interface the health server binds. |
105+
| `WORKOS_SYNC_HEALTH_MAX_POLL_AGE_MS` | `60000` | `/healthz` returns 503 once any environment's last successful poll is older than this. |
103106
| `LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error` or `silent`. Logs are JSON lines on stdout. |
104107

105108
### Commands
@@ -108,7 +111,7 @@ the same setting, the flag wins.
108111
| ----------- | ------------- | ------------------------------------------------------------------------------------------------------------ |
109112
| `init` | | Create the schema and bookkeeping tables. Idempotent; safe on every deploy. |
110113
| `backfill` | `--resume` | Load current state from the REST list endpoints. `--resume` continues from the last checkpoint. |
111-
| `listen` | | Poll the Events API continuously. Finishes the current batch on `SIGINT` or `SIGTERM`. |
114+
| `listen` | `--health-port <n>` | Poll the Events API continuously. Finishes the current batch on `SIGINT` or `SIGTERM`. With a port, serves `/healthz` and `/metrics`. |
112115
| `tick` | | One poll, then exit. For a cron job instead of a long-running process. |
113116
| `reconcile` | `--repair` | Diff ids against the API. Exits 2 on drift; `--repair` tombstones missed deletes and inserts missed creates. |
114117
| `refresh` | | Re-list top-level streams to refresh fields that event payloads omit. |

‎docker/healthcheck.mjs‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
/**
2+
* Docker HEALTHCHECK for a long-running `listen`.
3+
*
4+
* The image has no curl or wget, so Node's global fetch does the probe. The
5+
* check runs in its own process and cannot ask the listener which port it is
6+
* serving, so it resolves the port the way `listen` does: a `--health-port
7+
* <n>` flag wins over WORKOS_SYNC_HEALTH_PORT. The flag is read from PID 1's
8+
* command line, which is the entrypoint itself under a plain `docker run`,
9+
* or an init (`--init`, tini) whose argv carries the same command. Anything
10+
* unreadable falls back to the variable, which the image sets to 8080.
11+
*
12+
* The loopback address is used rather than the bind host so the check works
13+
* whatever WORKOS_SYNC_HEALTH_HOST is set to, as long as it includes this
14+
* interface.
15+
*/
16+
import { readFileSync } from 'node:fs'
17+
18+
function effectivePort() {
19+
try {
20+
// /proc/<pid>/cmdline is argv joined by NUL bytes. The flag takes the
21+
// next token as its value unless that token is another flag, which is
22+
// the rule the CLI's own parser applies.
23+
const argv = readFileSync('/proc/1/cmdline', 'utf8').split('\0')
24+
const i = argv.indexOf('--health-port')
25+
const value = argv[i + 1]
26+
if (i !== -1 && value && !value.startsWith('--')) return value
27+
} catch {
28+
// Not Linux, or /proc is hidden: the variable is all there is.
29+
}
30+
return process.env.WORKOS_SYNC_HEALTH_PORT
31+
}
32+
33+
fetch(`http://127.0.0.1:${effectivePort()}/healthz`).then(
34+
(res) => process.exit(res.ok ? 0 : 1),
35+
() => process.exit(1),
36+
)

‎docs/operations.md‎

Lines changed: 105 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,43 @@ Two of these numbers look similar but mean different things:
5151
It is `null` until the first event is applied. The engine warns at 75 days
5252
and logs an error past 90. Alert well before that.
5353

54+
`status` opens a database connection and queries every stream, which is the
55+
wrong shape for something that runs every few seconds. The same poll age is
56+
also served by `listen` itself, from memory, when `WORKOS_SYNC_HEALTH_PORT`
57+
is set (the container image sets it to `8080`):
58+
59+
```bash
60+
curl -s localhost:8080/healthz
61+
```
62+
63+
```json
64+
{
65+
"state": "listening",
66+
"fatal": null,
67+
"startedAt": "2026-10-22T17:58:02.114Z",
68+
"healthy": true,
69+
"environments": {
70+
"production": {
71+
"lastPolledAt": "2026-10-22T18:04:11.902Z",
72+
"pollAgeSeconds": 3.1,
73+
"lastEventAt": "2026-10-22T18:04:09.471Z",
74+
"cursor": "event_01HXXXXXXXXXXXXXXXXXXXXXX",
75+
"lastError": null,
76+
"counters": { "ticks": 71, "events": 42, "created": 3, "updated": 38, "skipped": 1, "deleted": 0, "errors": 0 }
77+
}
78+
}
79+
}
80+
```
81+
82+
The status code is the verdict: `200` while `state` is `starting` (the
83+
process is up, the first poll has not finished) and while every environment
84+
has polled successfully within `WORKOS_SYNC_HEALTH_MAX_POLL_AGE_MS` (default
85+
60 seconds); `503` once any environment's poll is older than that, once the
86+
engine has stopped on a fatal error, or once `listen` has returned. No
87+
request to it touches the database, so it keeps answering when the database
88+
is the thing that is down, and the answer is then a `503` from a climbing
89+
poll age rather than a timeout.
90+
5491
## What to monitor
5592

5693
Every table carries `_updated_at` (source time) and `_synced_at` (write time),
@@ -103,7 +140,7 @@ SELECT DATE_TRUNC('hour', created_at) AS hour, COUNT(*) AS events
103140

104141
| Alert | Condition | Why |
105142
| ---------------------- | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
106-
| Sync stalled | `pollAgeSeconds > 300`, or `null` long after deploy | The poller has stopped. |
143+
| Sync stalled | `pollAgeSeconds > 300`, or `null` long after deploy; `/healthz` returns 503 | The poller has stopped. |
107144
| Cursor near retention | `cursorAgeDays > 60` | Replay is still possible, but the window is closing. |
108145
| Stream not writing | `max(_synced_at) < now() - interval '1 hour'` on a busy stream | One stream failing while others continue. |
109146
| Skipped writes spiking | `skipped` in the logs | Out-of-order delivery or a backfill colliding with the tail. Usually benign, occasionally not. |
@@ -131,6 +168,37 @@ The engine emits structured JSON logs with per-batch counts:
131168
staleness gate doing its job. A sudden spike means something changed about
132169
delivery ordering.
133170

171+
### Prometheus metrics
172+
173+
With `WORKOS_SYNC_HEALTH_PORT` set, `listen` also serves `GET /metrics` in
174+
the Prometheus text format, from the same in-memory state as `/healthz`.
175+
Every per-environment series carries an `env` label.
176+
177+
| Metric | Type | What it is |
178+
| ------------------------------ | ------- | ----------------------------------------------------------------------------------------------------- |
179+
| `workos_sync_up` | gauge | `1` while the engine is running; `0` once it has stopped or hit a fatal error. |
180+
| `workos_sync_poll_age_seconds` | gauge | Seconds since the last successful poll. **Alert on this.** Absent until the first successful poll, so a listener that has never succeeded needs the `WorkOSSyncNeverPolled` rule below. |
181+
| `workos_sync_cursor_age_days` | gauge | Age of the newest event applied: distance from the 90-day cliff, not liveness. Absent until the first event. |
182+
| `workos_sync_ticks_total` | counter | Polls attempted, successful or not. |
183+
| `workos_sync_events_total` | counter | Events applied. |
184+
| `workos_sync_writes_total` | counter | Rows written, with a `kind` label of `created`, `updated`, `skipped` or `deleted`. Graph `skipped`. |
185+
| `workos_sync_errors_total` | counter | Polls that failed with a retryable error. Fatal errors stop the process instead. |
186+
187+
```yaml
188+
# prometheus.rules.yml
189+
- alert: WorkOSSyncStalled
190+
expr: workos_sync_poll_age_seconds > 300 or workos_sync_up == 0
191+
for: 2m
192+
# An environment with a ticks series but no poll-age series has never polled
193+
# successfully: every request since startup failed, `/healthz` is 503, and
194+
# nothing above matches. The `for` is the startup grace period.
195+
- alert: WorkOSSyncNeverPolled
196+
expr: workos_sync_ticks_total unless workos_sync_poll_age_seconds
197+
for: 5m
198+
- alert: WorkOSSyncCursorNearRetention
199+
expr: workos_sync_cursor_age_days > 60
200+
```
201+
134202
## Reconciliation schedule
135203
136204
Three cadences, cheapest first:
@@ -313,6 +381,42 @@ environment as the CLI. It honors `HTTPS_PROXY` and `NO_PROXY` for calls to
313381
the WorkOS API. To run the schema creation on each deploy,
314382
run `init` as a one-shot container before starting `listen`.
315383

384+
The image sets `WORKOS_SYNC_HEALTH_PORT=8080`, exposes that port, and
385+
declares a `HEALTHCHECK` that fetches `/healthz` every 30 seconds after a
386+
30-second start period, so `docker ps` and Compose's `depends_on:
387+
condition: service_healthy` reflect whether the tail is actually polling.
388+
The check follows the port `listen` resolved, so changing it with either
389+
`-e WORKOS_SYNC_HEALTH_PORT=<n>` or `listen --health-port <n>` keeps the
390+
container healthy. The check only means something for `listen`: one-shot
391+
commands exit before it would ever run. Publish the port (`-p 8080:8080`)
392+
to scrape `/metrics` from outside the container.
393+
394+
**On Kubernetes**, point both probes at `/healthz`. The engine answers `200`
395+
while it is still starting, so the liveness probe does not need a long
396+
initial delay; what it needs is a `failureThreshold` and `periodSeconds` that
397+
together exceed `WORKOS_SYNC_HEALTH_MAX_POLL_AGE_MS`, or a transient API
398+
outage restarts a pod that would have recovered on its own.
399+
400+
```yaml
401+
containers:
402+
- name: sync
403+
image: ghcr.io/workos/sync-postgres:0.2.0
404+
args: ["listen"]
405+
ports:
406+
- name: health
407+
containerPort: 8080
408+
readinessProbe:
409+
httpGet: { path: /healthz, port: health }
410+
periodSeconds: 10
411+
livenessProbe:
412+
httpGet: { path: /healthz, port: health }
413+
periodSeconds: 30
414+
failureThreshold: 4
415+
```
416+
417+
Add a `prometheus.io/scrape: "true"` annotation (or a `ServiceMonitor`) on
418+
port `health` to collect `/metrics`.
419+
316420
Run **one listener per environment**, each with its own `WORKOS_API_KEY` and
317421
`WORKOS_ENVIRONMENT_NAME`; they can share a database. Two processes tailing the
318422
same environment with the same `syncId` will both advance the same cursor and

‎packages/sync-engine/README.md‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -70,15 +70,33 @@ and the staleness gate decides which version wins.
7070
| `logger` | JSON to stdout | `{ debug, info, warn, error }`. |
7171

7272
Methods: `init()`, `backfill({ resume })`, `tick()`, `listen()`, `stop()`,
73-
`reconcile({ repair, streams })`, `refresh()`, `status()`, `redact(userId)`,
74-
`processWebhook()`, `close()`. Every method applies the schema first, so
75-
`init()` is optional.
73+
`reconcile({ repair, streams })`, `refresh()`, `status()`, `health()`,
74+
`redact(userId)`, `processWebhook()`, `close()`. Every method except
75+
`health()` applies the schema first, so `init()` is optional.
7676

7777
`status()` reports, per environment, the cursor, `pollAgeSeconds` (time since
7878
the last successful poll; alert on this), `cursorAgeDays` (age of the newest
7979
event applied; distance from the 90-day retention cliff, not liveness), and
8080
per-stream row counts, tombstones, staleness and lag.
8181

82+
`health()` is the same liveness picture without the database: a synchronous
83+
snapshot of in-memory state (`starting` / `listening` / `stopped`, any fatal
84+
error, and per environment the last poll, last event, cursor, last error and
85+
running counters). `startHealthServer(engine, { port, host?, maxPollAgeMs? })`
86+
serves it on `GET /healthz` (`200` or `503`) and `GET /metrics` (Prometheus
87+
text), which is what the `listen` command does when given a port. To mount
88+
the same endpoints on a server you already run, `healthReport(engine.health(),
89+
{ maxPollAgeMs })` gives the JSON body and verdict, and
90+
`renderMetrics(engine.health())` the exposition text.
91+
92+
```ts
93+
import { startHealthServer } from "@workos/sync-engine";
94+
95+
const health = await startHealthServer(engine, { port: 8080 });
96+
await engine.listen();
97+
await health.close();
98+
```
99+
82100
A rejected credential (401 or 403) and a cursor that pages backwards are
83101
fatal: the engine stops rather than retrying. Everything else is retried or
84102
isolated by stream.

0 commit comments

Comments
 (0)