Skip to content

Commit 2f454e2

Browse files
authored
feat: Generate a _live view for every mirror stream (#9)
1 parent a2f8c4a commit 2f454e2

17 files changed

Lines changed: 1119 additions & 83 deletions

File tree

‎README.md‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -18,12 +18,14 @@ Against a mirror, it becomes one statement:
1818

1919
```sql
2020
SELECT o.name AS organization, u.email, m.role_slug
21-
FROM workos.organization_memberships m
22-
JOIN workos.users u ON u.id = m.user_id AND u._env = m._env
23-
JOIN workos.organizations o ON o.id = m.organization_id AND o._env = m._env
24-
WHERE u._deleted_at IS NULL AND m._deleted_at IS NULL;
21+
FROM workos.organization_memberships_live m
22+
JOIN workos.users_live u ON u.id = m.user_id AND u._env = m._env
23+
JOIN workos.organizations_live o ON o.id = m.organization_id AND o._env = m._env;
2524
```
2625

26+
The `_live` views are generated next to the tables and hide deleted rows; the
27+
base tables keep them as tombstones for when you want to ask who left.
28+
2729
## Try it in 30 seconds
2830

2931
No credentials, no Docker, no network. Runs the whole lifecycle against a
@@ -109,7 +111,7 @@ the same setting, the flag wins.
109111

110112
| Command | Flags | What it does |
111113
| ----------- | ------------- | ------------------------------------------------------------------------------------------------------------ |
112-
| `init` | | Create the schema and bookkeeping tables. Idempotent; safe on every deploy. |
114+
| `init` | `--repair` | Create the schema and bookkeeping tables. Idempotent; safe on every deploy. Fails if a generated `_live` view must be rebuilt; `--repair` rebuilds it and carries its grants over. |
113115
| `backfill` | `--resume` | Load current state from the REST list endpoints. `--resume` continues from the last checkpoint. |
114116
| `listen` | `--health-port <n>` | Poll the Events API continuously. Finishes the current batch on `SIGINT` or `SIGTERM`. With a port, serves `/healthz` and `/metrics`. |
115117
| `tick` | | One poll, then exit. For a cron job instead of a long-running process. |
@@ -163,7 +165,9 @@ A field added upstream can never break ingest — it is queryable as
163165
`_raw->>'field'` immediately, and promoting it to a typed column is pure DDL
164166
computed from rows already present. There are **no foreign keys**: events
165167
routinely arrive with children before parents, and an FK turns that ordering
166-
artifact into a stuck stream.
168+
artifact into a stuck stream. Every mirror table also gets a `<table>_live`
169+
view (`workos.users_live`) that is the table minus its tombstoned rows, so the
170+
common query needs no `_deleted_at` filter.
167171

168172
## What it mirrors
169173

‎docs/consistency.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -119,9 +119,12 @@ Deleted objects are marked, not removed:
119119
- `_deleted_at IS NOT NULL`: deleted upstream, payload retained
120120
- no row at all: never synced, or hard-deleted by `redact`
121121

122-
Filter on `_deleted_at IS NULL` for live data. The distinction between "deleted"
123-
and "never seen" is usually worth having, and the retained payload is the only
124-
record of what an object looked like when it was removed.
122+
Filter on `_deleted_at IS NULL` for live data, or read the generated
123+
`<table>_live` view, which is the table with that filter built in
124+
([Live views](schema.md#live-views)). The distinction between "deleted" and
125+
"never seen" is only visible on the base table, and is usually worth having:
126+
the retained payload is the only record of what an object looked like when it
127+
was removed.
125128

126129
Once set, `_deleted_at` is never cleared on a mirror object. A late `updated`
127130
event refreshes the payload but does not resurrect the row, because WorkOS ids

‎docs/getting-started.md‎

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -95,18 +95,17 @@ the library) to turn it off.
9595

9696
```sql
9797
SELECT o.name AS organization, u.email, m.role_slug
98-
FROM workos.organization_memberships m
99-
JOIN workos.users u ON u.id = m.user_id AND u._env = m._env
100-
JOIN workos.organizations o ON o.id = m.organization_id AND o._env = m._env
101-
WHERE u._deleted_at IS NULL AND m._deleted_at IS NULL
98+
FROM workos.organization_memberships_live m
99+
JOIN workos.users_live u ON u.id = m.user_id AND u._env = m._env
100+
JOIN workos.organizations_live o ON o.id = m.organization_id AND o._env = m._env
102101
ORDER BY o.name, u.email;
103102
```
104103

105104
Two conventions to know:
106105

107-
- **Always filter `_deleted_at is null`** for live rows. Deleted objects are
108-
tombstoned rather than removed, so you can still see what was deleted and
109-
when.
106+
- **Read the `_live` views for live rows.** Deleted objects are tombstoned in
107+
the base table (`_deleted_at`) rather than removed, so you can still see what
108+
was deleted and when by querying `workos.users` directly.
110109
- **Always join on `_env` as well as the id.** Object ids are unique within an
111110
environment, and the mirror can hold several environments at once.
112111

‎docs/queries.md‎

Lines changed: 52 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,10 @@ than one, add `_env = 'production'` or join on `_env`.
55

66
There are two conventions throughout:
77

8-
- Filter `_deleted_at is null` for live rows.
8+
- Read the `_live` views (`workos.users_live`, `workos.organizations_live`,
9+
...) for live rows. Each is its table minus the tombstoned rows, so no
10+
`_deleted_at is null` filter is needed. Read the base table when you want
11+
deleted rows too; see [Live views](schema.md#live-views).
912
- Join on `_env` as well as the id.
1013

1114
## Seats and adoption
@@ -17,10 +20,10 @@ SELECT o.name AS organization,
1720
COUNT(*) AS seats,
1821
COUNT(*) FILTER (WHERE u.last_sign_in_at IS NULL
1922
OR u.last_sign_in_at < NOW() - INTERVAL '30 days') AS dormant
20-
FROM workos.organization_memberships m
21-
JOIN workos.users u ON u.id = m.user_id AND u._env = m._env
22-
JOIN workos.organizations o ON o.id = m.organization_id AND o._env = m._env
23-
WHERE u._deleted_at IS NULL AND m._deleted_at IS NULL AND m.status = 'active'
23+
FROM workos.organization_memberships_live m
24+
JOIN workos.users_live u ON u.id = m.user_id AND u._env = m._env
25+
JOIN workos.organizations_live o ON o.id = m.organization_id AND o._env = m._env
26+
WHERE m.status = 'active'
2427
GROUP BY o.name
2528
HAVING COUNT(*) FILTER (WHERE u.last_sign_in_at IS NULL
2629
OR u.last_sign_in_at < NOW() - INTERVAL '30 days') > 0
@@ -36,27 +39,30 @@ seats counted either way.
3639

3740
```sql
3841
SELECT u.email, u.created_at, NOW() - u.created_at AS age
39-
FROM workos.users u
40-
LEFT JOIN workos.organization_memberships m
41-
ON m.user_id = u.id AND m._env = u._env AND m._deleted_at IS NULL
42-
WHERE u._deleted_at IS NULL
43-
AND m.id IS NULL
42+
FROM workos.users_live u
43+
LEFT JOIN workos.organization_memberships_live m
44+
ON m.user_id = u.id AND m._env = u._env
45+
WHERE m.id IS NULL
4446
AND u.created_at < NOW() - INTERVAL '7 days'
4547
ORDER BY u.created_at;
4648
```
4749

50+
Note that the `_live` view on the outer side of the `LEFT JOIN` does the work
51+
that `m._deleted_at IS NULL` in the `ON` clause would otherwise have to: a user
52+
whose only membership was revoked still shows up here.
53+
4854
## Enterprise readiness
4955

5056
**Organizations with a verified domain but no directory.**
5157

5258
```sql
5359
SELECT o.name AS organization, d.domain, o.created_at
54-
FROM workos.organizations o
55-
JOIN workos.organization_domains d
60+
FROM workos.organizations_live o
61+
JOIN workos.organization_domains_live d
5662
ON d.organization_id = o.id AND d._env = o._env AND d.state = 'verified'
57-
LEFT JOIN workos.directories dir
58-
ON dir.organization_id = o.id AND dir._env = o._env AND dir._deleted_at IS NULL
59-
WHERE o._deleted_at IS NULL AND dir.id IS NULL
63+
LEFT JOIN workos.directories_live dir
64+
ON dir.organization_id = o.id AND dir._env = o._env
65+
WHERE dir.id IS NULL
6066
ORDER BY o.created_at;
6167
```
6268

@@ -66,26 +72,26 @@ mirror kept fresh by `reconcile` — is how you find them.
6672

6773
```sql
6874
SELECT o.name AS organization, d.name AS directory, d.type, d.state, d.updated_at
69-
FROM workos.directories d
70-
JOIN workos.organizations o ON o.id = d.organization_id AND o._env = d._env
71-
WHERE d._deleted_at IS NULL AND d.state <> 'linked'
75+
FROM workos.directories_live d
76+
JOIN workos.organizations_live o ON o.id = d.organization_id AND o._env = d._env
77+
WHERE d.state <> 'linked'
7278
ORDER BY d.updated_at;
7379
```
7480

7581
**SSO adoption by connection type.**
7682

7783
```sql
7884
SELECT connection_type, state, COUNT(*) AS connections
79-
FROM workos.connections
80-
WHERE _deleted_at IS NULL
85+
FROM workos.connections_live
8186
GROUP BY connection_type, state
8287
ORDER BY connections DESC;
8388
```
8489

8590
## Authentication
8691

8792
These read `auth_attempts`, which exists only because the mirror captured it —
88-
there is no endpoint to ask for this history.
93+
there is no endpoint to ask for this history. It is append-only, so it has no
94+
`_live` view and nothing to filter.
8995

9096
**Failed sign-ins by user, last hour.**
9197

@@ -102,6 +108,10 @@ SELECT COALESCE(u.email, a.email) AS email,
102108
ORDER BY failures DESC;
103109
```
104110

111+
This one joins the base `workos.users` table on purpose: an attempt against an
112+
account that has since been deleted should still be reported under its email,
113+
and the tombstone is where that email lives.
114+
105115
**Sign-in success rate by method.** The `type` column is the method
106116
(`password`, `sso`, `oauth`, `magic_auth`, ...); `status` is derived from
107117
whether the event name ended in `_succeeded` or `_failed`.
@@ -147,10 +157,9 @@ SELECT u.email,
147157
u.metadata->>'plan' AS plan,
148158
COUNT(s.id) AS sessions_30d,
149159
MAX(s.created_at) AS last_active
150-
FROM workos.users u
160+
FROM workos.users_live u
151161
JOIN app.accounts a ON a.id = u.external_id
152162
LEFT JOIN app.sessions s ON s.account_id = a.id AND s.created_at > NOW() - INTERVAL '30 days'
153-
WHERE u._deleted_at IS NULL
154163
GROUP BY u.email, u.metadata->>'plan'
155164
ORDER BY sessions_30d DESC;
156165
```
@@ -160,11 +169,10 @@ already on the organization object:
160169

161170
```sql
162171
SELECT o.name, s.status AS subscription, COUNT(m.id) AS seats
163-
FROM workos.organizations o
172+
FROM workos.organizations_live o
164173
JOIN billing.subscriptions s ON s.customer = o.stripe_customer_id
165-
LEFT JOIN workos.organization_memberships m
166-
ON m.organization_id = o.id AND m._env = o._env AND m._deleted_at IS NULL
167-
WHERE o._deleted_at IS NULL
174+
LEFT JOIN workos.organization_memberships_live m
175+
ON m.organization_id = o.id AND m._env = o._env
168176
GROUP BY o.name, s.status;
169177
```
170178

@@ -194,7 +202,8 @@ SELECT DATE_TRUNC('day', created_at) AS day,
194202

195203
## Deleted objects
196204

197-
Tombstones mean you can still ask who left.
205+
Tombstones mean you can still ask who left. This is the one kind of question
206+
the `_live` views cannot answer, so these read the base table.
198207

199208
```sql
200209
SELECT email, _deleted_at AS deleted_at, last_sign_in_at
@@ -203,24 +212,29 @@ SELECT email, _deleted_at AS deleted_at, last_sign_in_at
203212
ORDER BY _deleted_at DESC;
204213
```
205214

215+
The base table is also the only place that tells "deleted" apart from "never
216+
synced": a row with `_deleted_at` set was seen and then removed upstream; no row
217+
at all means the mirror never saw it.
218+
206219
## Convenience views
207220

208-
If filtering tombstones on every query is tedious, create views once:
221+
The `_live` views are generated for you, one per mirror table, and re-created
222+
every time the schema is applied, so they always carry the table's current
223+
columns. If you want them under different names or in a separate schema, build
224+
your own views on top of them rather than editing them: the engine never drops
225+
a view it did not create.
209226

210227
```sql
211228
CREATE SCHEMA IF NOT EXISTS workos_live;
212229

213-
CREATE VIEW workos_live.users AS
214-
SELECT * FROM workos.users WHERE _deleted_at IS NULL;
215-
216-
CREATE VIEW workos_live.organizations AS
217-
SELECT * FROM workos.organizations WHERE _deleted_at IS NULL;
230+
CREATE VIEW workos_live.users AS SELECT * FROM workos.users_live;
218231
```
219232

220233
## Giving an agent read access
221234

222235
A read-only role over the mirror is a far better-scoped grant than an API key,
223-
and one SQL tool replaces a pile of per-endpoint tools:
236+
and one SQL tool replaces a pile of per-endpoint tools. `ALL TABLES` covers the
237+
`_live` views as well:
224238

225239
```sql
226240
CREATE ROLE analytics_reader LOGIN PASSWORD '...';
@@ -238,11 +252,10 @@ CREATE SCHEMA IF NOT EXISTS workos_safe;
238252

239253
CREATE VIEW workos_safe.users AS
240254
SELECT u.id, u._env, u.created_at, u.last_sign_in_at,
241-
COUNT(m.id) FILTER (WHERE m._deleted_at IS NULL) AS organizations
242-
FROM workos.users u
243-
LEFT JOIN workos.organization_memberships m
255+
COUNT(m.id) AS organizations
256+
FROM workos.users_live u
257+
LEFT JOIN workos.organization_memberships_live m
244258
ON m.user_id = u.id AND m._env = u._env
245-
WHERE u._deleted_at IS NULL
246259
GROUP BY u.id, u._env, u.created_at, u.last_sign_in_at;
247260

248261
GRANT USAGE ON SCHEMA workos_safe TO analytics_reader;

‎docs/schema.md‎

Lines changed: 60 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,69 @@ Everything else is `GENERATED ALWAYS AS ... STORED`, projected out of `_raw`.
1818

1919
Two consequences worth internalising:
2020

21-
- **Filter `_deleted_at is null`** unless you specifically want deleted rows.
21+
- **Read the `_live` view** (next section) unless you specifically want
22+
deleted rows. On a base table, that means filtering `_deleted_at is null`.
2223
- **Join on `_env` as well as the id.** Ids are unique within an environment,
2324
and one database can mirror several.
2425

26+
## Live views
27+
28+
Every table that has a `_deleted_at` column — every `mirror` and `poll`
29+
stream — also gets a view named `<table>_live`:
30+
31+
```sql
32+
CREATE VIEW workos.users_live AS
33+
SELECT * FROM workos.users WHERE _deleted_at IS NULL;
34+
```
35+
36+
Forgetting the tombstone filter is the most common mistake made against the
37+
mirror, so the view exists to make the common query the safe one:
38+
`SELECT ... FROM workos.users_live` can only ever return live rows.
39+
40+
Things to know:
41+
42+
- **`select *` on the view omits nothing.** The view has exactly the columns of
43+
its table, in the same order — `_env`, `_raw`, the timestamps, every
44+
generated column, and `_deleted_at` itself (always `null` through the view).
45+
A column promoted later with `extraColumns` appears in the view after the
46+
next `init`, because the view is re-created every time the schema is applied.
47+
- **The base table is still there, and still the only place that distinguishes
48+
"deleted" from "never seen".** A row absent from the view could be either;
49+
the table answers which, with the payload the object had when it was
50+
removed. The [Deleted objects](queries.md#deleted-objects) queries read the
51+
table on purpose.
52+
- **The engine never reads or writes through the views.** Upserts, `redact`,
53+
`reconcile`, `status` and freshness all use the base tables, so the views are
54+
purely a convenience for your queries.
55+
- **Do not edit them.** They are generated from the catalog, carry a comment
56+
saying so, and are re-created on every schema apply. A generated view you
57+
edited by hand while keeping its comment and its shape is simply overwritten
58+
on the next apply. Build your own views on top of the generated ones if you
59+
want different names or a separate schema; the engine never drops a view it
60+
did not create.
61+
- **Rebuilding a view needs `--repair`.** If a generated view's shape can no
62+
longer be replaced in place (for instance it changed between releases), the
63+
engine does not rebuild it on its own: `create or replace view` keeps a
64+
view's grants, a drop and re-create does not, and re-granting a reader on a
65+
different definition is an authorization decision, not a repair. Instead
66+
`init` fails, names the view, and lists the grants it would have to carry
67+
over. Run `init --repair` (`repairViews: true` on `PostgresDestination`) to
68+
consent: the view is dropped and re-created in one transaction, those grants
69+
are re-applied and each is logged, and a re-create that fails leaves the old
70+
view where it was. Or drop the view by hand and grant selectively. Two kinds
71+
of view are never rebuilt, with or without `--repair`, and `init` fails
72+
asking you to rename or drop them: one without the generated comment, which
73+
is yours — even with the same shape, where Postgres itself would have
74+
overwritten it in place — and one with the comment but not the shape of
75+
`select *` over its table, which was rebuilt by hand. A view and its comment
76+
are created in one transaction, so an interrupted `init` cannot leave one
77+
without the other; should a view with exactly the generated definition and
78+
no comment at all exist anyway, `init` recognises it as its own and finishes
79+
it. One carrying another tool's comment is never adopted, whatever its
80+
definition: that comment is the other tool's claim on it.
81+
- **Append-only streams have no view.** `events` and `auth_attempts` have no
82+
`_deleted_at` column, so there is nothing to filter.
83+
2584
## Bookkeeping tables
2685

2786
| Table | Purpose |

‎examples/basic/e2e.ts‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -168,11 +168,11 @@ async function main() {
168168
table(
169169
(
170170
await db.query(
171+
// The _live views are the tables minus tombstones, so no _deleted_at filter.
171172
`select o.name as organization, u.email, m.role_slug as role
172-
from workos.organization_memberships m
173-
join workos.users u on u.id = m.user_id and u."_env" = m."_env"
174-
join workos.organizations o on o.id = m.organization_id and o."_env" = m."_env"
175-
where u."_deleted_at" is null and m."_deleted_at" is null
173+
from workos.organization_memberships_live m
174+
join workos.users_live u on u.id = m.user_id and u."_env" = m."_env"
175+
join workos.organizations_live o on o.id = m.organization_id and o."_env" = m."_env"
176176
order by o.name, u.email`,
177177
)
178178
).rows,

0 commit comments

Comments
 (0)