Skip to content

Commit 81b7d42

Browse files
committed
feat(new-deepnotes): realtime user-hash REQUEST in UserRealtimeRoom and batch Vitest
Execute legacy realtime REQUEST batches (HGET/HSET/SUBSCRIBE/UNSUBSCRIBE) against Redis key user:{userId} when Upstash is configured, with RESPONSE and DATA_NOTIFICATION fan-out in the Durable Object. Add executeRealtimeWsBatch helper and tests; update TRPC_REST_MAP and PLAN_PROGRESS.
1 parent eb8431b commit 81b7d42

5 files changed

Lines changed: 600 additions & 10 deletions

File tree

new-deepnotes/PLAN_PROGRESS.md

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@
1212

1313
| Area | State |
1414
|------|--------|
15-
| **API / REST** | TRPC_REST_MAP HTTP rows largely done; **collab WS** MVP (**Durable Object** relay + Postgres append, legacy lib0 framing via `@deepnotes/collab-wire`); **realtime** `USER_NOTIFICATION` path: **`GET /api/realtime-ws`** + **`UserRealtimeRoom` DO** + `@deepnotes/realtime-wire`. **`@deepnotes/realtime-wire`** now implements legacy **REQUEST / RESPONSE / DATA_NOTIFICATION** framing + Vitest round-trips (Worker still pushes **`USER_NOTIFICATION` only**; Redis-backed hash **`HGET`/`SUBSCRIBE`** parity **not** wired—SPA stays REST-heavy for titles). |
16-
| **SPA** | Auth, lists, **`PageEditorView`** (Tiptap+Y+collab **WebSocket when configured** else REST, **encrypted awareness + collaboration carets** on WS, link/underline/placeholder, **tables, images, tasks, code (lowlight), KaTeX math (inline + block), YouTube embeds (Vue node view + resize handle like legacy)**, highlight, align, sub/sup, HR, path breadcrumb, bump/favorite/recent, **snapshots list/save/restore/delete**, **set-as-main + soft-delete + purge**, **cross-group move/reencrypt**), groups/**members**/invite/join + **group settings** (join policy, soft-delete, **make public / make private**, **group purge**), notifications list **with decrypt**, **`/account`**, home **recents/favorites/starting/defaults** UIs, theme, **live notification toast** when **`/api/realtime-ws`** connected. Missing: **realtime hash cache** (full legacy parity), **spatial/world** canvas, native shells. |
15+
| **API / REST** | TRPC_REST_MAP HTTP rows largely done; **collab WS** MVP (**Durable Object** relay + Postgres append, legacy lib0 framing via `@deepnotes/collab-wire`); **realtime** **`GET /api/realtime-ws`** + **`UserRealtimeRoom` DO** + `@deepnotes/realtime-wire`. Framing + Vitest round-trips; **`USER_NOTIFICATION`** push after `performNotifyUsers`. **Hash slice:** when **Upstash** is configured (`UPSTASH_REDIS_*`), DO handles legacy **REQUEST** batches for **`user:{userId}`** Redis (**HGET**/**HSET**/**SUBSCRIBE**/**UNSUBSCRIBE****RESPONSE** + **DATA_NOTIFICATION** intra-DO fan-out). **Still open:** **`page`** / **`group`** hash ACL vs Postgres (legacy DataAbstraction); cross-process Redis pub/sub (legacy `expiremember`/KeyDB) not replicated. SPA remains REST-heavy for titles until client opts into hash WS. |
16+
| **SPA** | Auth, lists, **`PageEditorView`** (Tiptap+Y+collab **WebSocket when configured** else REST, **encrypted awareness + collaboration carets** on WS, link/underline/placeholder, **tables, images, tasks, code (lowlight), KaTeX math (inline + block), YouTube embeds (Vue node view + resize handle like legacy)**, highlight, align, sub/sup, HR, path breadcrumb, bump/favorite/recent, **snapshots list/save/restore/delete**, **set-as-main + soft-delete + purge**, **cross-group move/reencrypt**), groups/**members**/invite/join + **group settings** (join policy, soft-delete, **make public / make private**, **group purge**), notifications list **with decrypt**, **`/account`**, home **recents/favorites/starting/defaults** UIs, theme, **live notification toast** when **`/api/realtime-ws`** connected. Missing: **`page`/`group` realtime hashes + SPA wired** (full legacy parity; worker has **`user:{id}`** hash slice when Upstash is set); **spatial/world** canvas, native shells. |
1717

1818
**Deferred (confirm vs parity):** optional anon `GET …/groups/:id/pages`; richer Stripe webhook tests. **Infra naming:** Workers/DO replaces standalone collab/realtime/scheduler processes.
1919

@@ -26,7 +26,7 @@
2626
| **0** | Done | Map, OpenAPI, Drizzle baseline, CLIENT_FORKS |
2727
| **1** | Skip? | Legacy monorepo only |
2828
| **2** | Done | Turbo/CI/template DB/deploy docs |
29-
| **3** | WIP | **realtime:** `USER_NOTIFICATION` WS + DO + invite **notifyUsers** persistence; legacy **hash `HGET`/`SUBSCRIBE`** still **not** (REST/UI does not depend on it yet); collab WS MVP **done** |
29+
| **3** | WIP | **realtime:** `USER_NOTIFICATION` WS + DO + invite **`notifyUsers`**; **`user`-hash** REQUEST (**Upstash**) + RESP/DATA_NOTIFICATION in-DO (**`page`/`group` ACL + cross-node pub/sub** still open); collab WS MVP **done** |
3030
| **4** | WIP | See checklist below |
3131
| **5** | Todo | Cutover after **parity gate** + metrics + decrypt spot-checks |
3232

@@ -46,7 +46,7 @@
4646
- [x] `[parity]` Page ops + group settings — **make private** + cross-group **move/reencrypt** + **purge** UI (page + group); **make public**, snapshots + main + soft-delete **done** in SPA
4747
- [ ] `[parity]` Editor UX vs legacy (rich + spatial/world if in scope) — **partial:** WS awareness+carets; TipTap **tables, images, tasks**, **code (lowlight), math (inline + block), YouTube** (Vue node view + resize handle), typography (highlight, align, sub/sup, HR), link/underline/placeholder (**no** spatial/world canvas yet)
4848
- [x] `[parity]` Notifications: decrypt/display as legacy
49-
- [x] Legacy **realtime****partial:** `USER_NOTIFICATION` over **`/api/realtime-ws`** + per-user DO (`@deepnotes/realtime-wire`); join-invite **DB notifications** + E2EE payloads; **not** legacy Redis/DataAbstraction hash sync
49+
- [x] Legacy **realtime****partial:** `USER_NOTIFICATION` over **`/api/realtime-ws`** + per-user DO (`@deepnotes/realtime-wire`); join-invite **DB notifications** + E2EE payloads; **`user:{id}`** hash **REQUEST**/RESP (**Upstash** optional); **`page`/`group`** caches + Redis pub/sub parity **not** done
5050
- [ ] Capacitor/Tauri **after web parity**
5151

5252
---
@@ -69,7 +69,7 @@
6969

7070
**Open:**
7171

72-
- [ ] Collab **and** realtime: ≥1 integration test each — `collab-wire` **decodeIncoming** unit coverage; **realtime-wire** RESPONSE/DATA_NOTIFICATION/REQUEST Vitest round-trips; full DO / Redis hash path **TBD** in CI
72+
- [ ] Collab **and** realtime: ≥1 integration test each — `collab-wire` **decodeIncoming** unit coverage; **realtime-wire** framing Vitest round-trips; **api-worker** `executeRealtimeWsBatch` (**user** hash) Vitest — full DO + Redis in CI **TBD**
7373
- [ ] Stripe / high-risk: deeper tests when secrets allow
7474
- [ ] Staging CF: Hyperdrive + Postgres + Redis + WS topology load-tested
7575
- [ ] **UI parity** in `@deepnotes/web` (account + notification decrypt + page prefs/home/editor + group join/delete + rich TipTap **done**; **realtime hash cache** + **spatial/world** still open — goal above)
@@ -80,6 +80,7 @@
8080

8181
| Date | Note |
8282
|------|------|
83+
| 2026-04-29 | **Realtime Redis hash slice (Upstash):** `UserRealtimeRoom` handles binary **REQUEST** (`executeRealtimeWsBatch`); **`user:{userId}`** HGET/HSET/SUBSCRIBE/UNSUBSCRIBE + subscriber fan-out; **`realtime-ws-batch.test.ts`**. |
8384
| 2026-04-29 | **Realtime wire framing parity:** `@deepnotes/realtime-wire` decodes/encodes legacy **RESPONSE**, **DATA_NOTIFICATION**, and client **REQUEST** batches (msgpackr + lib0) with Vitest; Worker DO unchanged (`USER_NOTIFICATION` only). Moves toward Redis hash cache parity without E2E. |
8485
| 2026-04-29 | **Realtime `USER_NOTIFICATION` parity slice:** `@deepnotes/realtime-wire`, `UserRealtimeRoom` DO, `GET /api/realtime-ws`, `performNotifyUsers` + join-invite optional `notifications` + bootstrap `?inviteeUserId=` keyrings; SPA toast + `buildGroupInviteSentNotifications`. |
8586
| 2026-04-29 | **TipTap deep parity:** code blocks (**lowlight** + atom-one-dark CSS), **YouTube** embeds, **KaTeX** inline + block math (Vue node views, legacy `inline-math` / `math-block` HTML tags); direct **`@tiptap/core` + `@tiptap/pm`** deps to avoid parent-monorepo TipTap v2 resolution. |
Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
import { describe, expect, it } from "vitest";
2+
3+
import {
4+
encodeRealtimeClientRequest,
5+
decodeRealtimeClientBinaryMessage,
6+
RealtimeCommandType,
7+
} from "@deepnotes/realtime-wire";
8+
9+
import {
10+
canRealtimeHashAccess,
11+
executeRealtimeWsBatch,
12+
realtimeFullKey,
13+
type RealtimeHashPort,
14+
} from "./realtime-ws-batch.js";
15+
16+
function createMemoryHashPort(
17+
initial: Record<string, Record<string, unknown>> = {},
18+
): { port: RealtimeHashPort; snapshot: () => Record<string, Record<string, unknown>> } {
19+
const store = new Map<string, Record<string, unknown>>();
20+
for (const [k, row] of Object.entries(initial)) {
21+
store.set(k, { ...row });
22+
}
23+
const port: RealtimeHashPort = {
24+
async hmget(key, fields) {
25+
const row = store.get(key) ?? {};
26+
return fields.map((f) => {
27+
if (!Object.prototype.hasOwnProperty.call(row, f)) {
28+
return null;
29+
}
30+
return row[f] ?? null;
31+
});
32+
},
33+
async hset(key, entries) {
34+
let row = store.get(key);
35+
if (row == null) {
36+
row = {};
37+
store.set(key, row);
38+
}
39+
Object.assign(row, entries);
40+
},
41+
};
42+
return {
43+
port,
44+
snapshot: () => {
45+
const o: Record<string, Record<string, unknown>> = {};
46+
for (const [k, v] of store) {
47+
o[k] = { ...v };
48+
}
49+
return o;
50+
},
51+
};
52+
}
53+
54+
describe("executeRealtimeWsBatch", () => {
55+
it("denies group/page hash access (user-only until Postgres ACL)", () => {
56+
expect(canRealtimeHashAccess("u1", "group", "g1")).toBe(false);
57+
expect(canRealtimeHashAccess("u1", "page", "p1")).toBe(false);
58+
expect(canRealtimeHashAccess("u1", "user", "u1")).toBe(true);
59+
expect(canRealtimeHashAccess("u1", "user", "u2")).toBe(false);
60+
});
61+
62+
it("HGET returns batched hmget values for own user hash", async () => {
63+
const { port } = createMemoryHashPort({
64+
"user:u1": { email: "a@b.c", plan: "pro" },
65+
});
66+
const req = encodeRealtimeClientRequest({
67+
firstCommandId: 1,
68+
commands: [
69+
{
70+
type: RealtimeCommandType.HGET,
71+
args: ["user", "u1", "email"],
72+
},
73+
{
74+
type: RealtimeCommandType.HGET,
75+
args: ["user", "u1", "plan"],
76+
},
77+
],
78+
});
79+
const decoded = decodeRealtimeClientBinaryMessage(req);
80+
if (decoded == null) {
81+
throw new Error("decode");
82+
}
83+
const hooks = { subscribeField: () => {}, unsubscribeField: () => {} };
84+
const out = await executeRealtimeWsBatch({
85+
userId: "u1",
86+
decoded,
87+
redis: port,
88+
hooks,
89+
});
90+
expect(out.responseBytes).not.toBeNull();
91+
expect(out.subscribeNotifyBytes).toBeNull();
92+
expect(out.hsetBroadcastItems).toEqual([]);
93+
});
94+
95+
it("SUBSCRIBE registers hook and sends initial DATA_NOTIFICATION", async () => {
96+
const { port } = createMemoryHashPort({
97+
"user:u1": { email: "x@y.z" },
98+
});
99+
const subs: string[] = [];
100+
const req = encodeRealtimeClientRequest({
101+
firstCommandId: 0,
102+
commands: [
103+
{
104+
type: RealtimeCommandType.SUBSCRIBE,
105+
args: ["user", "u1", "email"],
106+
},
107+
],
108+
});
109+
const decoded = decodeRealtimeClientBinaryMessage(req);
110+
if (decoded == null) {
111+
throw new Error("decode");
112+
}
113+
const out = await executeRealtimeWsBatch({
114+
userId: "u1",
115+
decoded,
116+
redis: port,
117+
hooks: {
118+
subscribeField: (fk) => {
119+
subs.push(fk);
120+
},
121+
unsubscribeField: () => {},
122+
},
123+
});
124+
expect(subs).toEqual([realtimeFullKey("user", "u1", "email")]);
125+
expect(out.subscribeNotifyBytes).not.toBeNull();
126+
});
127+
128+
it("HSET notifies broadcast list items for subscribed fan-out handled by DO", async () => {
129+
const { port } = createMemoryHashPort();
130+
const req = encodeRealtimeClientRequest({
131+
firstCommandId: 10,
132+
commands: [
133+
{
134+
type: RealtimeCommandType.HSET,
135+
args: ["user", "u1", "email", "new@mail"],
136+
},
137+
],
138+
});
139+
const decoded = decodeRealtimeClientBinaryMessage(req);
140+
if (decoded == null) {
141+
throw new Error("decode");
142+
}
143+
const out = await executeRealtimeWsBatch({
144+
userId: "u1",
145+
decoded,
146+
redis: port,
147+
hooks: { subscribeField: () => {}, unsubscribeField: () => {} },
148+
});
149+
expect(out.hsetBroadcastItems).toHaveLength(1);
150+
expect(out.hsetBroadcastItems[0]?.fullKey).toBe(
151+
realtimeFullKey("user", "u1", "email"),
152+
);
153+
});
154+
});

0 commit comments

Comments
 (0)