feat(sync): two-lane multi-device sync (per-user Durable Object log)#3333
Merged
Conversation
Greenfield workers/sync-hub wrangler project implementing the two-lane
sync plan's server core: SyncHub DO (per-user ordered op log, idempotent
pushOps, paginated getChanges, daily compaction alarm) plus a stateless
front Worker that owns token verification (canonical-userId binding,
KV-cached verdicts, fail-closed) and batch caps. ESLint anti-pattern
guards scoped to src/do; 35 vitest tests via vitest-pool-workers.
Marketplace rsync now excludes /workers. Plan doc included.
Deploy prerequisites (flagged in wrangler.jsonc): confirm cmem.ai verify
endpoint path + {userId} response contract; create AUTH_CACHE KV namespace.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Migration v41 adds origin identity columns (origin_device_id, origin_local_id, sync_rev) to observations/session_summaries/user_prompts with partial unique indexes, plus a sync_state table for the pull cursor and epoch. Narrows the v7 session_summaries rebuild predicate to table-level UNIQUE constraints only (PRAGMA origin 'u') — v41's new index would otherwise retrigger the rebuild every boot and destroy post-v7 column data; regression-tested. New SyncApply module applies remote hub ops in one transaction with the cursor advance: idempotent origin-keyed upserts pre-stamped synced_at (structural echo guard against the push drain), remote timestamps preserved, rev-guarded mutations (set_title/set_prompt_session/ remap_project), set_title convergence via sync_state title parking, epoch-mismatch cursor reset, FTS via existing triggers, post-commit fire-and-forget Chroma. Module header pins the Phase 3 push contract (body field mapping + rev minting rules). Avoids bun:sqlite .run().changes decisions (unreliable after RETURNING in the same tx). 22 new apply tests + v7 regression test; full suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Retargets the push drain to the sync hub's single POST /v1/sync/ops: bodies built verbatim from the SyncApply contract, ack-driven rev-matched synced_at stamping (mid-flight bumps re-push at the higher rev), partial acks fail into backoff (livelock guard), per-kind endpoints, toCloud mappers, stampGuard, and the live|backfill lane deleted. v42 sync_outbox migration + mutation ops at all four sites per the rev minting rules: set_title (clamped at enqueue), set_prompt_session (post-bump rev), and remap_project via a shared own-connection helper (remap-outbox) used by worktree adoption and the cwd remap. New SyncClient pull loop: 30s active / 5min idle / suspended after 1h, head_seq piggyback triggers, epoch-mismatch handling, and a 1.5s-bounded session-start pull in context injection. Hard cutover keyed on hub identity (sync_state cutover_hub_url; epoch mismatch re-nulls native rows) behind new CLAUDE_MEM_CLOUD_SYNC_HUB_URL setting. 83 sync tests green; two-device e2e vs wrangler dev hub 16/16 including kill-mid-flush recovery; full suite + typecheck clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hub: hibernation-native WS upgrade on the DO (acceptWebSocket + serializeAttachment device identity, auto-response ping/pong, empty downstream-only handlers) with per-push fan-out to non-origin sockets — pre-materialization COUNT/SUM gate emits a bare advance frame for large pushes so the isolate never buffers unbounded data; every send guarded, push durability independent of socket health. Client: Bun-native socket in SyncClient (header auth, protocol ping keepalive, full-jitter reconnect 1s..60s). Advisory protocol: epoch checked first, contiguous op frames apply through SyncApply, any anomaly closes the socket and self-heals with one forced HTTP pull. Connected socket stretches polling to the idle tier and drops the push debounce 1500ms -> 250ms (restored on close). Gated by CLAUDE_MEM_CLOUD_SYNC_WS (default on); with sockets disabled Phase 3 behavior is bit-identical — nothing durable rides the socket. CI: sync-hub job (DO anti-pattern grep + both suites), openclaw tests restored, root test script scoped to tests/. WS suite runs separately with --maxWorkers=1 --no-isolate per vitest-pool-workers limitation. 14 WS tests (incl. hibernation eviction + near-cap fan-out), 18 client socket tests; e2e 12/12: 289ms socket delivery, mid-stream kill heals via HTTP. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hourly watchdog as a scheduled handler on the stateless Worker: queries the DO GraphQL Analytics datasets (duration GB-s as the hibernation- defeat detector, rows read/written as runaway detectors, request count), thresholds derived from the validated workload model and env-overridable with min(alert,kill) breach gating, Discord alerts in the release-notify embed shape, and severe duration/rows breaches auto-tripping the kill switch — KV trip strictly before Discord so a dead webhook can never disable the brake. GraphQL failure alerts nothing and trips nothing. Kill switch: KV flag (AUTH_CACHE, control:kill-switch), per-isolate 30s cache, fail-open, checked before auth so every response — including auth failures — carries X-Sync-Mode: poll while tripped; WS upgrades refused before the DO is woken. Clients honor header presence at any status, treat absence as cleared only on OK, close the socket, suppress reconnects, and keep polling — product stays complete in poll mode. Suspended clients now tear the socket down (idle needs no speed layer), making the documented propagation bound structural. Canary: two-device trickle script with bounded --flood for alert-chain rehearsal. DEPLOY.md runbook covers provisioning, secrets, cron, canary deployment, and the trip/degrade/recover verification procedure. The DO itself is untouched — zero diff under src/do/. Hub 68+15 tests, client sync suites 113, e2e trip/recover PASS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hase 6) Full-matrix e2e harness (scripts/sync-matrix-e2e.ts, 39/39): fresh-device bootstrap, >500-op offline catch-up, concurrent writes, all four mutation types in both arrival orders, epoch reset with native-corpus re-push, and kill-switch degradation — plus a two-real-daemon run proving the session-start context-inject pull end-to-end. docs/public/cloud-sync.mdx rewritten for the two-lane architecture (durable HTTP lanes, advisory WebSocket, poll-mode fallback, mutation convergence, settings reference) with availability framed honestly — no live hub URLs. Canary header comment scoped to what it measures. plugin/scripts/worker-service.cjs regenerated — the Phase 5 commit had shipped it stale (missing pollModeOnly/onSyncModeHint/X-Sync-Mode). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Greptile SummaryThis PR replaces the legacy sync path with a two-lane multi-device sync system. The main changes are:
Confidence Score: 5/5The reviewed sync paths appear safe to merge. Core hub, canonical encoding, push drain, pull/apply loop, and migration paths include validation, idempotency, cursor handling, and epoch handling. No accepted issues remain from this review pass. No files require special attention.
 T-Rex LogsWhat T-Rex did
Important Files Changed
Sequence DiagramsequenceDiagram
participant A as Device A CloudSync
participant W as Sync Hub Worker
participant DO as Per-user SyncHub DO
participant P as Pro Projector
participant B as Device B SyncClient
A->>W: POST /v1/sync/ops (canonical ops)
W->>W: Check kill switch and authenticate token/user/device
W->>DO: pushOps(deviceId, ops)
DO->>DO: Validate, dedupe, append ordered log
DO-->>W: acked tuples + head_seq
W->>DO: acquireProjectionLease/head page
W->>P: Project canonical ops through target seq
P-->>W: projected_through_seq
W->>DO: advanceProjectionCheckpoint
W-->>A: 200 acked + head_seq + projected_seq
DO-->>B: Advisory WS op/advance frame
B->>W: "GET /v1/sync/changes?since=cursor"
W->>DO: getChanges(deviceId, since, limit)
DO-->>W: ordered ops + epoch + head_seq
W-->>B: changes page
B->>B: SyncApply applies rows/mutations and advances cursor in one transaction
Reviews (5): Last reviewed commit: "chore(sync-hub): fill real AUTH_CACHE KV..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
619
to
623
| let size = JSON.stringify(body).length; | ||
| if (size > MAX_OP_BYTES) { | ||
| body = clampFields(body, HARD_FIELD_BYTES); | ||
| size = JSON.stringify(body).length; | ||
| } |
Contributor
There was a problem hiding this comment.
The size guard uses JSON.stringify(...).length, but the hub validates UTF-8 bytes with TextEncoder.encode(body).length. A row with large non-ASCII text can pass this client check, be rejected by /v1/sync/ops as body exceeds 1990000 bytes, and remain permanently queued because every retry rebuilds the same oversized op. Use byte length for the packed op/body before sending.
Comment on lines
+186
to
208
| toBody: (r) => ({ | ||
| memory_session_id: r.memory_session_id ?? null, | ||
| project: r.project ?? 'unknown', | ||
| text: r.text ?? null, | ||
| type: r.type ?? null, | ||
| title: r.title ?? null, | ||
| subtitle: r.subtitle ?? null, | ||
| facts: parseJson(r.facts), | ||
| facts: jsonPayloadColumn(r.facts, 'facts', 'array'), | ||
| narrative: r.narrative ?? null, | ||
| concepts: parseJson(r.concepts), | ||
| filesRead: parseJson(r.files_read), | ||
| filesModified: parseJson(r.files_modified), | ||
| promptNumber: r.prompt_number ?? null, | ||
| discoveryTokens: r.discovery_tokens ?? 0, | ||
| createdAtEpoch: r.created_at_epoch ?? null, | ||
| concepts: jsonPayloadColumn(r.concepts, 'concepts', 'array'), | ||
| files_read: jsonPayloadColumn(r.files_read, 'files_read', 'array'), | ||
| files_modified: jsonPayloadColumn(r.files_modified, 'files_modified', 'array'), | ||
| prompt_number: decimalPayload(r.prompt_number, 'prompt_number', true), | ||
| discovery_tokens: decimalPayload(r.discovery_tokens, 'discovery_tokens'), | ||
| content_hash: r.content_hash ?? null, | ||
| generated_by_model: r.generated_by_model ?? null, | ||
| agent_type: r.agent_type ?? null, | ||
| agent_id: r.agent_id ?? null, | ||
| metadata: jsonPayloadColumn(r.metadata, 'metadata', 'object'), | ||
| merged_into_project: r.merged_into_project ?? null, | ||
| created_at: r.created_at ?? null, | ||
| created_at_epoch: decimalPayload(r.created_at_epoch, 'created_at_epoch'), | ||
| }), |
Contributor
There was a problem hiding this comment.
observation ops are built without the session platform_source, and summaries use the same shape below. When a Cursor or Codex-origin session syncs to another device, SyncApply creates the remote session stub with the default platform and forwards observations/summaries to Chroma without the original platform. Those rows are then indexed and scoped as the default platform, while prompt ops preserve platform_source through their joined payload.
Artifacts
Repro: focused Bun test harness for non-default platform observation sync
- Contains supporting evidence from the run (text/typescript; charset=utf-8).
Repro: Bun test output showing dropped platform_source and default receiver stub
- Keeps the command output available without making the summary code-heavy.
Retire legacy client adoption and per-kind sync state, establish the one-time launch boundary, and expose authenticated Hub health. Add bounded device metadata and a loopback-only two-client protocol-v2 matrix E2E.
…der Linux readdir order Bun's mock.module is process-global and sticky: once a file registers a partial mock, it persists for every later test file in the same process. Test scheduling follows readdir order, which differs between macOS and Linux, so five poisoner files leaked partial mocks into cross-file victims only on CI. Fixed with the repo's snapshot + afterAll re-register pattern. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements multi-device memory sync over a two-lane architecture — durable HTTP lanes (per-user ordered op log on a Cloudflare Durable Object, cursor-based pull, idempotent push) plus an advisory WebSocket speed layer that can be wrong, dropped, or disabled with zero data loss. Replaces the old push-only per-kind lane in this branch.
Legacy sync code did ship in v13.11.0. All known consumers are user-confirmed test accounts; no separate production dependency has been evidenced, so this is scoped as a pre-launch replacement — not as a claim that no legacy client was released.
Server (
workers/sync-hub/, new wrangler project, not shipped in the plugin bundle):SyncHubDO: ordered op log, idempotentpushOps/ paginatedgetChanges(RPC), daily compaction alarm, hibernation-native WS fan-out with pre-materialization size gating. Zero timers, zero outbound I/O in the DO — ESLint guards + CI grep enforce it.X-Sync-Mode: pollkill switch checked before auth and before any DO wake.--flood,DEPLOY.mdrunbook.Client:
SyncApply: transactional apply with cursor-in-tx (crash-safe exactly-once), structural echo guard (synced_atpre-stamped), rev-guarded mutations, set_title parking for out-of-order convergence.CloudSyncretarget: singlePOST /v1/sync/ops, ack-driven rev-matched stamping, partial-ack livelock guard, all-kinds size clamps.SyncClient: tiered polling (30s/5min/suspend at 1h, socket torn down on suspend), 1.5s-bounded session-start pull, advisory socket with epoch-first frame checks and HTTP self-heal, poll-mode fallback honoring header presence at any status.Verification
scripts/sync-matrix-e2e.ts(39/39 — fresh-device bootstrap, >500-op offline catch-up, concurrent writes, all four mutation types both orders, epoch-change handling, no-echo) andscripts/sync-kill-switch-e2e.ts(trip → poll-mode convergence → recovery). Two-real-daemon run proved the session-start context-inject pull end-to-end. Socket delivery measured 267–289 ms.Deploy prerequisites (blocking activation, not merge)
Flagged in
wrangler.jsonc+workers/sync-hub/DEPLOY.md: cmem.ai verify endpoint ({userId}contract; the route is implemented at/api/pro/sync/verifyand must be deployed, withTOKEN_VERIFY_URLconfigured to its production URL, before Worker activation),AUTH_CACHEKV namespace, watchdog secrets + cron, canary deployment.🤖 Generated with Claude Code