Skip to content

progress.md

Latest commit

 

History

History
289 lines (211 loc) · 10.5 KB

progress.md

File metadata and controls

289 lines (211 loc) · 10.5 KB

LiveSQL Progress Tracker

Update this file as tasks are completed. Check boxes with [x].

Current Sprint — v1.0.0-beta.1 Release

Pre-Release Documentation

  1. Update README: npm badge to beta, remove alpha references, update roadmap Phase 3 to say "beta"
  2. Update svelte package description to include liveTable (already done)
  3. Package READMEs: review for accuracy, ensure install commands don't reference alpha
  4. CONTRIBUTING.md: brief contributor guide (setup, PR process, coding conventions)
  5. CHANGELOG.md: summarize changes from alpha → beta (features, breaking changes if any)
  6. Docs site (apps/docs): update intro page version references to beta (already version-agnostic)
  7. Integration guide: brief "Adding LiveSQL to an existing Express/Fastify app" page
  8. Deployment guide: brief "Running LiveSQL in production" (PostgreSQL config, env vars, health checks)
  9. PostgREST integration guide: "Using PostgREST + LiveSQL" — Docker Compose setup, architecture diagram, example showing PostgREST for CRUD + LiveSQL for real-time streaming

Release

  1. Bump all package versions to 1.0.0-beta.1
  2. Add publishConfig: { "access": "public" } to all packages (already set)
  3. Build all packages (pnpm build)
  4. Run full test suite (pnpm test) — 163 tests passing
  5. Commit version bump and doc updates
  6. Publish to npm in dependency order: core → client → server → react → vue → svelte
  7. Create git tag v1.0.0-beta.1 and GitHub release with beta pre-release flag
  8. Push tag and release

Post-Release

  1. Hacker News "Show HN" submission
  2. Reddit post (r/typescript, r/node, r/webdev)
  3. Twitter/X announcement thread
  4. Dev.to / Hashnode launch blog post (how LiveSQL works, benchmarks, quick start)
  5. PostgREST deeper integration: shared Docker Compose template, example repo (postgrest-livesql-starter), optional auto-discovery of PostgREST tables
  6. Collect feedback for 2–4 weeks
  7. Triage issues and fix API ergonomic problems (if any)
  8. Promote to v1.0.0 stable release once feedback is addressed

Previous Sprint — Phase 3: Production Hardening — DONE

  1. Chaos tests (23 tests across 6 failure modes) → all passing
  2. Replication slot failover detection and auto-recovery → PostgresProvider.recoverFromSlotLoss()
  3. Comprehensive TypeScript types → audit clean, added useLiveTable to Vue/Svelte for parity
  4. Firebase migration guide → apps/docs/docs/guides/migration-firebase.md
  5. Fix integration test event batching assertions → predicate-based wait instead of exact count

Previous Sprint (2026-03-07) — DONE

  1. Authorization header support (Bearer <token>) → Phase 1 complete
  2. k6 load test (1,000 concurrent clients) → p95 96ms, all thresholds passed
  3. Publish benchmark results → tests/load/RESULTS.md
  4. Observability hooks (onEvent, onClientConnect, onClientDisconnect) → done

Phase 0 — Foundation & PoC

Monorepo Setup

  • Initialize git repository
  • Create pnpm-workspace.yaml
  • Create root package.json with workspace scripts
  • Create root tsconfig.base.json (ES2022, NodeNext, strict)
  • Configure eslint + prettier
  • Configure husky + lint-staged
  • Create .gitignore
  • Create docker-compose.test.yml (PostgreSQL 16, wal_level=logical)

packages/core

  • Create package.json for @livesql/core
  • Create tsconfig.json with project references
  • Define ChangeType and ChangeEvent types
  • Define SubscribeMessage and UnsubscribeMessage types
  • Define SyncMessage and ErrorMessage types
  • Define ClientMessage and ServerMessage union types
  • Define ChangeProvider interface
  • Create index.ts with public exports
  • Build succeeds with tsup

packages/server

  • Create package.json for @livesql/server
  • Create tsconfig.json
  • Implement ListenNotifyProvider (LISTEN/NOTIFY for PoC)
  • Create trigger function for INSERT/UPDATE/DELETE notifications
  • Implement minimal WebSocket server (createLiveSQLServer)
  • Handle subscribe/unsubscribe messages
  • Fan-out events to matching subscribers
  • Create index.ts with public exports
  • Build succeeds

packages/client

  • Create package.json for @livesql/client
  • Create tsconfig.json
  • Implement LiveSQLClient class
  • WebSocket connect and message handling
  • Subscribe/unsubscribe methods
  • Basic reconnection (simple retry)
  • Offset tracking
  • Create index.ts with public exports
  • Build succeeds

apps/demo

  • Create demo application (HTML + vanilla JS)
  • Live order status dashboard
  • Auto-updates on INSERT/UPDATE/DELETE
  • SQL setup script for demo tables

Documentation

  • Write README.md with 5-minute quickstart
  • Verify: clone → install → docker up → insert row → see in browser < 5 min

Phase 1 — WAL-Based CDC Engine + Alpha

PostgreSQL WAL Provider

  • Implement PostgresProvider class
  • Create publication and replication slot
  • Open replication connection
  • Parse pgoutput binary messages (Relation, Begin, Insert, Update, Delete, Commit)
  • Cache Relation messages by OID
  • Map column positions to names
  • Emit ChangeEvent for each row change
  • Track monotonic offset per event
  • Implement replayFrom(offset) for reconnection backfill

WAL Slot Health

  • Implement checkSlotHealth() function
  • Query pg_replication_slots for lag_bytes
  • Configurable warning threshold (default 512MB)
  • Emit slot:lag-warning event
  • Emit slot:inactive event
  • Run health check on 30s interval

Authentication & Permissions

  • JWT verification on WebSocket handshake
  • Token from query string (?token=)
  • Token from Authorization header (Bearer <token>)
  • opts.permissions(userId, table) callback on subscribe
  • Reject with FORBIDDEN error code

Filter Validation

  • Implement validateFilter() — parse column operator value
  • Validate column against allowedFilterColumns
  • Validate operator against allowlist (=, !=, <, >, <=, >=)
  • Implement matchesFilter() — in-process filter evaluation
  • Reject with INVALID_FILTER error code

Reconnection

  • Client stores lastOffset on every received event
  • Client sends offset in subscribe message on reconnect
  • Server replays from offset via replayFrom()
  • Exponential backoff: 250ms → 30s cap

Integration Tests

  • Test: INSERT event delivered to subscribed client
  • Test: UPDATE event delivered with old + new row
  • Test: DELETE event delivered
  • Test: Reconnection backfills missed events
  • Test: Permission rejection returns FORBIDDEN
  • Test: Invalid filter returns INVALID_FILTER
  • Test: Unsubscribe stops event delivery

Publish Alpha

  • @livesql/core v0.1.0-alpha.2 on npm (updated to alpha.3 in Phase 2)
  • @livesql/server v0.1.0-alpha.2 on npm (updated to alpha.3 in Phase 2)
  • @livesql/client v0.1.0-alpha.2 on npm (updated to alpha.3 in Phase 2)

Phase 2 — React SDK & Ecosystem + Beta

React Package

  • LiveSQLProvider context provider
  • useLiveQuery<T>() hook with insert/update/delete handling
  • useLiveTable<T>() Map-based hook
  • useLiveSQLClient() escape hatch

Vue Package

  • useLiveQuery composable
  • useLiveTable composable (Map-based, O(1) row lookups)
  • createLiveSQLPlugin Vue plugin (provide/inject via LIVESQL_CLIENT_KEY)

Svelte Package

  • liveQuery store (lazy Readable<{data, loading, error}>, explicit client param)
  • liveTable store (Map-based, O(1) row lookups)

Server Hardening

  • Row-level permission: opts.rowPermission(userId, table, row) (Phase 1)
  • EventBatcher — coalesce, flush at 50 events or 16ms
  • Backpressure detection (ws.bufferedAmount > 1MB)
  • opts.onBackpressure(userId) callback

Client Hardening

  • LiveSQLError exported from @livesql/client
  • onError callback in subscribe() routes server errors to callers
  • filter param in subscribe() sent in subscribe/re-subscribe messages

Publish Alpha

  • @livesql/core v0.1.0-alpha.3 on npm
  • @livesql/server v0.1.0-alpha.3 on npm
  • @livesql/client v0.1.0-alpha.3 on npm
  • @livesql/react v0.1.0-alpha.4 on npm (first publish)
  • @livesql/vue v0.1.0-alpha.4 on npm (first publish)
  • @livesql/svelte v0.1.0-alpha.4 on npm (first publish)

Documentation Site

  • Set up Docusaurus in apps/docs
  • 5-minute quickstart tutorial
  • Concepts page: how sync works
  • API reference for all packages
  • Migration guide from Supabase Realtime

Demo & Load Testing

  • Polished React demo application (apps/react-demo — Vite + @livesql/react)
  • k6 load test: 1,000 concurrent clients (p95 event latency 96ms, all thresholds passed)
  • Published benchmark results (tests/load/RESULTS.md)

Publish Beta

  • Publish beta before stable — v1.0.0-beta.1 for early adopter feedback

Phase 3 — Production Hardening & v1.0

Chaos Tests

  • Network partition (simulated disconnect + reconnect with offset replay)
  • Slow client / high bufferedAmount (backpressure detection)
  • PostgreSQL primary failover (slot loss detection + auto-recovery)
  • WAL disk approaching limit (checkSlotHealth lag detection)
  • Mass reconnect / thundering herd (200 simultaneous connections)
  • SQL injection via filter (13 injection vectors tested)

Production Features

  • Exponential backoff with jitter (±25%, implemented in client)
  • Observability hooks: onEvent, onError, onSlotLag, onClientConnect, onClientDisconnect
  • Replication slot failover detection and auto-recovery (reconnectOnSlotLoss, onSlotLost)
  • Comprehensive TypeScript types — audit clean, useLiveTable added to Vue/Svelte

Documentation

  • Performance benchmark (p50/p95/p99) — tests/load/RESULTS.md
  • Migration guide from Firebase Realtime Database
  • Launch blog post

Publish v1.0

  • v1.0.0-beta.1 → collect feedback → v1.0.0 stable
  • Hacker News submission

Phase 4 — MySQL & Enterprise

MySQL (demand-gated: >20 GitHub issues)

  • Implement MySQLProvider with binlog support
  • Integration tests for MySQL

Enterprise

  • Managed cloud service (LiveSQL Cloud)
  • Audit logging
  • SAML SSO