[Epic]: UDL as Production Data Layer

[dawidurbanski]

[Epic]: UDL as Production Data Layer

Vision Statement

Deploy UDL as a durable, webhook-driven data cache that build workers and production apps consume, instead of each consumer directly hitting external APIs.

Primary function: Production data layer that build workers and apps query via GraphQL
Secondary function: WebSocket relay for local dev machines to receive real-time updates (opt-in)

External APIs (Contentful, Shopify, etc.)
         │
         │ N webhooks (e.g., 30 entry publishes)
         ▼
┌─────────────────────────────────────────────┐
│              Remote UDL                     │
│  ┌───────────────────────────────────────┐  │
│  │  Webhook Queue + Debounce (Xs wait)   │  │
│  └───────────────────────────────────────┘  │
│         │                                   │
│         ▼ (after quiet period)              │
│  ┌───────────────────────────────────────┐  │
│  │  Node Store + Deletion Log (30d TTL)  │  │
│  └───────────────────────────────────────┘  │
│         │                │          │       │
│   GraphQL API    WebSocket    Outbound      │
│                  (opt-in)     Webhook       │
└─────────────────────────────────────────────┘
      │                │              │
      ▼                ▼              ▼
Build workers    Local devs      Vercel/CI
ISR/SSG/Apps     (real-time)     (1 rebuild)

Why This Matters

• Performance: One UDL instance handles external API calls, not N consumers
• Consistency: Single source of truth for all data consumers
• Real-time: Webhooks keep data current; WebSocket relays to local devs
• Smart Builds: Webhook batching prevents 30 publishes = 30 builds (→ 1 build)
• Partial Sync: Deletion log enables catch-up without full refetch

Key Design Decisions

• GraphQL queries for consumers (no WebSocket needed for reads)
• Webhook queue + debounce prevents N webhooks → N builds
• WebSocket for local dev sync (opt-in, relay changes to connected clients)
• Deletion log (30-day) enables partial sync without full refetch
• Pluggable cache storage already exists
• All config lives in udl.config.ts under remote field

Config API

export const config = defineConfig({
  port: 4000,
  plugins: [ ... ],
  codegen: { ... },
  remote: {
    url: 'https://udl.example.com/graphql',  // Remote UDL URL (for local instances)
    websockets: true,                         // Enable subscriptions over WebSocket
    webhooks: {
      listen: true,                           // Enable listening at /webhooks
      trigger: [                              // Outgoing webhooks (for rebuilds)
        {
          url: 'https://my-webhook-endpoint.com/udl',
          events: ['*'],                      // Which events ('*' = all)
          batch: true,                        // Enable batching
          debounceMs: 1000,                   // Wait 1s for more webhooks
        }
      ]
    }
  }
});

---

What's Already Built

Component	Status
Pluggable Cache Storage	✅ Ready
Node Store with indexes	✅ Ready
GraphQL API	✅ Ready
Hot Reload (dev mode)	✅ Ready
Plugin Lifecycle hooks	✅ Ready

---

What's Missing

Feature	Priority	Purpose
Webhook Infrastructure	CRITICAL	Receive updates from data sources
Webhook Queue & Debounce	CRITICAL	Batch incoming webhooks, trigger ONE outbound rebuild
WebSocket Server	HIGH	Relay changes to local dev machines
Deletion Log	HIGH	Enable partial sync (30-day TTL)
Production Hardening	HIGH	Health checks, graceful shutdown
Deployment Guide	MEDIUM	Docs for running in production

---

Implementation Phases

Phase 1: Production Server Hardening

1. Health check endpoints (/health, /ready)
2. Graceful shutdown (SIGTERM/SIGINT handlers)
3. Request logging (optional)

Files:

• packages/core/src/server.ts
• packages/core/src/start-server.ts

Phase 2: Webhook Infrastructure

1. Plugin webhook registration in SourceNodesContext
2. Webhook routing (POST /_webhooks/:plugin/:handler)
3. Signature verification utilities

Files:

• packages/core/src/loader.ts - add webhook registry
• packages/core/src/server.ts - add webhook routes
• packages/core/src/nodes/index.ts - extend context

API:

// In plugin's sourceNodes hook
registerWebhook({
  path: 'contentful-update',
  handler: async (req, res) => { /* process, update nodes */ },
  verifySignature: (req) => verifyContentfulSignature(req, secret),
});

Phase 3: Webhook Queue & Debounce (Build Trigger Optimization)

1. Queue incoming webhooks instead of immediate processing
2. Configurable debounce window (e.g., 5 seconds)
3. After quiet period, process batch and trigger ONE outbound webhook
4. Outbound webhook URL configurable per environment

Flow:

Webhook 1 arrives → start timer (5s)
Webhook 2 arrives → reset timer (5s)
Webhook 3 arrives → reset timer (5s)
... 5 seconds of quiet ...
→ Process all queued webhooks (update nodes)
→ Trigger ONE outbound webhook (rebuild)
→ Broadcast changes via WebSocket to local devs

Phase 4: Deletion Log

1. Track node deletions with timestamp
2. 30-day TTL (auto-cleanup)
3. Query API: "what was deleted since X?"

Storage: Extend cache storage interface or separate log file

Phase 5: WebSocket Server (opt-in)

1. WebSocket upgrade on /ws or /_sync
2. Broadcast events on node changes: node:created, node:updated, node:deleted
3. Initial sync endpoint: GET /_sync?since={ISO8601}

Protocol:

// Server broadcasts after webhook processing
ws.send(JSON.stringify({
  type: 'node:updated',
  nodeId: 'Product-123',
  timestamp: '2025-12-21T10:30:00Z',
  data: { /* full node or diff */ }
}));

Phase 6: Documentation

1. Production deployment guide (Vercel, Railway, Docker, VPS)
2. Plugin webhook implementation guide
3. Local sync setup guide

---

Success Criteria

• UDL receives webhooks from Contentful/Shopify and updates nodes
• 30 rapid webhooks result in 1 outbound rebuild trigger (batching works)
• Build workers query UDL GraphQL instead of external APIs
• Local dev machines connect via WebSocket and receive real-time updates
• After 24h offline, local machine can partial-sync (deletions included)
• < 5 min to deploy with docs

---

Out of Scope (Future)

• Custom cache backends (Redis, S3) - interface exists, implementations later
• Auth/rate limiting - handle externally (nginx, Cloudflare)
• Multi-region federation

---

Related Work

• Depends on: Core node store infrastructure (complete)
• Enables: Shopify plugin sync ([Epic]: Shopify Source Plugin - Product Sourcing #53), future source plugins
• Parallel with: Contentful uses native Sync API (different approach)

---

Linked Issues

Phase 1: Production Server Hardening

• [Issue]: Add health check endpoints #58 - Add health check endpoints
• [Issue]: Implement graceful shutdown #59 - Implement graceful shutdown

Phase 2: Webhook Infrastructure

• [Issue]: Add plugin webhook registration API #60 - Add plugin webhook registration API
• [Issue]: Add webhook HTTP routing #61 - Add webhook HTTP routing

Phase 3: Webhook Queue & Debounce

• [Issue]: Implement webhook queue with debouncing #62 - Implement webhook queue with debouncing
• [Issue]: Implement outbound webhook triggering #63 - Implement outbound webhook triggering

Phase 4: Deletion Log

• [Issue]: Implement deletion log for partial sync #64 - Implement deletion log for partial sync
• [Issue]: Add sync query API for partial updates #65 - Add sync query API for partial updates

Phase 5: WebSocket Server

• [Issue]: Implement WebSocket server for real-time sync #66 - Implement WebSocket server for real-time sync

Phase 6: Documentation

• [Issue]: Write production deployment documentation #67 - Write production deployment documentation
• [Issue]: Write plugin webhook implementation guide #68 - Write plugin webhook implementation guide
• [Issue]: Write local sync setup guide #69 - Write local sync setup guide

Dependency Graph

Phase 1: Production Hardening
#58 (Health checks)
  │
#59 (Graceful shutdown) ← depends on #58

Phase 2: Webhook Infrastructure
#60 (Webhook registration API)
  │
#61 (Webhook routing) ← depends on #60

Phase 3: Webhook Queue & Debounce
#62 (Webhook queue) ← depends on #61
  │
#63 (Outbound webhooks) ← depends on #62

Phase 4: Deletion Log (can run parallel with Phase 2-3)
#64 (Deletion log)
  │
#65 (Sync API) ← depends on #64

Phase 5: WebSocket Server
#66 (WebSocket server) ← depends on #62

Phase 6: Documentation (after implementation)
#67, #68, #69 ← depends on respective features