feat(core): UDL as Production Data Layer (Epic #57)

.changeset/adapter-nextjs-config-utils.md

@@ -0,0 +1,34 @@
+---
+'@universal-data-layer/adapter-nextjs': minor
+---
+
+Add config file support and UDL_ENDPOINT injection for Next.js adapter
+
+The adapter commands now read the UDL port from `udl.config.ts` and automatically inject the `UDL_ENDPOINT` environment variable when spawning Next.js processes.
+
+**Features:**
+
+- `dev`, `build`, and `start` commands read port from config file
+- Port priority: CLI option > config file > default (4000)
+- Next.js processes receive `UDL_ENDPOINT` env var automatically
+- Supports `udl.config.ts`, `udl.config.js`, and `udl.config.mjs`
+
+**Benefits:**
+
+- No need to manually set `UDL_ENDPOINT` in environment
+- `udl.query()` client automatically uses the correct endpoint
+- Consistent port configuration between UDL server and Next.js
+
+**Example:**
+
+```typescript
+// udl.config.ts
+export const { config } = defineConfig({
+  port: 5000, // Adapter commands will use this port
+});
+```
+
+```typescript
+// In Next.js code, udl.query() automatically uses the right endpoint
+const result = await udl.query(GetProducts);
+```

.changeset/add-health-endpoints.md

@@ -0,0 +1,25 @@
+---
+'universal-data-layer': minor
+---
+
+Add health check endpoints for production deployments
+
+Introduces `/health` and `/ready` endpoints to support container orchestration (Kubernetes, Docker Swarm), load balancers, and deployment verification.
+
+**Endpoints:**
+
+- `GET /health` - Liveness probe, returns 200 when server is running
+- `GET /ready` - Readiness probe, returns 200 when fully initialized, 503 during startup
+
+**Response format:**
+
+```json
+// /health
+{ "status": "ok", "timestamp": "2025-12-21T10:30:00Z" }
+
+// /ready (when ready)
+{ "status": "ready", "timestamp": "2025-12-21T10:30:00Z", "checks": { "graphql": true, "nodeStore": true } }
+
+// /ready (during startup)
+{ "status": "initializing", "timestamp": "2025-12-21T10:30:00Z", "checks": { "graphql": false, "nodeStore": false } }
+```

.changeset/add-webhook-http-routing.md

@@ -0,0 +1,37 @@
+---
+'universal-data-layer': minor
+---
+
+Add webhook HTTP routing with convention-based URL pattern
+
+Routes incoming webhook requests to the appropriate plugin handler using a fixed URL pattern `POST /_webhooks/{pluginName}/sync`.
+
+**Features:**
+
+- Convention-based routing: all webhooks use the `/sync` path
+- Routes webhooks to correct handler based on plugin name
+- Validates HTTP method (only POST allowed)
+- Collects raw request body for handler processing
+- Parses JSON body when content-type is `application/json`
+- Provides `WebhookHandlerContext` with store, actions, rawBody, and body
+- Enforces 1MB body size limit to prevent abuse
+- Returns appropriate HTTP status codes (405, 404, 400, 413)
+- Queues webhooks for batch processing with debounce
+
+**URL Format:**
+
+```
+POST /_webhooks/{plugin-name}/sync
+
+Examples:
+POST /_webhooks/contentful/sync
+POST /_webhooks/shopify/sync
+POST /_webhooks/my-plugin/sync
+```
+
+**Exports:**
+
+- `isWebhookRequest` - Check if URL is a webhook request
+- `getPluginFromWebhookUrl` - Extract plugin name from webhook URL
+- `webhookHandler` - HTTP handler for webhook requests
+- `WEBHOOK_PATH_PREFIX` - URL prefix constant (`/_webhooks/`)

.changeset/add-webhook-registration-api.md

@@ -0,0 +1,70 @@
+---
+'universal-data-layer': minor
+---
+
+Add plugin webhook handler export API
+
+Plugins can now export a `registerWebhookHandler` function to handle webhooks with custom logic. When exported, it replaces the default CRUD handler for the plugin's `/_webhooks/{plugin-name}/sync` endpoint.
+
+**Usage in plugins:**
+
+```typescript
+// Plugin's udl.config.ts
+import { defineConfig } from 'universal-data-layer';
+
+export const config = defineConfig({
+  name: 'my-cms-plugin',
+  type: 'source',
+});
+
+// Custom webhook handler replaces the default
+export async function registerWebhookHandler({ req, res, actions, body, store, rawBody }) {
+  // Verify signature using your CMS's method
+  const signature = req.headers['x-webhook-signature'];
+  if (!verifySignature(rawBody, signature)) {
+    res.writeHead(401);
+    res.end('Invalid signature');
+    return;
+  }
+
+  // Handle different event types
+  const eventType = req.headers['x-webhook-type'];
+
+  if (eventType === 'entry.publish') {
+    await actions.createNode(transformEntry(body), { ... });
+  } else if (eventType === 'entry.delete') {
+    await actions.deleteNode(body.sys.id);
+  }
+
+  res.writeHead(200);
+  res.end();
+}
+```
+
+**Key benefits:**
+
+- Clear separation: `sourceNodes` for sourcing, `registerWebhookHandler` for webhooks
+- Convention-based URL: always `/_webhooks/{plugin-name}/sync`
+- Plugin controls its own routing and signature verification internally
+- Replaces default handler - no confusion about which handler runs
+
+**Handler context:**
+
+The handler receives a flattened context object:
+
+```typescript
+interface PluginWebhookHandlerContext {
+  req: IncomingMessage; // The incoming HTTP request
+  res: ServerResponse; // The server response
+  actions: NodeActions; // Node CRUD operations
+  store: NodeStore; // Access to all nodes
+  body: unknown; // Parsed JSON body
+  rawBody: Buffer; // Raw body for signature verification
+}
+```
+
+**New exports:**
+
+- `PluginWebhookHandler` - Type for the handler function
+- `PluginWebhookHandlerContext` - Type for the handler context
+- `registerPluginWebhookHandler` - Internal utility for registering custom handlers

.changeset/cache-manager.md

@@ -0,0 +1,38 @@
+---
+'universal-data-layer': minor
+---
+
+Add centralized cache manager for plugin cache coordination
+
+Introduces a `CacheManager` module that provides a central point for coordinating plugin cache updates. This enables webhook handlers and remote sync to persist changes to disk after store modifications.
+
+**Features:**
+
+- `registerPluginCache(pluginName, cache)`: Register a plugin's cache storage
+- `initPluginCache(pluginName, cacheLocation, customCache?)`: Initialize and register a cache
+- `savePluginCache(pluginName, store?)`: Save a specific plugin's nodes to cache
+- `saveAffectedPlugins(affectedPlugins, store?)`: Save caches for multiple plugins
+- `replaceAllCaches(store?)`: Replace all plugin caches (for remote sync)
+- `setStore(store)`: Set the node store reference for cache operations
+
+**Integration:**
+
+- Loader now uses cache manager for plugin cache operations
+- Webhook batch processing automatically saves affected plugin caches
+- Remote sync persists fetched nodes to cache for offline support
+
+**New exports:**
+
+```typescript
+import {
+  setStore,
+  getStore,
+  registerPluginCache,
+  initPluginCache,
+  savePluginCache,
+  saveAffectedPlugins,
+  replaceAllCaches,
+  clearAllCaches,
+  resetCacheManager,
+} from 'universal-data-layer';
+```

.changeset/config-env-var-support.md

@@ -0,0 +1,23 @@
+---
+'universal-data-layer': patch
+---
+
+Add UDL_ENDPOINT environment variable support to config
+
+The `getConfig()` function now checks for the `UDL_ENDPOINT` environment variable when config hasn't been explicitly initialized. This allows the `udl.query()` client to automatically use the correct endpoint in child processes.
+
+**Features:**
+
+- `UDL_ENDPOINT_ENV` constant for the environment variable name
+- `DEFAULT_UDL_PORT` constant (4000) for consistent default port
+- `isConfigInitialized()` to check if config was explicitly set
+- `resetConfig()` for testing isolation
+
+**How it works:**
+
+When `getConfig()` is called and no config was explicitly set via `createConfig()`, it checks for the `UDL_ENDPOINT` environment variable and uses that endpoint if present.
+
+This enables scenarios like:
+
+- Next.js adapter sets `UDL_ENDPOINT` when spawning Next.js
+- `udl.query()` in Next.js code automatically uses the right endpoint

.changeset/default-webhook-handler.md

@@ -0,0 +1,67 @@
+---
+'universal-data-layer': minor
+---
+
+Add default webhook handler for standardized CRUD operations
+
+This release introduces a default webhook handler that provides a standardized way to create, update, and delete nodes via webhooks. Every loaded plugin automatically gets a webhook endpoint registered with zero configuration required.
+
+**Features:**
+
+- Automatic registration of `/_webhooks/{plugin-name}/sync` endpoint for every plugin
+- Standardized payload format for `create`, `update`, `delete`, and `upsert` operations
+- Support for custom `idField` to look up nodes by external identifiers
+- Won't overwrite custom handlers if plugin registers its own
+
+**Zero Configuration:**
+
+```typescript
+// No config needed - default webhooks just work
+// Every plugin gets: /_webhooks/{plugin-name}/sync
+export const { config } = defineConfig({
+  plugins: ['@universal-data-layer/plugin-source-contentful'],
+});
+```
+
+**Payload format:**
+
+```typescript
+interface DefaultWebhookPayload {
+  operation: 'create' | 'update' | 'delete' | 'upsert';
+  nodeId: string; // External ID or internal node ID
+  nodeType: string; // Node type (e.g., 'Product', 'Article')
+  data?: Record<string, unknown>; // Node data (required for create/update/upsert)
+}
+```
+
+**Example requests:**
+
+```bash
+# Create a node
+curl -X POST http://localhost:4000/_webhooks/my-plugin/sync \
+  -H "Content-Type: application/json" \
+  -d '{"operation":"create","nodeId":"123","nodeType":"Product","data":{"name":"Widget"}}'
+
+# Update a node
+curl -X POST http://localhost:4000/_webhooks/my-plugin/sync \
+  -d '{"operation":"update","nodeId":"123","nodeType":"Product","data":{"name":"Updated Widget"}}'
+
+# Delete a node
+curl -X POST http://localhost:4000/_webhooks/my-plugin/sync \
+  -d '{"operation":"delete","nodeId":"123","nodeType":"Product"}'
+
+# Upsert (create or update)
+curl -X POST http://localhost:4000/_webhooks/my-plugin/sync \
+  -d '{"operation":"upsert","nodeId":"123","nodeType":"Product","data":{"name":"Widget"}}'
+```
+
+**idField support:**
+
+When a plugin specifies an `idField` in its config, the default webhook handler looks up existing nodes by that field:
+
+```typescript
+// Plugin config
+export const config = defineConfig({
+  idField: 'externalId', // Webhook will look up nodes by this field
+});
+```

.changeset/deletion-log.md

@@ -0,0 +1,36 @@
+---
+'universal-data-layer': minor
+---
+
+Add deletion log for partial sync support
+
+This release introduces a DeletionLog class that tracks node deletions with timestamps, enabling clients to perform partial sync without needing a full refetch.
+
+**Features:**
+
+- `DeletionLog` class for tracking deleted nodes
+- `recordDeletion(node)`: Record a node deletion with timestamp
+- `getDeletedSince(timestamp)`: Query deletions after a given time
+- `cleanup()`: Remove entries older than TTL (default: 30 days)
+- Serialization support via `toJSON()` and `fromJSON()` for persistence
+- Configurable TTL (time-to-live) for deletion entries
+
+**Example usage:**
+
+```typescript
+import { DeletionLog } from 'universal-data-layer';
+
+const log = new DeletionLog(30); // 30 day TTL
+
+// Record a deletion
+log.recordDeletion(deletedNode);
+
+// Query deletions since last sync
+const deletedSince = log.getDeletedSince(lastSyncTimestamp);
+
+// Serialize for persistence
+const data = log.toJSON();
+
+// Restore from persistence
+const restored = DeletionLog.fromJSON(data);
+```

.changeset/graceful-shutdown.md

@@ -0,0 +1,12 @@
+---
+'universal-data-layer': minor
+---
+
+feat(core): add graceful shutdown for production deployments
+
+- Handle SIGTERM and SIGINT signals for graceful shutdown
+- Complete in-flight requests before closing server
+- Return 503 on `/ready` endpoint during shutdown
+- Configurable grace period (default: 30 seconds)
+- Clean up file watchers and resources on shutdown
+- Log shutdown progress to console

.changeset/instant-webhook-relay.md

@@ -0,0 +1,43 @@
+---
+'universal-data-layer': minor
+---
+
+Add instant webhook relay for remote sync
+
+Local UDL instances can now receive and process webhooks instantly via WebSocket relay, eliminating the need to wait for batch debounce on the production server.
+
+**How it works:**
+
+1. Production UDL receives a webhook and queues it
+2. Immediately broadcasts `webhook:received` message to WebSocket subscribers
+3. Local UDL instances receive the message and process the webhook locally
+4. Local caches are updated instantly
+
+**Features:**
+
+- New `webhook:queued` event on WebhookQueue for instant relay
+- New `webhook:received` WebSocket message type
+- `broadcastWebhookReceived(webhook)` method on UDLWebSocketServer
+- `onWebhookReceived` callback on WebSocketClient and RemoteSyncConfig
+- Local UDL instances can process relayed webhooks using registered handlers
+- Node change events are skipped when handling webhooks locally (avoids double processing)
+
+**Configuration:**
+
+The instant relay is automatically enabled when using remote sync. Local instances register webhook handlers by loading plugins with `isLocal: true` option.
+
+**Message format:**
+
+```typescript
+interface WebhookReceivedMessage {
+  type: 'webhook:received';
+  pluginName: string;
+  body: unknown;
+  headers: Record<string, string | string[] | undefined>;
+  timestamp: string;
+}
+```
+
+**Exports:**
+
+- `WebhookReceivedEvent`: Event data passed to onWebhookReceived callback