feat(core): UDL as Production Data Layer (Epic #57)#70
Merged
Conversation
Add /health (liveness) and /ready (readiness) endpoints to support container orchestration and load balancer health checks. - /health returns 200 when server is running - /ready returns 200 when fully initialized, 503 during startup - Includes timestamp and component status in responses Closes #58
18 tasks
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Implement proper SIGTERM/SIGINT signal handlers to gracefully shut down the UDL server, ensuring in-flight requests complete before exit and resources are properly cleaned up. - Add shutdown state management module - Handle SIGTERM and SIGINT signals - Complete in-flight requests before closing server - Return 503 on /ready endpoint during shutdown - Configurable grace period (default: 30 seconds) - Clean up file watchers and debounce timers on shutdown - Log shutdown progress to console Closes #59
Extend SourceNodesContext with registerWebhook function, allowing plugins to register webhook handlers for receiving payloads from external sources. - Add WebhookRegistry class for managing webhook handlers - Add WebhookRegistration, WebhookHandler, WebhookHandlerContext types - Extend loadPlugins and loadConfigFile to pass registerWebhook to plugins - Export webhook utilities from core package - Add comprehensive unit tests (33 tests) Closes #60
Add HTTP routing for incoming webhooks at POST /_webhooks/:plugin/:path. Routes webhook payloads to the correct plugin handler registered via the webhook registry. Features: - Route webhooks based on plugin name and path from URL - Validate HTTP method (only POST allowed, returns 405 otherwise) - Collect raw request body for signature verification - Parse JSON body when content-type is application/json - Provide WebhookHandlerContext with store, actions, rawBody, and body - Enforce 1MB body size limit to prevent abuse (returns 413) - Return appropriate HTTP status codes (405, 404, 401, 400, 500) - Log webhook activity for debugging Closes #61
Implement webhook queue system that batches incoming webhooks and processes them after a configurable debounce period. This prevents N rapid webhook events from triggering N separate processing cycles. Features: - WebhookQueue class with configurable debounceMs (default 5s) - Maximum queue size before forced processing (default 100) - Lifecycle hooks for custom processing: - onWebhookReceived: transform/filter webhooks before queuing - onBeforeWebhookTriggered: run before batch (e.g., cache invalidation) - onAfterWebhookTriggered: run after batch (e.g., trigger rebuild) - Graceful shutdown flushes pending webhooks - Config via remote.webhooks.debounceMs, hooks, maxQueueSize Breaking changes: - Webhook HTTP response changed from 200 to 202 Accepted - Webhooks processed asynchronously after debounce period Closes #62
Add OutboundWebhookManager to notify external systems (e.g., Vercel deploy hooks, CI systems) after webhook batches are processed. This enables the "30 webhooks → 1 build" optimization. Features: - Configurable outbound endpoints via remote.webhooks.trigger - Retry logic with exponential backoff (default: 3 retries) - Custom headers support for authentication - Parallel triggering to multiple endpoints - Payload includes batch summary (webhookCount, plugins, timestamp)
dawidurbanski
changed the title
Add a default webhook handler system that provides a standardized endpoint for create/update/delete/upsert operations on nodes. This enables external systems to sync data with UDL using a consistent payload format. Key features: - DefaultWebhookPayload type for standardized request format - createDefaultWebhookHandler() factory with idField support for custom lookups - registerDefaultWebhook/registerDefaultWebhooks utilities for auto-registration - Per-plugin configuration with path overrides and disable options - Handles both string and numeric ID lookups for flexibility The handler supports looking up existing nodes by a custom idField (e.g., externalId, contentfulId) instead of internal IDs, enabling seamless integration with external data sources.
Integrate the default webhook handler system into the plugin loading process: - Add `idField` config option for plugins to specify their external ID field - Auto-register default webhook handlers for plugins when `defaultWebhook` config is enabled - Automatically index the idField for O(1) lookups - Update UDLConfig interface with defaultWebhook configuration - Re-export default webhook types and functions from core package When a plugin specifies an idField (e.g., 'contentfulId'), the default webhook handler uses this field to look up existing nodes, enabling external systems to reference nodes by their source IDs.
Add MSW mock handlers for testing the default webhook functionality with a remote todo API simulation. Includes manual test feature for verifying webhook-based CRUD operations in development mode.
Add GET /_sync endpoint that enables clients to query for changes since a given timestamp. Returns updated nodes and deleted node IDs for efficient partial sync operations. - Add defaultDeletionLog singleton following defaultStore pattern - Create sync handler with SyncResponse interface - Register /_sync route in server - Export SyncResponse, defaultDeletionLog, setDefaultDeletionLog - Add 23 unit tests and 10 integration tests Closes #65
Implement an opt-in WebSocket server that broadcasts node changes to connected clients immediately when nodes are created, updated, or deleted. - Add WebSocketConfig to RemoteConfig with enabled, port, path, and heartbeatIntervalMs options - Create node event emitter that fires on createNode, deleteNode, and extendNode actions - Implement UDLWebSocketServer class with subscription filtering, heartbeat for connection health, and graceful shutdown - Integrate WebSocket server initialization in start-server.ts - Add 31 unit and integration tests for events and WebSocket server Clients can subscribe to specific node types or all types (*) and receive full node data in real-time, enabling local dev machines to receive updates immediately when webhooks modify the data layer. Closes #66
Add changeset documentation for: - WebSocket server for real-time node change notifications - Sync query API for partial updates - Default webhook handler for standardized CRUD operations
Add `remote.url` config option that allows local UDL servers to sync data from a remote production server instead of sourcing from plugins. Features: - Fetches all nodes from remote /_sync endpoint on startup - Connects to remote WebSocket for real-time updates (if enabled) - Gracefully handles WebSocket unavailability - Skips local plugin loading when remote.url is configured New files: - packages/core/src/websocket/client.ts - WebSocket client - packages/core/src/sync/remote.ts - Remote sync logic New exports: - UDLWebSocketClient, WebSocketClientConfig - fetchRemoteNodes, tryConnectRemoteWebSocket, initRemoteSync Documentation: - Added deployment section with production, standalone, webhook, and remote sync guides
When remote.url points to the same host:port as the local server, UDL now detects this and loads plugins instead of syncing from remote. This allows sharing the same config file between production and local development - production will source from plugins while local dev will sync from the remote production server. Self-detection handles: - Exact host:port matches - Local host aliases (localhost, 127.0.0.1, 0.0.0.0)
The previous approach compared host:port which didn't work for shared configs where both production and local dev use the same URL. New approach: check if the remote server is actually reachable. - If reachable → sync from remote (we are local dev) - If not reachable → load plugins (we are production) This correctly handles: - Same machine, same port (production starts first) - Same machine, different ports - Multi-tenant localhost setups - Remote production servers
Let the config file determine the default port instead of hardcoding it in CLI args and adapter commands. Port flag is now only passed when explicitly specified by the user.
Introduce a simplified webhook registration system that allows plugins
to register handlers at convention-based URLs (/_webhooks/{plugin}/sync).
- WebhookRegistry class with register/getHandler/has methods
- WebhookRegistration interface with handler, description, verifySignature
- SignatureVerifier type for optional request signature validation
- Default singleton registry with setDefaultWebhookRegistry for testing
Route webhooks to handlers using convention-based URLs:
POST /_webhooks/{plugin-name}/sync
- isWebhookRequest() and getPluginFromWebhookUrl() for URL parsing
- Signature verification before queuing (returns 401 if invalid)
- JSON body parsing with 400 error for invalid JSON
- Request body collection with 1MB limit
- Integration tests for full webhook routing flow
Provide a built-in webhook handler that processes standardized payloads for node CRUD operations, automatically registered for plugins. - DefaultWebhookPayload with create/update/delete/upsert operations - createDefaultWebhookHandler factory with idField support - registerDefaultWebhook for auto-registration during plugin loading - registerPluginWebhookHandler for custom plugin handlers - Integration with webhook processor for batch execution
Trigger outbound webhooks after batch processing to notify external systems (Vercel deploy hooks, CI) with the "30 webhooks → 1 build" optimization. - OutboundWebhookManager for parallel webhook triggering - Configurable via remote.webhooks.outbound array - HTTP method option (POST/GET, default POST) - transformPayload callback for custom payload shaping - Retry logic with exponential backoff (3 retries, 1s base delay) - Default payload includes items array with webhook details
Update dev and start command tests to reflect that UDL now uses its own default port from config rather than requiring explicit --port flag.
- Add unit tests for webhook processor, outbound, queue, and default handler - Add websocket server and client unit tests with coverage ignore markers - Add remote sync unit tests - Add node actions unit tests - Expand integration tests for loader - Add coverage ignore comments for unreachable code paths
Introduces CacheManager module that provides a central point for coordinating plugin cache updates. This enables webhook handlers and remote sync to persist changes to disk after store modifications. - registerPluginCache/initPluginCache for cache registration - savePluginCache for saving individual plugin caches - saveAffectedPlugins for batch webhook processing - replaceAllCaches for remote sync persistence
getConfig() now checks for UDL_ENDPOINT env var when config hasn't been explicitly initialized. This allows udl.query() client to automatically use the correct endpoint in child processes. - Export DEFAULT_UDL_PORT and UDL_ENDPOINT_ENV constants - Add isConfigInitialized() to check if config was explicitly set - Add resetConfig() for testing isolation
Enable instant webhook relay from production to local UDL instances via WebSocket, eliminating wait time for batch debounce. Server: - Add broadcastWebhookReceived() method - Add WebhookReceivedMessage type Client: - Add onWebhookReceived callback for instant webhook processing - Handle webhook:received message type - Skip node events when processing webhooks locally (avoid double processing) - Save plugin cache after node updates Export WebhookReceivedEvent interface for callback typing.
Wire up the instant webhook relay feature across remote sync, loader, and start-server for complete local-first UDL workflow. Remote sync: - Cache fetched nodes via replaceAllCaches() for offline support - Support onWebhookReceived callback for instant webhook processing Loader: - Use cache manager for plugin cache operations - Add isLocal option to skip sourceNodes (data from remote) - Initialize plugin caches even in local mode for persistence Start-server: - Set UDL_ENDPOINT env var for child processes - Load plugins in local mode for webhook handler registration - Process relayed webhooks locally using registered handlers - Save affected plugin caches after webhook batch processing - Wire up webhook:queued event to WebSocket broadcast
Emit webhook:queued event immediately when a webhook is queued, before the debounce timer. This enables instant relay to WebSocket subscribers without waiting for batch processing.
Add shared config utilities and update commands to read port from udl.config.ts and inject UDL_ENDPOINT env var to Next.js processes. New utils/config.ts: - loadPortFromConfig() reads port from udl.config.ts/.js/.mjs - resolveUdlPort() with priority: CLI > config > default - createNextEnv() builds env with UDL_ENDPOINT set Commands updated: - dev: resolve port from config, inject UDL_ENDPOINT to Next.js - build: resolve port from config, inject UDL_ENDPOINT to Next.js - start: resolve port from config, inject UDL_ENDPOINT to Next.js This allows udl.query() to automatically use the correct endpoint in Next.js code without manual configuration.
Core tests: - cache/manager.test.ts: cache manager registration and operations - config.test.ts: UDL_ENDPOINT env var, resetConfig, isConfigInitialized - sync/remote.test.ts: onWebhookReceived callback, cache persistence - webhooks/queue.test.ts: webhook:queued event emission - websocket/client.test.ts: webhook:received handling, onWebhookReceived - websocket/server.test.ts: broadcastWebhookReceived - loader.test.ts: isLocal option, cache manager integration Adapter-nextjs tests: - utils/config.test.ts: loadPortFromConfig, resolveUdlPort - dev.test.ts: config loading, UDL_ENDPOINT injection - build.test.ts: config loading, UDL_ENDPOINT injection - start.test.ts: config loading, UDL_ENDPOINT injection
Add `updateStrategy` config option that allows plugins to specify how incremental updates from webhooks should be handled: - `'webhook'` (default): Process webhook payload directly via handler - `'sync'`: Treat webhooks as notifications and re-run sourceNodes This enables plugins with native sync APIs (like Contentful) to reuse their existing sourceNodes logic for incremental updates, rather than maintaining separate webhook transformation code. Key changes: - Add UpdateStrategy type and config option to UDLConfig - Create PluginRegistry to store sourceNodes references - Modify webhook processor to group by strategy and call sourceNodes once per sync-strategy plugin (not once per webhook) - Update Contentful plugin to use updateStrategy: 'sync' - Warn if both updateStrategy: 'sync' and registerWebhookHandler set
Add comprehensive tests for the new updateStrategy feature: - Add registry.test.ts with 100% coverage for PluginRegistry class - Add tests for updateStrategy: 'sync' in loader.test.ts - Verifies sync plugins register default webhook handler - Verifies warning when both sync strategy and custom handler provided - Add tests for sync strategy in processor.test.ts - Tests invokeSourceNodes function (success, errors, missing plugin) - Tests deduplication (multiple webhooks -> one sync) - Tests mixed strategies (sync + webhook plugins in same batch) All target files now have 100% test coverage.
Update webhook URL pattern to properly extract scoped package names like @universal-data-layer/plugin-source-contentful. Add URL decoding to handle encoded @ symbols (%40) in webhook paths.
Restructure cache directory to use `.udl/` as a parent directory for all internal UDL files. This allows future expansion for storing other library internals (logs, state, etc.) while keeping a single directory to gitignore. - Update FileCacheStorage to use .udl/cache/nodes.json - Update FileSyncTokenStorage to use .udl/cache/contentful-sync-tokens.json - Update all gitignore, prettierignore, and eslint ignore patterns - Update documentation to reflect new paths - Update all related tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements Epic #57: UDL as Production Data Layer, adding all the infrastructure needed to run UDL in production environments with proper health checks, graceful shutdown, webhook support, and partial sync capabilities.
Features
Phase 1: Health & Lifecycle
/health,/ready) for container orchestration and load balancer integrationPhase 2: Webhook Infrastructure
registerWebhook()POST /_webhooks/:plugin/:pathPhase 3: Partial Sync
Related Issues
Closes #57, #58, #59, #60, #61, #62, #64
Type of Change
Breaking Changes
202 Acceptedinstead of200 OKExample Configuration
Checklist
Code Quality
npm run lintand fixed all issuesnpm run typecheckwith no errorsnpm run fixto format my codeTesting
npm run testand all tests passDocumentation & Changesets