Synchronize state between multiple replicas of Gateway Controller

[VirajSalaka]

Problem

Gateway Controller needs to run multiple replicas for the sake of high availability and all those replicas should have the same XDS cache state populated.

Solution

Abstract

Introduce new tables for maintaining states and events for each category. The data won't be linked to any available data via foreign keys. When api deploy request is received, gateway controller updates the tables and process ends. There is a separate go routine running in each gateway controller which polls on the states table. And if what received from state table for APIs is different compared to what it already knows, then it would do a lookup on Events table based on timestamp.

Key Characteristics

• Event-Driven: Changes are recorded as events and propagated to other instances
• Eventually# Consistent: All instances converge to the same state given time
• Database-Backed: Uses shared database (SQLite/PostgreSQL) as source of truth
• Poll-Based: Instances periodically check for state changes (no pub/sub required)
• Multi-Entity Support: Handles APIs, Certificates, and LLM Provider Templates
• Multi-Tenancy Ready: Organization-based isolation built in

DDL

-- States table: tracks version for API entity type
CREATE TABLE IF NOT EXISTS entity_states (
    entity_type TEXT NOT NULL PRIMARY KEY,  -- 'API'
    version_id TEXT NOT NULL,               -- UUID updated on every change
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

-- API Events table
CREATE TABLE IF NOT EXISTS api_events (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    processed_timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    originated_timestamp TIMESTAMP NOT NULL,
    action TEXT NOT NULL CHECK(action IN ('CREATE', 'UPDATE', 'DELETE')),
    entity_id TEXT NOT NULL,
    event_data TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_api_events_lookup ON api_events(processed_timestamp);

Synchronization Flow

Write Path (Instance A modifies an API)

User/API Request → REST API Handler
                        ↓
                 SyncAwareStorage.UpdateConfig()
                        ↓
            ┌───────────┴───────────┐
            │   BEGIN TRANSACTION   │
            └───────────┬───────────┘
                        ↓
            ┌───────────┴───────────┐
            │ Underlying Storage    │
            │ (save to apis table)  │
            └───────────┬───────────┘
                        ↓
            ┌───────────┴───────────┐
            │ EventStore.RecordEvent│
            │ (save to api_events)  │
            └───────────┬───────────┘
                        ↓
            ┌───────────┴───────────┐
            │ StateManager.UpdateState│
            │ (new UUID in entity_states)│
            └───────────┬───────────┘
                        ↓
            ┌───────────┴───────────┐
            │   COMMIT TRANSACTION  │
            └───────────────────────┘

Read Path (Instance A/B detects and applies change)

SyncPoller (every 5s + jitter)
            ↓
    Poll entity_states
            ↓
    ┌───────┴────────┐
    │ Version Match? │
    └───────┬────────┘
            │
         NO │ (version mismatch detected)
            ↓
    GetEventsSince(lastApplied)
            ↓
    ┌───────┴────────┐
    │ Events Found?  │
    └───────┬────────┘
            │
        YES │
            ↓
    ProcessAPIEvents(events)
            ↓
    ┌───────┴────────────┐
    │ For each event:    │
    │ - Deserialize JSON │
    │ - ConfigStore.Add/ │
    │   Update/Delete    │
    └───────┬────────────┘
            ↓
    SnapshotManager.UpdateSnapshot()
            ↓
    Update knownVersions & lastApplied

Cleanup Path

Hourly Timer
      ↓
CleanupOldEvents(time.Now() - 24h)
      ↓
DELETE FROM api_events WHERE processed_timestamp < cutoff
DELETE FROM certificate_events WHERE processed_timestamp < cutoff
DELETE FROM llm_template_events WHERE processed_timestamp < cutoff

Alternative Approaches

• Broadcast to others using headless services (k8s)/ dnsrr (docker-compose)

Issue here is this becomes heavily dependent on network.

• Bring in a message broker

Then we increase the number of items required for running api-platform.