fix: Redis rate limiting, sync retry queue, migrate.ts, i18n HTTP backend, health checks

Redis infrastructure
- lib/redis.ts: singleton client, optional (warns in prod if not set), graceful close
- rateLimiter.ts: all 7 limiters now use shared Redis store via rate-limit-redis
  Falls back to in-memory with a production warning when REDIS_URL unset
- docker-compose.yml: Redis 7-alpine service, health check, redis_data volume
  App service depends_on redis, REDIS_URL=redis://redis:6379 passed
- index.ts: Redis connect at startup, closeRedis() in graceful shutdown
- .env.example: REDIS_URL documented with examples
- ioredis + rate-limit-redis added to api-server package.json

Federation sync retry queue (lib/syncRetryQueue.ts)
- Exponential backoff: 30s → 2m → 10m → 1h → 6h (capped), ±20% jitter
- Max 10 attempts per siteDomain:targetNode pair, then abandoned with warning
- Polls every 15 seconds, skips offline peers, cleans up deleted sites
- deploy.ts: failed/error syncs now enqueued instead of silently dropped
  X-Federation-From header added to outgoing syncs
- startSyncRetryQueue() / stopSyncRetryQueue() in index.ts lifecycle
- getSyncQueueStats() exposed in GET /api/health

Database migration runner (lib/db/src/migrate.ts)
- Reads SQL files from migrations/ in alphabetical order
- Tracks applied migrations in _migrations table
- Full transaction rollback on failure — never partial schema
- pnpm migrate script now points to migrate.ts via tsx/esm

i18n async loading
- Removed bundled en.json + id.json from JavaScript bundle
- i18next-http-backend fetches translations via HTTP on demand
- Translation files served from public/locales/{lng}/translation.json
- Added public/locales/en/translation.json + public/locales/id/translation.json
- i18next-http-backend added to frontend package.json

Health endpoint improvements (routes/health.ts)
- Redis status + latency added to services object
- domainCache stats (entry counts, TTL, max sizes)
- syncQueue stats (queued count, breakdown by target domain)

Documentation
- ROADMAP.md: 13 critical/high items updated from ⚠️/❌ to ✅
  Remaining gaps: ACME stub, audit log, content dedup, Prometheus
- HONEST_ASSESSMENT.md: resolved issues table added at bottom

Author: The No Hands Company

.env.example

@@ -150,3 +150,11 @@ DOMAIN_CACHE_TTL_MS=300000
 DOMAIN_CACHE_MAX=10000
 # Max file cache entries (default: 50000)
 FILE_CACHE_MAX=50000
+
+# ── Redis (strongly recommended for production) ───────────────────────────────
+# Without Redis, rate limiting is per-process only (broken in multi-instance).
+# Redis also enables future session sharing and cache invalidation pub/sub.
+# Format: redis://[:password@]host[:port][/db-number]
+# Example: redis://redis:6379  (Docker Compose default)
+# Example: redis://user:pass@managed-redis.example.com:6380
+REDIS_URL=redis://redis:6379

ROADMAP.md

@@ -25,16 +25,16 @@ A living document tracking what is built, what is in progress, and what must be
 | Replit Auth (OIDC) | ✅ | Browser flows work |
 | Ed25519 key pair generation + signing | ✅ | Correct implementation |
 | `/.well-known/federation` discovery | ✅ | |
-| Federation handshake + ping | ⚠️ | Replay attack window not enforced |
-| Node health monitor | ⚠️ | Marks offline after 1 failure — needs N=3 threshold |
-| Object storage (file upload/download) | ⚠️ | **Replit sidecar only — S3/MinIO non-functional** |
-| Site file serving (host-header routing) | ⚠️ | 2-3 DB queries per request, no caching |
+| Federation handshake + ping | ✅ | 5-minute timestamp window enforced |
+| Node health monitor | ✅ | N=3 consecutive failures required, exponential backoff |
+| Object storage (file upload/download) | ✅ | S3StorageProvider + ReplitStorageProvider, env-var selected |
+| Site file serving (host-header routing) | ✅ | LRU cache (10K domains, 50K files), invalidated on deploy |
 | Capacity tracking | ✅ | |
-| Rate limiting | ⚠️ | **In-memory only — broken in multi-instance** |
+| Rate limiting | ✅ | Redis-backed when REDIS_URL set; warns in prod if missing |
 | Structured logging + error handling | ✅ | Pino, AppError, stack traces redacted in prod |
 | Graceful shutdown | ✅ | |
-| DB connection pool | ⚠️ | No max/timeout config |
-| Database migrations | ❌ | **Zero migration files — only `db push`** |
+| DB connection pool | ✅ | Explicit max/min/timeout config, error handler |
+| Database migrations | ✅ | 0000_initial_schema.sql + migrate.ts runner |
 
 ---
 
@@ -51,7 +51,7 @@ A living document tracking what is built, what is in progress, and what must be
 | Onboarding flow | ✅ | |
 | Node Marketplace | ✅ | |
 | API Reference page | ✅ | |
-| Bahasa Indonesia i18n | ⚠️ | Translations complete but bundled synchronously |
+| Bahasa Indonesia i18n | ✅ | HTTP backend (i18next-http-backend), loaded on demand from /locales/ |
 | React lazy loading | ✅ | All 14 routes code-split |
 
 ---
@@ -63,7 +63,7 @@ A living document tracking what is built, what is in progress, and what must be
 | API tokens (Bearer auth) | ✅ | SHA-256 hashed |
 | Site team members (owner/editor/viewer) | ✅ | |
 | Site visibility (public/private) | ✅ | |
-| Password-protected sites | ⚠️ | **Cookie not server-verified — security gap** |
+| Password-protected sites | ✅ | HMAC-signed cookie, timingSafeEqual verified |
 | Custom domain CNAME+TXT verification | ✅ | |
 | Custom domain routing in host router | ✅ | Subject to caching gap above |
 
@@ -75,8 +75,8 @@ A living document tracking what is built, what is in progress, and what must be
 |---|---|---|
 | Site sync push (notify peers on deploy) | ✅ | Ed25519 signed |
 | Federation manifest endpoint | ✅ | Presigned URLs valid 1 hour |
-| Site sync pull (file replication) | ⚠️ | Works but no retry queue — failed syncs are lost |
-| Gossip-based peer discovery | ⚠️ | Works but no Redis sharing in multi-instance |
+| Site sync pull (file replication) | ✅ | Retry queue with exponential backoff (30s→2m→10m→1h→6h), max 10 attempts |
+| Gossip-based peer discovery | ⚠️ | Works; gossip peer list is in-memory per instance |
 | Same-domain conflict resolution | ✅ | First-write-wins + pubkey tiebreaker |
 | Bootstrap node registry | ✅ | |
 
@@ -86,11 +86,11 @@ A living document tracking what is built, what is in progress, and what must be
 
 | Feature | Status | Notes |
 |---|---|---|
-| Analytics buffer → hourly rollup | ⚠️ | **Bulk delete uses unsafe SQL — use inArray()** |
+| Analytics buffer → hourly rollup | ✅ | Uses inArray() — correct and safe |
 | Per-site analytics page | ✅ | |
 | Network-wide analytics | ✅ | |
-| Node operator admin dashboard | ⚠️ | **No RBAC — any authenticated user can access** |
-| Admin node settings | ⚠️ | No RBAC |
+| Node operator admin dashboard | ✅ | requireAdmin middleware, isAdmin DB flag + ADMIN_USER_IDS env var |
+| Admin node settings | ✅ | requireAdmin enforced |
 | Webhook notifications (Ed25519 signed) | ✅ | |
 
 ---
@@ -104,7 +104,7 @@ A living document tracking what is built, what is in progress, and what must be
 | GitHub Actions deploy workflow | ✅ | |
 | GitHub Actions CI (typecheck, lint, build) | ✅ | |
 | GitHub Actions npm publish workflow | ✅ | Needs `NPM_TOKEN` secret |
-| Docker Compose | ⚠️ | **App cannot talk to MinIO — storage abstraction broken** |
+| Docker Compose | ✅ | Redis + MinIO + S3StorageProvider wired, REDIS_URL passed to app |
 | Dockerfile (multi-stage) | ✅ | |
 
 ---
@@ -113,34 +113,36 @@ A living document tracking what is built, what is in progress, and what must be
 
 | Feature | Status | Notes |
 |---|---|---|
-| ACME/Let's Encrypt automation | ❌ | **Issues challenge token only — never gets cert** |
+| ACME/Let's Encrypt automation | ❌ | Stub — issues HTTP-01 token only; full ACME flow not implemented |
 | TLS via Caddy (documented) | ✅ | Caddy instruction accurate |
 | Geographic routing (closest-node redirect) | ✅ | Region inference + 302 redirect |
 | Geo routing: latency probing | ❌ | Mentioned in code comment, not implemented |
 
 ---
 
-## Must Fix Before Production (Priority Order)
+## Production Gaps Remaining
 
-| # | Issue | Severity | Est. Work |
+| # | Issue | Severity | Status |
 |---|---|---|---|
-| 1 | Rewrite objectStorage.ts with real S3 support | CRITICAL | 2–3 weeks |
-| 2 | Generate + commit Drizzle migrations | CRITICAL | 1 day |
-| 3 | Redis for rate limiting + session sharing | CRITICAL | 3–5 days |
-| 4 | Fix unlock cookie verification (HMAC-signed) | HIGH | 1 day |
-| 5 | Add admin RBAC (isAdmin flag) | HIGH | 2 days |
-| 6 | Host router LRU cache (domain → siteId) | HIGH | 2 days |
-| 7 | DB pool configuration (max, timeouts) | MEDIUM | 2 hours |
-| 8 | Session expiry cleanup job | MEDIUM | 2 hours |
-| 9 | Fix analytics bulk delete → `inArray()` | MEDIUM | 30 min |
-| 10 | Health monitor N=3 consecutive failure threshold | MEDIUM | 2 hours |
-| 11 | Replay attack: enforce 5-min timestamp window | MEDIUM | 2 hours |
-| 12 | Mark ACME as non-functional or implement it | MEDIUM | 2 weeks |
-| 13 | Admin audit logging | MEDIUM | 1 day |
-| 14 | Lazy-load i18n translations | LOW | 2 hours |
-| 15 | Federation sync retry queue | MEDIUM | 1 week |
-| 16 | Content deduplication for site files | LOW | 3 days |
-| 17 | Prometheus metrics endpoint | LOW | 1 day |
+| 1 | S3/MinIO object storage | CRITICAL | ✅ Fixed — S3StorageProvider, AWS SDK v3 |
+| 2 | Drizzle migrations | CRITICAL | ✅ Fixed — 0000_initial_schema.sql + migrate.ts |
+| 3 | Redis rate limiting | CRITICAL | ✅ Fixed — shared Redis store, falls back with warning |
+| 4 | Unlock cookie security | HIGH | ✅ Fixed — HMAC-signed, timingSafeEqual |
+| 5 | Admin RBAC | HIGH | ✅ Fixed — requireAdmin middleware, isAdmin flag |
+| 6 | Host router LRU cache | HIGH | ✅ Fixed — 10K domain + 50K file entries |
+| 7 | DB pool configuration | MEDIUM | ✅ Fixed — max/min/timeout/error handler |
+| 8 | Session expiry cleanup | MEDIUM | ✅ Fixed — 6-hour background job |
+| 9 | Analytics bulk delete | MEDIUM | ✅ Fixed — uses inArray() |
+| 10 | Health monitor threshold | MEDIUM | ✅ Fixed — N=3 consecutive failures |
+| 11 | Replay attack window | MEDIUM | ✅ Fixed — 5-minute timestamp check |
+| 12 | i18n async loading | LOW | ✅ Fixed — i18next-http-backend, HTTP-fetched |
+| 13 | Federation sync retry | MEDIUM | ✅ Fixed — exponential backoff queue, 10 max attempts |
+| 14 | ACME TLS automation | MEDIUM | ❌ Still a stub — use Caddy for now |
+| 15 | Admin audit logging | MEDIUM | 📋 Not yet built |
+| 16 | Content deduplication | LOW | 📋 Not yet built |
+| 17 | Prometheus metrics | LOW | 📋 Not yet built |
+| 18 | Gossip in-memory per-instance | LOW | ⚠️ Multi-instance gossip not Redis-shared |
+| 19 | Session store (multi-instance) | MEDIUM | ⚠️ PostgreSQL sessions work; not Redis-backed |
 
 ---
 

artifacts/api-server/package.json

@@ -23,9 +23,11 @@
     "express-slow-down": "^3.1.0",
     "google-auth-library": "^10.6.2",
     "helmet": "^8.1.0",
+    "ioredis": "^5.6.1",
     "openid-client": "^6.8.2",
     "pino": "^10.3.1",
     "pino-http": "^11.0.0",
+    "rate-limit-redis": "^4.2.0",
     "uuid": "^13.0.0"
   },
   "devDependencies": {
@@ -36,6 +38,7 @@
     "@types/node": "catalog:",
     "esbuild": "^0.27.3",
     "pino-pretty": "^13.1.3",
-    "tsx": "catalog:"
+    "tsx": "catalog:",
+    "@types/ioredis": "^4.28.10"
   }
 }

artifacts/api-server/src/index.ts

@@ -5,6 +5,8 @@ import { generateKeyPair } from "./lib/federation";
 import { startHealthMonitor } from "./lib/healthMonitor";
 import { startAnalyticsFlusher, stopAnalyticsFlusher } from "./lib/analyticsFlush";
 import { startGossipPusher, stopGossipPusher } from "./routes/gossip";
+import { getRedisClient, closeRedis } from "./lib/redis";
+import { startSyncRetryQueue, stopSyncRetryQueue } from "./lib/syncRetryQueue";
 import { db, sessionsTable } from "@workspace/db";
 import { lt } from "drizzle-orm";
 import { seedBundledSites } from "./lib/seedBundledSites";
@@ -60,6 +62,8 @@ function gracefulShutdown(server: http.Server, signal: string): void {
     try {
       stopAnalyticsFlusher();
       stopGossipPusher();
+      stopSyncRetryQueue();
+      await closeRedis();
       const { pool } = await import("@workspace/db");
       await pool.end();
       logger.info("Database pool closed");
@@ -92,6 +96,13 @@ ensureLocalNode()
     startHealthMonitor();
     startAnalyticsFlusher();
     startGossipPusher();
+    startSyncRetryQueue();
+
+    // Initialise Redis connection (optional — falls back to in-memory if not configured)
+    const redis = getRedisClient();
+    if (redis) {
+      await redis.connect().catch(() => {}); // errors handled via 'error' event
+    }
 
     // Session expiry cleanup — purge expired sessions every 6 hours
     // Prevents unbounded growth of the sessions table

artifacts/api-server/src/lib/redis.ts

@@ -0,0 +1,61 @@
+/**
+ * Redis client singleton.
+ *
+ * Used by:
+ *   - Rate limiter store (express-rate-limit + rate-limit-redis)
+ *   - Future: session store for multi-instance deployments
+ *   - Future: pub/sub for cache invalidation signals
+ *
+ * Connection is optional — if REDIS_URL is not set, the app runs in
+ * single-instance mode with in-memory rate limiting. A warning is logged.
+ *
+ * In production with multiple API server instances, REDIS_URL MUST be set
+ * or rate limiting will be bypassed (each instance has its own counter).
+ */
+
+import { Redis } from "ioredis";
+import logger from "./logger";
+
+let redisClient: Redis | null = null;
+
+export function getRedisClient(): Redis | null {
+  if (redisClient) return redisClient;
+
+  const url = process.env.REDIS_URL;
+  if (!url) {
+    return null;
+  }
+
+  try {
+    redisClient = new Redis(url, {
+      maxRetriesPerRequest: 3,
+      connectTimeout: 5_000,
+      lazyConnect: true,
+      enableReadyCheck: true,
+    });
+
+    redisClient.on("connect", () => {
+      logger.info("[redis] Connected");
+    });
+
+    redisClient.on("error", (err) => {
+      logger.warn({ err: err.message }, "[redis] Connection error");
+    });
+
+    redisClient.on("close", () => {
+      logger.warn("[redis] Connection closed");
+    });
+
+    return redisClient;
+  } catch (err) {
+    logger.error({ err }, "[redis] Failed to create client");
+    return null;
+  }
+}
+
+export async function closeRedis(): Promise<void> {
+  if (redisClient) {
+    await redisClient.quit();
+    redisClient = null;
+  }
+}