Implement external cache storage for AI service scalability

[ErnieAtLYD]

Problem

Currently the AI reflection service uses in-memory Maps for caching and rate limiting, which doesn't scale horizontally and could cause memory leaks in high-traffic scenarios.

Solution

Migrate to external cache storage (Redis recommended) for production scalability.

Tasks

• Add Redis client configuration and connection handling
• Replace in-memory cache Map with Redis-backed caching
• Replace in-memory rate limiting with Redis-backed rate limiting
• Add Redis connection health checks
• Update environment configuration for Redis settings
• Add Redis to deployment documentation

Acceptance Criteria

• Multiple server instances can share cache and rate limiting state
• Memory usage remains constant under load
• Cache hit/miss ratios are maintained or improved
• Rate limiting works correctly across multiple instances

Priority

Medium - Important for production scale but current implementation works for initial deployment

Related

Addresses scalability concerns from PR #12 code review

[ErnieAtLYD]

@claude how would you implement this?

[claude]

Claude finished @ErnieAtLYD's task —— View job

---

Implementation Analysis for Redis Cache Migration

I've analyzed the current codebase and here's my comprehensive implementation approach:

Todo List

• Examine current AI service implementation
• Analyze existing cache and rate limiting patterns
• Design Redis integration architecture
• Provide step-by-step implementation plan
• Consider testing and deployment strategies

Current State Analysis

Key Finding: The AI service with in-memory caching and rate limiting does not exist yet. The Reflect app currently has:

• ✅ Complete frontend UI with journal entry functionality
• ✅ Infrastructure planning (port 11434 configured for Ollama service in src/config/ports.ts)
• ❌ No AI API routes (src/app/api/ directory doesn't exist)
• ❌ No caching or rate limiting implementations
• ❌ No AI service dependencies (OpenAI, Redis, etc.)

The main page has a TODO placeholder: // TODO: Implement reflection functionality

Recommended Redis Implementation Architecture

Since we're building from scratch, here's my approach for a scalable Redis-based AI service:

1. Project Structure

src/
├── app/
│   └── api/
│       └── reflect/
│           └── route.ts          # Main AI endpoint
├── lib/
│   ├── ai/
│   │   ├── client.ts             # AI service client (OpenAI/Ollama)
│   │   ├── prompts.ts            # Reflection prompts
│   │   └── types.ts              # AI response types
│   ├── cache/
│   │   ├── redis.ts              # Redis client & connection
│   │   ├── cache-service.ts      # High-level cache operations
│   │   └── rate-limiter.ts       # Redis-backed rate limiting
│   └── config/
│       ├── redis.ts              # Redis configuration
│       └── ai.ts                 # AI service configuration
└── types/
    └── api.ts                    # API response types

2. Redis Integration Design

Dependencies to Add:

{
  "dependencies": {
    "redis": "^4.6.0",
    "openai": "^4.0.0", // or ollama client
    "zod": "^3.22.0"    // for validation
  },
  "devDependencies": {
    "@types/redis": "^4.0.0"
  }
}

Redis Client Setup (src/lib/cache/redis.ts):

import { createClient, RedisClientType } from 'redis'

class RedisManager {
  private static instance: RedisManager
  private client: RedisClientType | null = null

  // Singleton pattern for connection management
  public static getInstance(): RedisManager {
    if (!RedisManager.instance) {
      RedisManager.instance = new RedisManager()
    }
    return RedisManager.instance
  }

  // Connection with health checks and reconnection logic
  async connect(): Promise<void> {
    this.client = createClient({
      url: process.env.REDIS_URL || 'redis://localhost:6379',
      retry_delay_ms: 100,
      max_retry_delay_ms: 5000
    })
    
    await this.client.connect()
  }

  // Graceful shutdown
  async disconnect(): Promise<void> {
    await this.client?.disconnect()
  }
}

3. Cache Strategy Design

Multi-Layer Caching:

• L1: Response caching (AI responses by prompt hash)
• L2: User session caching (recent entries, preferences)
• L3: System caching (prompt templates, configuration)

Cache Service (src/lib/cache/cache-service.ts):

export class CacheService {
  private redis: RedisClientType

  // Hash-based cache keys for consistent lookup
  private getCacheKey(type: 'response' | 'session' | 'config', identifier: string): string {
    return `reflect:${type}:${identifier}`
  }

  // TTL-based expiration strategy
  async setResponse(promptHash: string, response: AIResponse): Promise<void> {
    await this.redis.setEx(
      this.getCacheKey('response', promptHash),
      3600, // 1 hour TTL for AI responses
      JSON.stringify(response)
    )
  }

  // Hit/miss ratio tracking
  async getWithMetrics(key: string): Promise<{ data: any; hit: boolean }> {
    const data = await this.redis.get(key)
    const hit = data !== null
    
    // Track metrics for monitoring
    await this.redis.incr(`metrics:cache:${hit ? 'hits' : 'misses'}`)
    
    return { data: data ? JSON.parse(data) : null, hit }
  }
}

4. Rate Limiting Design

Redis-Backed Rate Limiter (src/lib/cache/rate-limiter.ts):

export class RateLimiter {
  // Sliding window rate limiting
  async checkLimit(
    userId: string, 
    windowMs: number = 60000, 
    maxRequests: number = 10
  ): Promise<{ allowed: boolean; remaining: number; resetTime: number }> {
    
    const key = `rate_limit:${userId}`
    const now = Date.now()
    const windowStart = now - windowMs

    // Remove old entries and count current
    await this.redis.zRemRangeByScore(key, 0, windowStart)
    const currentCount = await this.redis.zCard(key)

    if (currentCount >= maxRequests) {
      return { allowed: false, remaining: 0, resetTime: windowStart + windowMs }
    }

    // Add current request
    await this.redis.zAdd(key, { score: now, value: now.toString() })
    await this.redis.expire(key, Math.ceil(windowMs / 1000))

    return { 
      allowed: true, 
      remaining: maxRequests - currentCount - 1,
      resetTime: now + windowMs 
    }
  }
}

5. Environment Configuration

Environment Variables:

# Redis Configuration
REDIS_URL=redis://localhost:6379
REDIS_PASSWORD=optional_password
REDIS_TLS=false

# AI Service Configuration  
OPENAI_API_KEY=sk-...
OLLAMA_BASE_URL=http://localhost:11434

# Cache Configuration
CACHE_TTL_RESPONSES=3600
CACHE_TTL_SESSIONS=86400
RATE_LIMIT_WINDOW_MS=60000
RATE_LIMIT_MAX_REQUESTS=10

6. Health Checks & Monitoring

Health Check Endpoint (src/app/api/health/route.ts):

export async function GET() {
  const redis = RedisManager.getInstance()
  
  try {
    await redis.ping() // Redis connectivity check
    const metrics = await redis.getMetrics() // Cache hit/miss ratios
    
    return Response.json({
      status: 'healthy',
      redis: { connected: true },
      cache: { hitRatio: metrics.hitRatio }
    })
  } catch (error) {
    return Response.json({ status: 'unhealthy', error }, { status: 503 })
  }
}

Implementation Phases

Phase 1: Foundation Setup

1. Install Redis and AI service dependencies
2. Create Redis client with connection management
3. Add environment configuration
4. Implement health checks

Phase 2: Core Services

1. Build cache service with TTL management
2. Implement Redis-backed rate limiting
3. Create AI client wrapper
4. Add metrics collection

Phase 3: API Integration

1. Create /api/reflect endpoint
2. Integrate caching and rate limiting
3. Add request validation and error handling
4. Implement response formatting

Phase 4: Testing & Optimization

1. Unit tests for cache and rate limiting
2. Integration tests for API endpoints
3. Performance testing under load
4. Cache hit/miss ratio optimization

Key Benefits of This Architecture

✅ Horizontal Scalability: Multiple app instances share Redis state
✅ Memory Efficiency: External cache prevents memory leaks
✅ Production Ready: Health checks, metrics, and error handling
✅ Developer Friendly: Clear abstractions and TypeScript support
✅ Performance Optimized: Multi-layer caching with hit ratio tracking

This approach builds a production-ready AI service from the ground up with Redis as the backbone for scalability. The modular design allows for incremental implementation and easy testing.

Would you like me to implement any specific phase or component to get you started?

---