Merge pull request #178 from menloresearch/feat/170/add-redis-support-models-loading

Add caching with backward compatibility for UserService and Model Registry

Author: locnguyen1986

apps/jan-api-gateway/LOCAL_DEV_SETUP.md

@@ -18,7 +18,7 @@ jan-api-gateway/
 │   ├── launch.json                 # Debug and launch configurations
 │   └── tasks.json                  # Automated tasks (database management)
 ├── docker/                         # Docker configuration
-│   ├── docker-compose.yml         # PostgreSQL service configuration
+│   ├── docker-compose.yml         # PostgreSQL and Valkey cache service configuration
 │   └── init.sql                   # Database initialization script
 ├── application/                    # Go application code
 │   ├── cmd/server/                # Main server entry point
@@ -41,7 +41,8 @@ jan-api-gateway/
 1. **Press `F5`** or **Run → Start Debugging**
 2. **Select "Launch Jan API Gateway (Debug)"** from the dropdown
 3. **Wait for automatic setup:**
-   - Database starts automatically
+   - PostgreSQL database starts automatically
+   - Valkey cache service starts automatically (Redis-compatible)
    - Environment variables are set
    - Application launches with debugger attached
 
@@ -52,7 +53,7 @@ That's it! Your development environment is ready. 🎉
 ### 1. **Launch Jan API Gateway (Debug)** ⭐ *Recommended*
 - **Purpose**: Full development environment with debugging
 - **What it does**:
-  - Automatically starts PostgreSQL database
+  - Automatically starts PostgreSQL database and Valkey cache (Redis-compatible)
   - Sets all required environment variables
   - Launches the application with debugger attached
   - Opens integrated terminal for logs
@@ -68,7 +69,7 @@ That's it! Your development environment is ready. 🎉
 ### 3. **Launch Tests**
 - **Purpose**: Debug unit tests
 - **What it does**:
-  - Starts database for testing
+  - Starts database and Valkey cache for testing
   - Runs tests with debugging enabled
   - Allows setting breakpoints in test code
 - **When to use**: Debugging test failures or test logic
@@ -112,29 +113,34 @@ While the launch configurations handle the database automatically, you can also
 1. **Press `Ctrl+Shift+P` (Windows/Linux) or `Cmd+Shift+P` (macOS)**
 2. **Type "Tasks: Run Task"**
 3. **Select one of:**
-   - **Start Database** - Start PostgreSQL
-   - **Stop Database** - Stop PostgreSQL
+   - **Start Database** - Start PostgreSQL and Valkey cache
+   - **Stop Database** - Stop PostgreSQL and Valkey cache
    - **Wait for Database** - Check if database is ready
+   - **Wait for Cache** - Check if Valkey cache is ready
    - **Build Application** - Build the Go application
    - **Run Tests** - Run all tests
 
 ### Using Terminal
 ```bash
-# Start database
-docker-compose -f docker/docker-compose.yml up -d postgres
+# Start database and Valkey cache (primary cache service)
+docker-compose -f docker/docker-compose.yml up -d postgres valkey
 
-# Stop database
+# Stop all services
 docker-compose -f docker/docker-compose.yml down
 
-# Reset database (removes all data)
+# Reset database and Valkey cache (removes all data)
 docker-compose -f docker/docker-compose.yml down -v
-docker-compose -f docker/docker-compose.yml up -d postgres
+docker-compose -f docker/docker-compose.yml up -d postgres valkey
 
 # View logs
 docker-compose -f docker/docker-compose.yml logs postgres
+docker-compose -f docker/docker-compose.yml logs valkey
 
 # Connect to database
 docker-compose -f docker/docker-compose.yml exec postgres psql -U jan_user -d jan_api_gateway
+
+# Connect to Valkey cache
+docker-compose -f docker/docker-compose.yml exec valkey valkey-cli
 ```
 
 ## ⚙️ Environment Variables
@@ -153,16 +159,26 @@ The following environment variables are **automatically configured** in the laun
 | `OAUTH2_GOOGLE_CLIENT_ID` | Google OAuth2 client ID | `your-google-client-id` |
 | `OAUTH2_GOOGLE_CLIENT_SECRET` | Google OAuth2 client secret | `your-google-client-secret` |
 | `OAUTH2_GOOGLE_REDIRECT_URL` | Google OAuth2 redirect URL | `http://localhost:8080/auth/google/callback` |
+| `REDIS_URL` | Redis connection URL | `redis://localhost:6379` |
+| `REDIS_PASSWORD` | Redis authentication password | `` (empty for dev) |
+| `REDIS_DB` | Redis database number | `0` |
+
+**📝 Redis Cache Notes:**
+- **Redis** is used for caching inference models and improving performance
+- Cache keys are automatically managed by the application
+- Redis connection is required for optimal performance
 
 **Note**: You can modify these values in `.vscode/launch.json` if needed for your environment.
 
 ## 🐛 Troubleshooting
 
-### Database Connection Issues
+### Database & Redis Connection Issues
 1. **Check Docker**: Ensure Docker Desktop is running
-2. **Check Port**: Make sure port 5432 is available
+2. **Check Ports**: Make sure ports 5432 (PostgreSQL) and 6379 (Redis) are available
 3. **View Database Status**: Use Command Palette → "Tasks: Run Task" → "Wait for Database"
-4. **View Logs**: Check the integrated terminal for database startup logs
+4. **View Redis Status**: Use Command Palette → "Tasks: Run Task" → "Wait for Redis"
+5. **View Logs**: Check the integrated terminal for database and Redis startup logs
+6. **Redis Connection**: Ensure Redis is running and accessible on the configured port
 
 ### Go Extension Issues
 1. **Install Go Extension**: VS Code/Cursor should prompt you automatically

apps/jan-api-gateway/README.md

@@ -11,6 +11,7 @@ A comprehensive API gateway for Jan Server that provides OpenAI-compatible endpo
 - **Authentication & Authorization**: JWT-based auth with Google OAuth2 integration and role-based access control
 - **API Key Management**: Secure API key generation and management at organization and project levels with multiple key types (admin, project, organization, service, ephemeral)
 - **Model Registry**: Dynamic model endpoint management with automatic health checking and service discovery
+- **Cache Service**: High-performance caching for inference models using Redis to reduce load times and improve response performance
 - **Streaming Support**: Real-time streaming responses with Server-Sent Events (SSE) and chunked transfer encoding
 - **MCP Integration**: Model Context Protocol support for external tools and resources with JSON-RPC 2.0
 - **Web Search**: Serper API integration for web search capabilities via MCP with webpage fetching
@@ -78,6 +79,7 @@ A comprehensive API gateway for Jan Server that provides OpenAI-compatible endpo
   - Automatic migrations with Atlas
   - Generated query interfaces with GORM Gen
 - **Authentication**: JWT v5.3.0 + Google OAuth2 v3.15.0
+- **Caching**: Redis v9.14.0 for high-performance model caching
 - **API Documentation**: Swagger/OpenAPI v1.16.6
 - **Streaming**: Server-Sent Events (SSE) with chunked transfer
 - **Dependency Injection**: Google Wire v0.6.0
@@ -228,6 +230,45 @@ A comprehensive API gateway for Jan Server that provides OpenAI-compatible endpo
 | `SMTP_PASSWORD` | SMTP password | `your-smtp-password` |
 | `SMTP_SENDER_EMAIL` | Default sender email address | `noreply@yourdomain.com` |
 | `INVITE_REDIRECT_URL` | Redirect URL for invitation acceptance | `http://localhost:8080/invite/accept` |
+| `REDIS_URL` | Redis connection URL | `redis://localhost:6379` |
+| `REDIS_PASSWORD` | Redis authentication password | `` (empty for dev) |
+| `REDIS_DB` | Redis database number | `0` |
+
+## 🚀 Redis Caching
+
+The Jan API Gateway includes Redis caching for inference models to significantly improve performance by avoiding repeated model loading and caching identical requests.
+
+### Redis Features
+- **Model List Caching**: Cache model discovery for 10 minutes
+- **Transparent Integration**: No code changes needed in existing handlers
+- **Centralized Constants**: Redis cache keys defined as constants
+
+### Quick Setup
+
+1. **Deploy Redis Infrastructure**:
+   ```bash
+   helm dependency update charts/umbrella-chart/
+   helm install jan-server charts/umbrella-chart/
+   ```
+
+2. **Environment Variables**:
+   ```bash
+   REDIS_URL=redis://jan-server-redis-master:6379
+   REDIS_PASSWORD=""  # Empty for dev
+   REDIS_DB=0
+   ```
+
+3. **Verify Setup**:
+   ```bash
+   # Check Redis connectivity in logs
+   kubectl logs deployment/jan-server-jan-api-gateway | grep "Successfully connected to Redis"
+   ```
+
+### Performance Benefits
+- **Reduced latency** for model discovery calls
+- **Reduced CPU usage** by avoiding repeated model loading
+- **Better scalability** with reduced backend load
+- **Improved user experience** with faster response times
 
 ## 📚 API Usage Examples
 

apps/jan-api-gateway/application/app/domain/cron/cron_service.go

@@ -0,0 +1,34 @@
+package cron
+
+import (
+	"context"
+
+	"github.com/mileusna/crontab"
+	inference_model_registry "menlo.ai/jan-api-gateway/app/domain/inference_model_registry"
+	janinference "menlo.ai/jan-api-gateway/app/utils/httpclients/jan_inference"
+	"menlo.ai/jan-api-gateway/config/environment_variables"
+)
+
+type CronService struct {
+	JanInferenceClient     *janinference.JanInferenceClient
+	InferenceModelRegistry *inference_model_registry.InferenceModelRegistry
+}
+
+func NewService(janInferenceClient *janinference.JanInferenceClient, registry *inference_model_registry.InferenceModelRegistry) *CronService {
+	return &CronService{
+		JanInferenceClient:     janInferenceClient,
+		InferenceModelRegistry: registry,
+	}
+}
+
+func (cs *CronService) Start(ctx context.Context, ctab *crontab.Crontab) {
+	// Run initial check
+	cs.InferenceModelRegistry.CheckInferenceModels(ctx)
+
+	ctab.AddJob("* * * * *", func() {
+		cs.InferenceModelRegistry.CheckInferenceModels(ctx)
+
+		// Reload environment variables
+		environment_variables.EnvironmentVariables.LoadFromEnv()
+	})
+}