Skip to content

RATE_LIMITING_SOLUTION.md

Latest commit

 

History

History
188 lines (143 loc) · 6.83 KB

RATE_LIMITING_SOLUTION.md

File metadata and controls

188 lines (143 loc) · 6.83 KB
Raw

Rate Limiting Solution for OpenAI API 429 Errors

Problem

You were experiencing HTTP 429 "Too Many Requests" errors from OpenAI's API in production, indicating that your application was hitting OpenAI's rate limits.

Solution Implemented

1. Centralized Rate Limiting System

File: src/rate_limiter.py

Created a comprehensive rate limiting utility with:

  • Exponential backoff with jitter to prevent thundering herd effects
  • Smart retry logic that distinguishes between rate limits and quota errors
  • Production-optimized delays with configurable parameters
  • Automatic error handling for quota exceeded scenarios
  • Built-in logging for monitoring and debugging

Key Features:

  • APIRateLimiter class for centralized rate limit management
  • Convenience functions like make_openai_chat_completion() and make_openai_vision_request()
  • Decorator @rate_limited for easy function wrapping
  • Global rate limiter instance for consistent behavior

2. Production-Optimized Configuration

File: src/config.py

Added production-specific rate limiting configuration:

# Production settings (conservative for 429 prevention)
API_MAX_RETRIES = 5          # More retries for production
API_BASE_DELAY = 2.0         # Start with 2 second delay
API_MAX_DELAY = 60.0         # Max 60 seconds between retries
API_EXPONENTIAL_BASE = 2.5   # Aggressive backoff
API_JITTER_RANGE = 1.0       # Add up to 1 second jitter
API_REQUEST_DELAY = 0.5      # Delay between requests in parallel

3. Updated All OpenAI API Calls

Files: src/file_processors/*.py

Replaced all direct OpenAI API calls with rate-limited versions:

OpenAI Processor (openai_processor.py)

  • All Vision API calls now use make_openai_vision_request()
  • Text processing calls use make_openai_chat_completion()
  • Resume generation calls are rate-limited

Image Processor (image_processor.py)

  • Vision API calls for image analysis use centralized rate limiting
  • Automatic fallback to Gemini on quota errors

Translation Processor (translation_processor.py)

  • Replaced custom retry logic with centralized system
  • Simplified error handling while maintaining robustness

4. Enhanced Parallel Processing Controls

Files: src/pdf_chunker.py, src/file_processors/openai_processor.py

PDF Chunker

  • Added staggered delays between parallel chunk processing
  • Progressive delays: chunk 0 = no delay, chunk 1 = 0.5s, chunk 2 = 1.0s, etc.

OpenAI Processor

  • Production mode uses sequential processing (max_workers = 1) for maximum stability
  • Reduced parallel workers to prevent overwhelming the API

5. Smart Error Handling

Quota vs Rate Limit Detection

The system now distinguishes between:

  • Rate limits (429): Retryable with exponential backoff
  • Quota exceeded: Non-retryable, requires manual intervention

Comprehensive Logging

  • Clear error messages with troubleshooting steps
  • Success/retry attempt logging
  • Production-friendly log levels

Rate Limiting Strategy

Exponential Backoff Formula

delay = base_delay * (exponential_base ^ attempt) + jitter

Production example:

  • Attempt 1: 2.0s + jitter
  • Attempt 2: 5.0s + jitter
  • Attempt 3: 12.5s + jitter
  • Attempt 4: 31.25s + jitter
  • Attempt 5: 60s (capped at max_delay)

Parallel Processing Throttling

  1. Sequential processing in production (max_workers = 1)
  2. Staggered delays between concurrent requests
  3. Conservative timeouts for Vision API calls

Deployment Instructions

1. No Environment Variable Changes Required

The solution automatically detects production environment using existing ENVIRONMENT=production variable.

2. Optional Configuration Override

You can fine-tune rate limiting by setting these environment variables:

# Optional: Override default production settings
API_MAX_RETRIES=5
API_BASE_DELAY=2.0
API_MAX_DELAY=60.0
API_EXPONENTIAL_BASE=2.5
API_JITTER_RANGE=1.0
API_REQUEST_DELAY=0.5

3. Monitoring

The system provides comprehensive logging. Monitor for:

  • ⚠️ OpenAI rate limit error - Normal retries happening
  • ✅ API call succeeded on attempt X - Successful retries
  • ❌ OpenAI Quota Error - Requires billing/quota attention

Benefits

Immediate Benefits

  1. Eliminates 429 errors through intelligent retry logic
  2. Maintains throughput while respecting rate limits
  3. Automatic recovery from temporary rate limit issues

Production Stability

  1. Conservative settings prevent overwhelming the API
  2. Sequential processing in production for maximum reliability
  3. Graceful degradation with meaningful error messages

Monitoring & Debugging

  1. Clear logging for tracking API usage patterns
  2. Quota error detection with troubleshooting guidance
  3. Retry attempt tracking for performance optimization

Testing Recommendations

1. Load Testing

Test the system with multiple concurrent requests to verify rate limiting works correctly.

2. Monitor API Usage

Use OpenAI's dashboard to monitor your API usage patterns after deployment.

3. Log Analysis

Monitor application logs for rate limiting patterns and adjust configuration if needed.

Troubleshooting

If 429 Errors Still Occur

  1. Increase base delay: Set API_BASE_DELAY=3.0 or higher
  2. Reduce parallel processing: The system already uses sequential processing in production
  3. Check quota limits: Monitor OpenAI billing dashboard

If Processing Seems Slow

  1. Verify environment detection: Ensure ENVIRONMENT=production is set correctly
  2. Adjust retry settings: Reduce API_MAX_RETRIES if needed
  3. Monitor actual delays: Check logs for retry patterns

Quota Exceeded Errors

The system will log specific troubleshooting steps:

  1. Check billing at OpenAI dashboard
  2. Verify monthly usage limits
  3. Ensure payment method is valid
  4. Consider regenerating API key

Files Modified

  1. NEW: src/rate_limiter.py - Centralized rate limiting system
  2. UPDATED: src/config.py - Added rate limiting configuration
  3. UPDATED: src/file_processors/openai_processor.py - Applied rate limiting to all API calls
  4. UPDATED: src/file_processors/image_processor.py - Applied rate limiting to Vision API
  5. UPDATED: src/file_processors/translation_processor.py - Replaced custom retry logic
  6. UPDATED: src/pdf_chunker.py - Added parallel processing delays

Summary

This comprehensive rate limiting solution addresses your 429 errors by:

  • Implementing industry-standard exponential backoff with jitter
  • Using production-optimized conservative settings
  • Providing centralized, consistent rate limiting across all OpenAI API calls
  • Maintaining application performance while respecting API limits
  • Adding comprehensive monitoring and debugging capabilities

The solution is ready for immediate deployment with no additional configuration required.