Enhancement: Migrate from OpenAI SDK to LangGraph for Universal AI Model Support

[adawalli]

Enhancement: Migrate from OpenAI SDK to LangGraph for Universal AI Model Support

Problem Statement

The current AI integration architecture, while functional, has several limitations that constrain the project's flexibility and future growth:

1. Limited Provider Support: Currently supports only 3 providers (OpenAI, DeepSeek, Ollama) with custom backend implementations
2. Manual Provider Integration: Each new AI provider requires custom backend class implementation and maintenance
3. OpenRouter Access Gap: No direct access to OpenRouter's extensive catalog of 200+ AI models from various providers
4. Scalability Concerns: Adding new providers requires significant development effort and ongoing maintenance
5. Feature Limitations: Missing advanced AI workflow capabilities like multi-agent coordination, complex reasoning chains, and cyclical workflows

Proposed Solution: Migration to LangGraph

What is LangGraph?

LangGraph is a sophisticated orchestration framework designed for building complex AI workflows with multi-provider support, visual debugging, and advanced state management capabilities.

Key Benefits

🌐 Universal AI Model Access

• OpenRouter Integration: Direct access to 200+ models from providers like Anthropic, Google, Meta, Cohere, Mistral, and more through a single API
• Provider Agnostic: No need to implement custom backends for each provider
• Future-Proof: Automatic support for new models as they become available on supported platforms

🔧 Enhanced Architecture

• Visual Workflow Management: LangGraph Studio provides intuitive debugging and workflow visualization
• Complex State Management: Built-in support for cyclical workflows and interdependent operations
• Robust Error Handling: Enterprise-grade retry logic and failure recovery mechanisms

💰 Cost & Performance Benefits

• Model Selection Flexibility: Choose optimal models based on cost, speed, and quality requirements
• Transparent Pricing: OpenRouter's pay-per-token model with clear cost visibility
• Performance Optimization: Ability to route requests to fastest/cheapest appropriate models

🚀 Developer Experience

• LangChain Ecosystem: Access to extensive tooling, documentation, and community resources
• Standardized Patterns: Industry-standard approaches for AI workflow development
• Reduced Maintenance: Less custom code to maintain as providers are handled by the framework

Implementation Benefits for AI Marketplace Monitor

Current Architecture Challenges

# Current: Custom backend for each provider
class OpenAIBackend(AIBackend): ...
class DeepSeekBackend(AIBackend): ...
class OllamaBackend(AIBackend): ...

LangGraph Architecture Benefits

# Future: Universal model access through LangGraph
from langgraph import StateGraph
from langchain_openai import ChatOpenAI
from langchain_community.chat_models import ChatOpenRouter

# Access ANY model through unified interface
model = ChatOpenRouter(model="anthropic/claude-3-sonnet") 
# or model = ChatOpenRouter(model="google/gemini-pro")
# or model = ChatOpenRouter(model="meta-llama/llama-2-70b-chat")

Specific Use Cases Enabled

1. A/B Testing: Compare results across different models for listing evaluation
2. Cost Optimization: Route simple evaluations to cheaper models, complex ones to premium models
3. Specialized Models: Use coding-focused models for technical listings, general models for others
4. Fallback Strategies: Automatic failover between providers for reliability
5. Multi-Agent Workflows: Future expansion to multiple AI agents for different evaluation aspects

Migration Strategy

Phase 1: Foundation

• Add LangGraph and LangChain dependencies
• Implement LangGraph-based backend alongside existing system
• Configure OpenRouter integration for model access

Phase 2: Feature Parity

• Migrate core evaluation functionality to LangGraph
• Implement configuration system for model selection
• Add comprehensive testing for new backend

Phase 3: Enhanced Features

• Implement multi-model evaluation strategies
• Add cost optimization routing
• Enable advanced workflow capabilities

Phase 4: Deprecation

• Remove legacy backend implementations
• Clean up configuration system
• Update documentation

Technical Considerations

Backward Compatibility

• Configuration format can remain largely unchanged
• Existing API keys and settings will be preserved
• Gradual migration path ensures no disruption

Performance Impact

• LangGraph adds minimal overhead for simple use cases
• Complex workflows benefit from optimized execution paths
• Caching and retry mechanisms are enhanced

Dependencies

• LangGraph is actively maintained with regular updates
• Strong community support and comprehensive documentation
• Minimal additional dependencies beyond current requirements

Expected Outcomes

1. Model Access: Support for 200+ AI models through OpenRouter
2. Reduced Maintenance: Elimination of custom provider backends
3. Enhanced Reliability: Built-in retry logic and error handling
4. Future Extensibility: Platform for advanced AI workflow features
5. Developer Productivity: Faster implementation of new AI features

References

• LangGraph Documentation
• OpenRouter Model Catalog
• LangChain Community Resources

This migration represents a strategic investment in the project's AI capabilities, positioning it for future growth while immediately solving current limitations around model access and provider management.

Priority: High - This enhancement addresses fundamental architectural limitations
Effort: Medium - Phased migration approach minimizes risk and disruption
Impact: High - Enables access to entire AI model ecosystem through single integration

[BoPeng]

Please include also plan for UI/config implementation. Current approach is documented here.

[adawalli]

UI/Configuration Implementation Plan

Based on the current configuration system documented in docs/README.md, here's the implementation plan for maintaining backward compatibility while enabling LangGraph's universal model access:

Current AI Configuration Structure

The existing system uses TOML configuration sections like:

[ai.openai]
provider = "OpenAI"  # Optional, defaults to section name
api_key = "sk-proj-xxx"
base_url = "https://api.openai.com/v1"  # Optional, has defaults
model = "gpt-4"  # Optional, has defaults
max_retries = 10
timeout = 30

Proposed LangGraph Configuration (Backward Compatible)

Phase 1: Enhanced AI Section with Model Selection

[ai.openrouter_claude]
provider = "openrouter"  # New provider type
api_key = "${OPENROUTER_API_KEY}"
model = "anthropic/claude-3-sonnet"  # Full OpenRouter model ID
max_retries = 10
timeout = 30

[ai.openrouter_llama]
provider = "openrouter"
api_key = "${OPENROUTER_API_KEY}" 
model = "meta-llama/llama-3.2-90b-vision-instruct"

[ai.cost_optimizer]
provider = "langgraph"  # Advanced LangGraph provider
api_key = "${OPENROUTER_API_KEY}"
model = "auto"  # Enable intelligent model routing
cost_strategy = "cheapest_first"  # Options: cheapest_first, balanced, performance_first
fallback_models = ["anthropic/claude-3-haiku", "openai/gpt-4o-mini"]

Phase 2: Advanced Workflow Configuration

[ai.evaluation_workflow]
provider = "langgraph"
api_key = "${OPENROUTER_API_KEY}"
workflow_type = "multi_agent"  # New capability
primary_model = "anthropic/claude-3-sonnet"
secondary_model = "openai/gpt-4"
consensus_required = true  # Require agreement between models

Implementation Strategy

1. Configuration Schema Updates

• Maintain Existing: All current [ai.name] sections continue working unchanged
• Add New Provider Types: openrouter, langgraph alongside existing openai, deepseek, ollama
• Extend BaseConfig: Add new fields for LangGraph-specific options with sensible defaults

2. Backend Architecture Evolution

# Current: src/ai_marketplace_monitor/config.py:23-27
supported_ai_backends = {
    "openai": OpenAIBackend,
    "deepseek": DeepSeekBackend, 
    "ollama": OllamaBackend,
    # NEW:
    "openrouter": OpenRouterBackend,
    "langgraph": LangGraphBackend,
}

3. UI Configuration Management

• Config Validation: Enhanced validation in Config.get_ai_config() for new provider types
• Model Discovery: Auto-populate available models from OpenRouter API for UI dropdowns
• Cost Estimation: Display real-time cost estimates for model selections
• Preset Configurations: Provide common model combinations (cost-effective, balanced, premium)

4. Migration Path for Users

Immediate (No Changes Required):

[ai.openai]  # Existing config works unchanged
api_key = "sk-proj-xxx"

Enhanced (New Capabilities):

[ai.claude_sonnet]
provider = "openrouter" 
api_key = "${OPENROUTER_API_KEY}"
model = "anthropic/claude-3-sonnet"

# Use both for A/B testing
ai = ["openai", "claude_sonnet"]  # Existing item-level AI selection

UI Components Impact

1. Configuration Interface

• Provider Selection: Dropdown with OpenAI, DeepSeek, Ollama, OpenRouter, LangGraph
• Model Browser: Searchable model catalog with filtering by cost/performance
• Cost Calculator: Real-time pricing estimates based on expected usage
• Template Library: Pre-configured setups for common use cases

2. Runtime Interface

• Model Performance Metrics: Show response times, costs, success rates per model
• A/B Comparison Views: Side-by-side evaluation results from different models
• Cost Tracking: Running totals and budget alerts

3. Migration Wizard

• One-click Upgrades: Convert existing OpenAI configs to equivalent OpenRouter configs
• Recommendation Engine: Suggest optimal model combinations based on usage patterns
• Test Mode: Try new models alongside existing ones before full migration

Backward Compatibility Guarantees

1. Existing Configurations: All current [ai.openai], [ai.deepseek], [ai.ollama] sections work unchanged
2. API Compatibility: Existing ai = ["openai"] item selections continue functioning
3. Configuration Format: TOML structure and environment variable syntax preserved
4. Migration Timeline: Users can migrate at their own pace, no forced upgrades

Benefits for Configuration Management

• Unified Model Access: 200+ models through consistent configuration syntax
• Cost Transparency: Clear pricing visibility in configuration interface
• Smart Defaults: Intelligent model selection based on task complexity
• Easy Experimentation: Simple A/B testing without complex setup
• Future-Proof: Automatic support for new models as they become available

This approach ensures current users experience zero disruption while providing a clear path to enhanced capabilities.

[adawalli]

Clarification: LangGraph Routing Architecture

I wanted to elaborate that even though the Config is backwards compatible, we would not be using the openai SDK anymore.

Current Architecture

# Direct SDK usage
from openai import OpenAI

class OpenAIBackend(AIBackend):
    def __init__(self, config):
        self.client = OpenAI(api_key=config.api_key)  # Direct OpenAI SDK
    
    def evaluate(self, listing):
        response = self.client.chat.completions.create(...)  # Direct API call

Post-Migration Architecture

# All providers route through LangGraph
from langchain_openai import ChatOpenAI
from langchain_community.chat_models import ChatDeepSeek, ChatOllama
from langgraph import StateGraph

class LangGraphBackend(AIBackend):
    def __init__(self, config):
        # Convert existing config to LangGraph model
        if config.provider == "openai":
            self.model = ChatOpenAI(
                api_key=config.api_key,
                model=config.model,
                base_url=config.base_url
            )
        elif config.provider == "deepseek":
            self.model = ChatDeepSeek(api_key=config.api_key)
        # etc.
    
    def evaluate(self, listing):
        # All calls go through LangGraph, regardless of original provider
        graph = StateGraph(...)
        return graph.invoke({"listing": listing})

Benefits of Universal LangGraph Routing

1. Unified Interface: All providers use the same evaluation pipeline
2. Enhanced Features: Automatic retry logic, caching, and error handling for all providers
3. Future Extensibility: Multi-model workflows become available for any provider
4. Consistent Behavior: Same logging, monitoring, and debugging across all AI services

Configuration Transparency

Existing configs work unchanged, but internally route through LangGraph:

[ai.openai]           # User sees familiar config
api_key = "sk-xxx"    # ↓ Internally converts to ↓
model = "gpt-4"       # ChatOpenAI(api_key="sk-xxx", model="gpt-4")

[ai.deepseek]         # User sees familiar config  
api_key = "sk-xxx"    # ↓ Internally converts to ↓
                      # ChatDeepSeek(api_key="sk-xxx")

Migration Implementation

Phase 1: Replace backend implementations while preserving configuration interface

# Before: Multiple backend classes
class OpenAIBackend(AIBackend): ...
class DeepSeekBackend(AIBackend): ...  
class OllamaBackend(AIBackend): ...

# After: Single LangGraph backend with provider adapters
class LangGraphBackend(AIBackend):
    def _get_model(self, config):
        provider_map = {
            "openai": lambda c: ChatOpenAI(api_key=c.api_key, model=c.model),
            "deepseek": lambda c: ChatDeepSeek(api_key=c.api_key),
            "ollama": lambda c: ChatOllama(base_url=c.base_url, model=c.model),
        }
        return provider_map[config.provider](config)

Phase 2: Expose LangGraph-specific features while maintaining backward compatibility

User Experience Impact

• Zero Config Changes: Existing [ai.openai] sections work identically
• Improved Reliability: Better error handling and retry logic across all providers
• Enhanced Debugging: LangGraph Studio visualization works for all providers
• Future Features: Multi-model evaluation becomes available for existing configs

This approach ensures that migrating to LangGraph provides immediate benefits (better reliability, debugging) while preserving the familiar configuration experience.

[adawalli]

I create a PRD that I'm goign to start iterating on. I went with langchain over langgraph to keep it simple for now:
https://github.com/adawalli/ai-marketplace-monitor/blob/9c9cc850aab10bef950353af31b9693c6fcfe5ca/docs/langraph_migration_prd.md