Add support for multiple model configurations with litellm Router (#2808)

[devin-ai-integration]

Add Support for Multiple Model Configurations with liteLLM Router

This PR addresses issue #2808 by implementing support for configuring multiple language models with different API keys and configurations in CrewAI.

Changes

• Enhanced the LLM class to support the model_list parameter for configuring multiple models
• Added support for different routing strategies through the routing_strategy parameter
• Updated the Agent class to pass model configurations to the LLM
• Added comprehensive tests to verify the functionality
• Created documentation with examples for users

Features

• Load-balancing across multiple model deployments
• Fallback mechanisms for handling rate limits or errors
• Support for various routing strategies (simple-shuffle, least-busy, usage-based, latency-based, cost-based)
• Fine-grained control over model selection and usage

Documentation

Added a new documentation file docs/multiple_model_config.md with detailed examples of how to use the new functionality.

Testing

Added tests in tests/multiple_model_config_test.py that verify:

• LLM initialization with model_list
• LLM initialization with routing strategy
• Agent initialization with model_list
• LLM.call with router
• LLM.call without router

All tests are passing.

Link to Devin run

https://app.devin.ai/sessions/f1dbb2f840084e1ea3d4e680b749b391

Requested by

Joe Moura ([email protected])

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[joaomdmoura]

Disclaimer: This review was made by a crew of AI Agents.

Code Review Comment

Overview

The PR successfully implements support for multiple model configurations using the litellm Router, enhancing load balancing and fallback capabilities for language models in CrewAI. This is a significant enhancement that provides greater flexibility and performance improvements in model utilization.

Documentation Analysis (docs/multiple_model_config.md)

• Pros:

  • Well-structured documentation with clear examples.
  • Comprehensive coverage of features and configuration options.

• Suggestions for Improvement:

  • Please add a section specifically addressing error handling and troubleshooting for common issues related to model configurations.

Core Changes Analysis

1. Agent.py Changes

• Positive Aspects:

  • Clean integration of model_list and routing_strategy fields, providing clarity in model configuration.
  • Robust handling of different LLM initialization scenarios.

• Recommendations:

  1. Implement Type Validation:

     • Use an Enum for routing_strategy to enforce strict typing, allowing for better validation and avoiding potential misconfigurations.

     from enum import Enum
     
     class RoutingStrategy(str, Enum):
         SIMPLE_SHUFFLE = "simple-shuffle"
         LEAST_BUSY = "least-busy"
         USAGE_BASED = "usage-based"
         LATENCY_BASED = "latency-based"
         COST_BASED = "cost-based"
     
     routing_strategy: Optional[RoutingStrategy] = Field(default=None, description="Strategy for routing between multiple models")

  2. Enhance Model List Validation:

     • Ensure that all model configurations are correctly structured, which can prevent runtime errors and improve reliability.

     def validate_model_list(cls, v):
         if v is not None:
             for config in v:
                 if not isinstance(config, dict) or "model_name" not in config or "litellm_params" not in config:
                     raise ValueError("Each model config must contain 'model_name' and 'litellm_params'.")
         return v

2. LLM.py Changes

• Positive Aspects:

  • Clean initialization of the Router and appropriate parameter handling for calls to both the router and internal model functions.

• Recommendations:

  1. Add Error Handling in Router Initialization:

     • Adding a try-except block for error handling during router setup will aid in debugging and usage clarity.

     def _initialize_router(self):
         try:
             # Initialization logic...
         except Exception as e:
             logger.error(f"Failed to initialize router: {str(e)}")
             raise RuntimeError(f"Router initialization failed: {str(e)}")

  2. Implement Retry Logic for Router Calls:

     • By incorporating retry mechanisms, the system can handle transient issues more robustly.

     @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))
     def _execute_router_call(self, params):
         return self.router.completion(model=self.model, **params)

3. Test Coverage Analysis

• Positive Aspects:

  • The current tests effectively cover a wide range of scenarios with appropriate mocking of external dependencies.

• Recommendations:

  1. Edge Case Tests:

     • More thorough testing for invalid routing options will help improve error management.

     @pytest.mark.vcr(filter_headers=["authorization"])
     def test_llm_with_invalid_routing_strategy():
         with pytest.raises(ValueError) as exc_info:
             LLM(model="gpt-4", model_list=[{"model_name": "gpt-4", "litellm_params": {"model": "gpt-4"}}], routing_strategy="invalid-strategy")
         assert "Invalid routing strategy" in str(exc_info.value)

  2. Performance Tests:

     • Tests should also consider performance metrics to assess the efficiency of the router during typical workloads.

     @pytest.mark.performance
     def test_router_performance():
         # Performance test logic...

General Recommendations

1. Monitoring and Metrics:

   • Introduce telemetry and logging for performance metrics and error rates associated with model selection.

2. Documentation Enhancements:

   • Include architecture diagrams and recommended configurations, as well as failure scenarios and recovery procedures.

3. Security Best Practices:

   • Implement validation for sensitive information in model configurations and consider rate limiting for resource management.

4. Performance Optimizations:

   • Review options for connection pooling and caching to reduce response times and improve throughput.

Overall, while the implementation lays a solid foundation for multiple model configurations, addressing these recommendations will significantly enhance the robustness, maintainability, and performance of the system. Thank you for your hard work on this PR!

[devin-ai-integration]

Closing due to inactivity for more than 7 days.