feat:Updated Deep Infra models #106
Conversation
WalkthroughThe pull request introduces various modifications across multiple classes within the Changes
Possibly related PRs
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (4)
src/DeepInfra/src/DeepInfraModelProvider.cs (1)
Line range hint
1-90: Consider adding model capability documentationWhile the model metadata is well-structured, it would be helpful to include documentation about each model's specific capabilities (e.g., code generation, instruction following, etc.) to aid in model selection.
Would you like me to propose a documentation structure for the model capabilities?
src/Azure/src/AzureOpenAiChatModel.cs (3)
47-47: Fix typo in method nameThe method name contains a typo:
ExtarctToolsshould beExtractTools.- List<ChatTool> tools = ExtarctTools(request); + List<ChatTool> tools = ExtractTools(request);
256-261: Enhance tool name parsing validationWhile using
ElementAtOrDefaultprovides some safety, consider adding explicit validation for the expected format to provide more meaningful error messages.- var toolNameAndId = message.ToolName?.Split(':') ?? - throw new ArgumentException("Invalid functionCall name and id string"); - - var toolCall = OpenAI.Chat.ChatToolCall.CreateFunctionToolCall(toolNameAndId.ElementAtOrDefault(1), - toolNameAndId.ElementAtOrDefault(0), - BinaryData.FromString(message.Content)); + var toolNameAndId = message.ToolName?.Split(':') ?? + throw new ArgumentException("Tool name is required"); + + if (toolNameAndId.Length != 2) + { + throw new ArgumentException($"Invalid tool name format. Expected 'name:id' but got '{message.ToolName}'"); + } + + var toolCall = OpenAI.Chat.ChatToolCall.CreateFunctionToolCall( + toolNameAndId[1], + toolNameAndId[0], + BinaryData.FromString(message.Content));
Line range hint
314-342: Implement required interface methodsSeveral critical methods are currently throwing
NotImplementedException:
CountTokensoverloads for token countingTryCalculatePriceInUsdfor price calculationThese methods are part of the implemented interfaces and should be properly implemented to avoid runtime exceptions.
Would you like help implementing these methods? I can provide implementations based on Azure OpenAI's token counting and pricing models.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (17)
src/Abstractions/src/Chat/ChatResponse.cs(1 hunks)src/Abstractions/src/Chat/ChatResponseDelta.cs(1 hunks)src/Amazon.Bedrock/src/TextToImage/AmazonTitanImageGenerationModel.cs(0 hunks)src/Azure/src/AzureOpenAIConfiguration.cs(1 hunks)src/Azure/src/AzureOpenAiChatModel.cs(5 hunks)src/Azure/src/AzureOpenAiProvider.cs(2 hunks)src/DeepInfra/src/DeepInfraModelIds.cs(4 hunks)src/DeepInfra/src/DeepInfraModelProvider.cs(1 hunks)src/DeepInfra/src/Predefined/AllModels.cs(2 hunks)src/DeepInfra/tools/LangChain.Providers.DeepInfra.CodeGenerator/Helpers/DocumentHelper.cs(0 hunks)src/DeepInfra/tools/LangChain.Providers.DeepInfra.CodeGenerator/Main/DeepInfraCodeGenerator.cs(0 hunks)src/Google/src/GoogleChatModel.Tokens.cs(1 hunks)src/IntegrationTests/BaseTests.cs(2 hunks)src/OpenAI/src/OpenAiProvider.cs(0 hunks)src/OpenAI/src/TextToSpeech/OpenAiTextToSpeechSettings.cs(0 hunks)src/OpenRouter/tools/LangChain.Providers.OpenRouter.CodeGenerator/Helpers/DocumentHelper.cs(0 hunks)src/Pollinations/src/LangChain.Providers.Pollinations/PollinationsModelProvider.cs(0 hunks)
💤 Files with no reviewable changes (7)
- src/Amazon.Bedrock/src/TextToImage/AmazonTitanImageGenerationModel.cs
- src/DeepInfra/tools/LangChain.Providers.DeepInfra.CodeGenerator/Helpers/DocumentHelper.cs
- src/DeepInfra/tools/LangChain.Providers.DeepInfra.CodeGenerator/Main/DeepInfraCodeGenerator.cs
- src/OpenAI/src/OpenAiProvider.cs
- src/OpenAI/src/TextToSpeech/OpenAiTextToSpeechSettings.cs
- src/OpenRouter/tools/LangChain.Providers.OpenRouter.CodeGenerator/Helpers/DocumentHelper.cs
- src/Pollinations/src/LangChain.Providers.Pollinations/PollinationsModelProvider.cs
✅ Files skipped from review due to trivial changes (4)
- src/Abstractions/src/Chat/ChatResponseDelta.cs
- src/Azure/src/AzureOpenAIConfiguration.cs
- src/Azure/src/AzureOpenAiProvider.cs
- src/Google/src/GoogleChatModel.Tokens.cs
🔇 Additional comments (18)
src/Abstractions/src/Chat/ChatResponse.cs (2)
59-59: LGTM! Improved code formatting
The removal of unnecessary whitespace improves code readability while maintaining the method's functionality.
Line range hint 37-37: Great use of C# 12's collection expressions!
The initialization of ToolCalls using = [] is a clean and efficient way to ensure the property is never null. This aligns well with the class's overall null-safe design pattern.
src/DeepInfra/src/DeepInfraModelProvider.cs (3)
25-25: LGTM: Price update for MythomaxL213B model
The price adjustment from 1.0e-7 to 8e-8 USD per token aligns with the changes mentioned in the AI summary and appears to be part of a routine pricing update.
33-33: Verify pricing consistency across Qwen model family
The new Qwen257BInstruct model's pricing (6e-8 USD) and context length (32768) are consistent with other Qwen models in the same series (e.g., Qwen27BInstruct).
✅ Verification successful
Pricing and context length are consistent within Qwen 7B models
The verification confirms that Qwen2.5-7B-Instruct's pricing (6e-8 USD) and context length (32768) match other 7B models in the DeepInfra provider:
- Qwen2-7B-Instruct: 6e-8 USD, 32768 context
- Qwen2.5-Coder-7B: 6e-8 USD, 32768 context
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify consistency of pricing across Qwen models
rg "Qwen.*ToMetadata.*6[Ee]-8" --type cs
Length of output: 42
Script:
#!/bin/bash
# Search for all Qwen model metadata definitions to compare pricing
rg "Qwen.*ToMetadata" -A 1 --type cs
Length of output: 6455
15-17: Verify the context length for new models
The new models MetaLlama318BInstructTurbo and Qwen25Coder32BInstruct have been added with context lengths of 131072 and 32768 respectively. These values should be verified against the official DeepInfra documentation.
✅ Verification successful
Context lengths are consistent with model families
Based on the search results:
- The context length of 131072 for Meta-Llama-3.1 models is consistent across the family (Meta-Llama-3.1-70B-Instruct, Meta-Llama-3.1-8B-Instruct all use 131072)
- The context length of 32768 for Qwen models is consistent across all Qwen variants in both DeepInfra and Together providers (Qwen2-72B, Qwen1.5 series, Qwen2.5 series all use 32768)
The context lengths for both new models align with their respective model families' established patterns in the codebase.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Search for any documentation or tests that verify these context lengths
rg -l "131072.*Meta-Llama-3\.1-8B-Instruct-Turbo|32768.*Qwen2\.5-Coder-32B-Instruct" --type md
Length of output: 96
Script:
#!/bin/bash
# Let's try a broader search to understand the context window patterns in the codebase
rg "ToMetadata.*Meta-Llama.*131072|ToMetadata.*Qwen.*32768" -A 1 -B 1
# Also search for any tests that might verify these models
rg "MetaLlama318BInstructTurbo|Qwen25Coder32BInstruct" --type cs
# Check if there are any configuration files or documentation about model parameters
fd -e json -e yaml -e md -x grep -l "Meta-Llama-3\.1-8B-Instruct-Turbo\|Qwen2\.5-Coder-32B-Instruct" {} \;
Length of output: 9198
src/IntegrationTests/BaseTests.cs (3)
99-99: LGTM! Good addition of performance tracking.
The stopwatch initialization is a valuable addition for monitoring the timing of streaming responses.
133-133: LGTM! Enhanced logging with timing information.
The addition of timing information to the logging statements will help with debugging and performance analysis of streaming responses.
Also applies to: 139-139
Line range hint 144-146: Investigate the commented-out delta equivalence assertion.
The commented assertion suggests there might be inconsistencies in how different providers handle streaming deltas. This should be investigated to ensure consistent behavior across providers.
Let's analyze the delta handling across different providers:
src/Azure/src/AzureOpenAiChatModel.cs (2)
Line range hint 147-157: LGTM: Robust tool calls handling
The implementation properly handles tool calls with appropriate null checking and field mapping.
Line range hint 274-312: LGTM: Clean implementation with modern C# features
The method effectively handles both tool calls and regular messages, making good use of C# 12's collection expressions.
src/DeepInfra/src/Predefined/AllModels.cs (4)
18-20: LGTM! MetaLlama318BInstructTurboModel implementation
The class follows the established patterns and conventions for model implementations.
28-30: LGTM! Qwen25Coder32BInstructModel implementation
The class follows the established patterns and conventions for model implementations.
108-110: LGTM! Qwen257BInstructModel implementation
The class follows the established patterns and conventions for model implementations.
18-20: Verify model IDs exist in DeepInfraModelIds
Let's ensure the model IDs used by the new classes are properly defined.
Also applies to: 28-30, 108-110
✅ Verification successful
Model IDs are properly defined in DeepInfraModelIds
The verification shows that all referenced model IDs (MetaLlama318BInstructTurbo, Qwen25Coder32BInstruct, and Qwen257BInstruct) are properly defined in src/DeepInfra/src/DeepInfraModelIds.cs and are correctly used in both the model provider and predefined model classes.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check if the model IDs are defined in DeepInfraModelIds
# Search for model ID definitions
echo "Searching for model ID definitions..."
rg "MetaLlama318BInstructTurbo|Qwen25Coder32BInstruct|Qwen257BInstruct" --type cs
Length of output: 1994
src/DeepInfra/src/DeepInfraModelIds.cs (4)
42-51: LGTM: MetaLlama318BInstructTurbo model addition
The new model is well-documented with all required details including context length, pricing, and valid HuggingFace URL.
64-73: LGTM: Qwen25Coder32BInstruct model addition
The new code-specific model is well-documented with all required details including context length, pricing, and valid HuggingFace URL.
240-249: LGTM: Qwen257BInstruct model addition
The new model is well-documented with all required details including context length, pricing, and valid HuggingFace URL.
156-157: LGTM: Updated MythomaxL213B pricing
The cost update from $0.1/MTok to $0.08/MTok for both prompt and completion is properly documented.
Created by Github Actions
Summary by CodeRabbit
Release Notes
New Features
MetaLlama318BInstructTurbo,Qwen25Coder32BInstruct,Qwen257BInstruct, andQwen25Coder7Bto enhance model offerings.OpenAiTextToSpeechSettingsclass for improved text-to-speech configuration.Improvements
ToolCallsproperty inChatResponsefor better performance.Bug Fixes
Chores