fix: add deepseek-v4 models, fix calculate_cost, and improve error ha…#2736
Open
Comui520 wants to merge 1 commit into
Open
fix: add deepseek-v4 models, fix calculate_cost, and improve error ha…#2736Comui520 wants to merge 1 commit into
Comui520 wants to merge 1 commit into
Conversation
|
@Comui520 is attempting to deploy a commit to the Confident AI Team on Vercel. A member of the Team first needs to authorize it. |
a434a61 to
38b6cae
Compare
…ndling - Register deepseek-v4-flash and deepseek-v4-pro in DEEPSEEK_MODELS_DATA - Fix calculate_cost() across all 8 providers to return EvaluationCost(0.0, ...) instead of None when pricing is unknown (Anthropic, OpenAI, Azure, Gemini, Kimi, Grok, Amazon Bedrock, DeepSeek) - Fix ContextGenerator to raise DeepEvalError when ALL documents fail instead of silently returning empty contexts - Fix Synthesizer to raise DeepEvalError when contexts are empty deepseek-chat and deepseek-reasoner will be deprecated on 2026-07-24 in favor of v4-flash and v4-pro, so this is needed for forward compatibility.
38b6cae to
4d4664e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
deepseek-v4-flashanddeepseek-v4-proinDEEPSEEK_MODELS_DATAcalculate_cost()across all 8 providers to never returnNoneContextGeneratorandSynthesizerto surface errors instead of silently returning empty resultsDetails
Problem
When pricing data is unknown (model not in registry),
calculate_cost()returnedNoneacross ALL providers (DeepSeek, Anthropic, OpenAI, Azure, Gemini, Kimi, Grok, Bedrock),
causing
TypeErrorinContextGenerator.evaluate_chunk()(total_cost += None).This error was caught by a broad
except Exceptionand logged but never surfaced,resulting in empty goldens with no error to the user.
Changes
All 8 provider models —
calculate_cost()now returnsEvaluationCost(0.0, ...)when prices are unknown, instead of
None. Fixes theOptional[float]return typeto be consistently
float.constants.py— Added model definitions fordeepseek-v4-flashanddeepseek-v4-pro.context_generator.py—generate_contexts()anda_generate_contexts()now raiseDeepEvalErrorwhen ALL documents fail, instead of silently returning empty contexts.synthesizer.py—generate_goldens_from_docs()anda_generate_goldens_from_docs()now raise
DeepEvalErrorwhen contexts are empty.Why this matters now
Per DeepSeek API docs,
deepseek-chatanddeepseek-reasonerwill be deprecated on 2026-07-24 in favor ofdeepseek-v4-flashand
deepseek-v4-pro.