Skip to content

fix: add deepseek-v4 models, fix calculate_cost, and improve error ha…#2736

Open
Comui520 wants to merge 1 commit into
confident-ai:mainfrom
Comui520:fix/deepseek-v4-support
Open

fix: add deepseek-v4 models, fix calculate_cost, and improve error ha…#2736
Comui520 wants to merge 1 commit into
confident-ai:mainfrom
Comui520:fix/deepseek-v4-support

Conversation

@Comui520

@Comui520 Comui520 commented Jun 9, 2026

Copy link
Copy Markdown

Summary

  • Register deepseek-v4-flash and deepseek-v4-pro in DEEPSEEK_MODELS_DATA
  • Fix calculate_cost() across all 8 providers to never return None
  • Fix ContextGenerator and Synthesizer to surface errors instead of silently returning empty results

Details

Problem

When pricing data is unknown (model not in registry), calculate_cost() returned None
across ALL providers (DeepSeek, Anthropic, OpenAI, Azure, Gemini, Kimi, Grok, Bedrock),
causing TypeError in ContextGenerator.evaluate_chunk() (total_cost += None).
This error was caught by a broad except Exception and logged but never surfaced,
resulting in empty goldens with no error to the user.

Changes

All 8 provider modelscalculate_cost() now returns EvaluationCost(0.0, ...)
when prices are unknown, instead of None. Fixes the Optional[float] return type
to be consistently float.

constants.py — Added model definitions for deepseek-v4-flash and deepseek-v4-pro.

context_generator.pygenerate_contexts() and a_generate_contexts() now raise
DeepEvalError when ALL documents fail, instead of silently returning empty contexts.

synthesizer.pygenerate_goldens_from_docs() and a_generate_goldens_from_docs()
now raise DeepEvalError when contexts are empty.

Why this matters now

Per DeepSeek API docs, deepseek-chat and
deepseek-reasoner will be deprecated on 2026-07-24 in favor of deepseek-v4-flash
and deepseek-v4-pro.

@vercel

vercel Bot commented Jun 9, 2026

Copy link
Copy Markdown

@Comui520 is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

@Comui520 Comui520 force-pushed the fix/deepseek-v4-support branch 2 times, most recently from a434a61 to 38b6cae Compare June 9, 2026 11:24
…ndling

- Register deepseek-v4-flash and deepseek-v4-pro in DEEPSEEK_MODELS_DATA
- Fix calculate_cost() across all 8 providers to return EvaluationCost(0.0, ...)
  instead of None when pricing is unknown (Anthropic, OpenAI, Azure, Gemini,
  Kimi, Grok, Amazon Bedrock, DeepSeek)
- Fix ContextGenerator to raise DeepEvalError when ALL documents fail
  instead of silently returning empty contexts
- Fix Synthesizer to raise DeepEvalError when contexts are empty

deepseek-chat and deepseek-reasoner will be deprecated on 2026-07-24
in favor of v4-flash and v4-pro, so this is needed for forward compatibility.
@Comui520 Comui520 force-pushed the fix/deepseek-v4-support branch from 38b6cae to 4d4664e Compare June 9, 2026 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant