Issue #1611 Fixes

[PradyMagal]

Fix TruthfulQA compatibility with AnthropicModel

Issue

TruthfulQA benchmark was incompatible with AnthropicModel, always failing with:
ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.

Root Cause

• TruthfulQA attempted structured output (JSON schemas) with AnthropicModel
• AnthropicModel responded with natural language explanations instead of JSON
• JSON parsing failed but only TypeError was caught, not ValueError
• Benchmark crashed instead of falling back to text-based prompting

Line 175 (predict method):

# Before
except TypeError:

# After  
except (TypeError, ValueError, AttributeError):

Line 235 (batch_predict method):

# Before
except TypeError:

# After
except (TypeError, ValueError, AttributeError):

Result

• ✅ TruthfulQA now works with AnthropicModel
• ✅ Maintains backward compatibility with existing models
• ✅ Graceful fallback to text-based prompting when structured output fails
• ✅ More robust error handling for any model without structured output support

Testing

Confirmed working with:

from deepeval.benchmarks import TruthfulQA
from deepeval.benchmarks.tasks import TruthfulQATask
from deepeval.benchmarks.modes import TruthfulQAMode
from deepeval.models import AnthropicModel

benchmark = TruthfulQA(
    tasks=[TruthfulQATask.ADVERTISING, TruthfulQATask.FICTION],
    mode=TruthfulQAMode.MC2
)
benchmark.evaluate(model=AnthropicModel())

Output:
Filter: 100%|████████████████████████████████████████████████████████████████████████████| 817/817 [00:00<00:00, 24678.95 examples/s] Processing Fiction: 100%|████████████████████████████████████████████████████████████████████████████| 30/30 [05:04<00:00, 10.14s/it] TruthfulQA Task Accuracy (task=Fiction): 82.43333333333334 Overall TruthfulQA Accuracy: 82.43333333333334

[vercel]

@PradyMagal is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

[penguine-ip]

hey @PradyMagal thanks! But can you please undo poetry lock and all the adds to the toml file? We just spent a whole week cutting down deepeval's package size in prod

[PradyMagal]

Poetry Lock has been undone, same with the additions to the toml file

Let me know if there's anything I missed

[penguine-ip]

@PradyMagal perfect, thanks!