Open source LLM evaluation framework with cost + latency + hallucination metrics

[vignesh2027]

Hey DeepEval community!

Love what DeepEval is doing with LLM-as-judge evaluation. I built an open source framework that takes a complementary approach focused on production metrics.

Key difference from DeepEval: No LLM-as-judge needed. Everything runs locally or from real API responses.

5 metrics tracked simultaneously:

• Accuracy: 4-strategy cascade (exact, normalized, MC, fuzzy Levenshtein)
• Latency: p50/p75/p90/p95/p99 from real async API calls
• Cost per 1K tokens: from actual token counts, not estimates
• Hallucination Rate: linguistic signal analysis (hedging/uncertainty/grounding signals)
• Reasoning Quality: chain-of-thought depth score 1-10

One command benchmark:

pip install llm-evaluation-framework
llm-eval compare --models gpt-4o-mini --models gemini/gemini-1.5-flash --benchmark mmlu --samples 100

Works via LiteLLM so any model is supported.

Live demo (no API key needed): https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework

71 tests, 82% coverage. The two approaches complement well: DeepEval for semantic/behavioral evaluation, this for production metrics. Feedback welcome!