[Feature] Eval metrics for heterogeneous financial document chunks — handling mixed document types in a single test run

[Ruthwik-Data]

Background

I'm building a RAG evaluation pipeline for financial documents using DeepEval (related to my finrag-eval project). Financial document corpora are inherently heterogeneous — a single test run will include:

• 10-K filings (dense narrative + structured tables)
• Earnings call transcripts (dialogue, forward-looking statements, no tables)
• Balance sheets (pure tabular, structured financial data)
• Analyst reports (opinion + quantitative mixed)

Problem

Current DeepEval metrics (faithfulness, answer relevancy, contextual precision) use a single threshold across all document types. This creates two issues:

1. False negatives on structured docs: A faithfulness score of 0.7 on a balance sheet question (where the answer is a specific number) should be treated as a failure — but the same score on an earnings call narrative might be acceptable.
2. Threshold calibration is impossible: Setting a single threshold that works across document types means either too many false positives (narrative docs) or too many false negatives (structured docs).

Minimal Repro:

from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

# These two test cases need DIFFERENT thresholds
balance_sheet_case = LLMTestCase(
    input="What is Apple's total current assets?",
    actual_output="$135.4B",  # Either exactly right or completely wrong
    retrieval_context=[...]
)

earnings_call_case = LLMTestCase(
    input="What is management's outlook on AI investment?",
    actual_output="Management expressed cautious optimism...",  # Graded on a spectrum
    retrieval_context=[...]
)

metric = FaithfulnessMetric(threshold=0.7)  # Same threshold doesn't work for both

Proposed Solution

Add a document_type field to LLMTestCase that allows per-type threshold configuration:

# Option 1: Per-test-case document type
balance_sheet_case = LLMTestCase(
    input="What is Apple's total current assets?",
    actual_output="$135.4B",
    retrieval_context=[...],
    metadata={"document_type": "balance_sheet"}  # Using existing metadata field
)

# Option 2: MetricConfig per document type
metric = FaithfulnessMetric(
    threshold_by_type={
        "balance_sheet": 0.95,      # Structured — must be exact
        "earnings_call": 0.70,      # Narrative — some latitude
        "10k_filing": 0.85,         # Mixed — middle ground
        "default": 0.75
    }
)

Why This Matters

For production RAG systems over heterogeneous corpora, a single eval threshold creates misleading pass/fail signals. This is especially critical in financial/legal domains where structured data retrieval quality is binary (right number or wrong), but narrative retrieval is gradient.

I explored this problem in depth in finrag-eval and had to write custom threshold logic outside DeepEval. Native support would make this significantly cleaner.

Related

• My finrag-eval project: https://github.com/Ruthwik-Data/finrag-eval
• Building on top of the heterogeneous chunking patterns discussed in various RAG evaluation literature

[rohitmannur007]

Hi @Ruthwik-Data ,

I took a look through the current implementation and noticed that LLMTestCase already supports a metadata field. This might allow document-specific evaluation behavior without introducing any schema changes.

One possible approach could be:

LLMTestCase(
...,
metadata={"document_type": "balance_sheet"}
)

combined with metric-level threshold overrides, for example:

FaithfulnessMetric(
threshold=0.75,
threshold_overrides={
"balance_sheet": 0.95,
"earnings_call": 0.70,
"10k_filing": 0.85,
}
)

The metric could resolve an effective threshold from test_case.metadata["document_type"] and apply pass/fail evaluation against that threshold while preserving the existing scoring behavior.

This would remain backward compatible, avoid changes to the test case schema, and support heterogeneous corpora within a single evaluation run.

Would maintainers prefer an approach like this, or is there another direction that would better align with DeepEval's design philosophy?

I'd be happy to work on a PR once the preferred design is clarified.

[Ruthwik-Data]

@rohitmannur007 — the threshold_overrides via metadata["document_type"] approach is exactly right. Keeping it backward-compatible through the existing metadata field is the cleanest path since it avoids any schema changes to LLMTestCase.

A few thoughts on #2785 that might be worth reviewing:

• Fallback behavior: When document_type isn't set in metadata, the metric should silently fall back to the default threshold — worth making sure that's explicit in the implementation so existing test suites don't break.
• Type validation: If threshold_overrides accepts arbitrary string keys, it might be worth a small validation step to warn when an unrecognized document_type is passed (easy to miss otherwise).
• Docs / example: A short example showing the pattern for a mixed 10-K + earnings call test run would help adoption significantly.

Happy to contribute a test fixture or example to the PR if that's useful. Looking forward to seeing this merged!

[rohitmannur007]

Thanks for the feedback @Ruthwik-Data

Glad the metadata[“document_type”] + threshold_overrides approach aligns with the intended design.

The current implementation preserves existing behavior by falling back to the metric’s default threshold whenever document_type is not present or no override is configured.

I’ll also review adding explicit tests for:

• missing metadata fallback
• unknown document types
• document-specific threshold resolution

The documentation example in #2790 is a great suggestion and should help demonstrate mixed-document evaluation workflows clearly.

Appreciate the review and feedback.