Skip to content

fix(metrics): reject list params containing only empty strings (#2248)#2739

Open
kadiryonak wants to merge 2 commits into
confident-ai:mainfrom
kadiryonak:fix/2248-empty-retrieval-context
Open

fix(metrics): reject list params containing only empty strings (#2248)#2739
kadiryonak wants to merge 2 commits into
confident-ai:mainfrom
kadiryonak:fix/2248-empty-retrieval-context

Conversation

@kadiryonak

Copy link
Copy Markdown

Summary

Fixes #2248. check_llm_test_case_params only rejected None params, so a retrieval_context (or context) of [""] bypassed validation and let metrics like FaithfulnessMetric produce a meaningless score.

This treats a string-content list param that contains only empty/whitespace strings as a missing param and raises MissingTestCaseParamsError, mirroring the existing empty actual_output check in the same function.

Behavior

input before after
retrieval_context=[""] silent score raises MissingTestCaseParamsError
[" ", "\n"] silent score raises
[] (intentionally empty) unchanged unchanged
["real ctx"] / ["", "real"] unchanged unchanged

Multimodal lists (containing MLLMImage) are unaffected, since the check only triggers when every item is a blank string.

Tests

Adds tests/test_metrics/test_faithfulness_empty_retrieval_context.py (uses DummyModel, no API key). Verified the new tests fail without the fix and pass with it; existing param-validation tests still pass.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Jun 9, 2026

Copy link
Copy Markdown

@kadiryonak is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

@kadiryonak

Copy link
Copy Markdown
Author

Hi @penguine-ip — flagging a design decision in case it's useful for review.

In #2248 you noted that an empty retrieval_context ([]) is intentionally treated as "no context provided" → faithfulness 1.0, and that None is the way to get the missing-param error. This PR deliberately does not change that: [] and any list with at least one non-blank string keep their current behavior.

It only targets the narrower case the issue reports — a list that is present but contains only empty/whitespace strings (e.g. [""]). That input slips past the existing None-only check and yields a meaningless score. The fix mirrors the empty-actual_output check already in check_llm_test_case_params, so it stays consistent with existing validation rather than introducing new semantics.

Happy to adjust the scope (e.g. only retrieval_context/context, or normalize instead of raise) if you'd prefer a different direction.

…dent-ai#2248)

check_llm_test_case_params only rejected None params, so a
retrieval_context (or context) of [""] bypassed validation and produced
a meaningless faithfulness score. Treat a string-content list param that
contains only empty/whitespace strings as a missing param, mirroring the
existing empty actual_output check. Intentionally empty lists ([]) and
lists with at least one non-blank string are unaffected.
@kadiryonak kadiryonak force-pushed the fix/2248-empty-retrieval-context branch from 55934c3 to 78f9201 Compare June 9, 2026 13:56
@penguine-ip

Copy link
Copy Markdown
Contributor

hey @kadiryonak thanks for the PR! It looks good - i'd just kindly request the comments to be removed, once that's done it can be merged, thanks!

@kadiryonak

Copy link
Copy Markdown
Author

Thanks for the review! I've removed the comments and pushed the updates. Please let me know if there's anything else that needs to be adjusted. @penguine-ip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Faithfullness metric returns 1.0 even if retrieval_context is empty

2 participants