fix(metrics): reject list params containing only empty strings (#2248)#2739
fix(metrics): reject list params containing only empty strings (#2248)#2739kadiryonak wants to merge 2 commits into
Conversation
|
@kadiryonak is attempting to deploy a commit to the Confident AI Team on Vercel. A member of the Team first needs to authorize it. |
|
Hi @penguine-ip — flagging a design decision in case it's useful for review. In #2248 you noted that an empty It only targets the narrower case the issue reports — a list that is present but contains only empty/whitespace strings (e.g. Happy to adjust the scope (e.g. only |
…dent-ai#2248) check_llm_test_case_params only rejected None params, so a retrieval_context (or context) of [""] bypassed validation and produced a meaningless faithfulness score. Treat a string-content list param that contains only empty/whitespace strings as a missing param, mirroring the existing empty actual_output check. Intentionally empty lists ([]) and lists with at least one non-blank string are unaffected.
55934c3 to
78f9201
Compare
|
hey @kadiryonak thanks for the PR! It looks good - i'd just kindly request the comments to be removed, once that's done it can be merged, thanks! |
|
Thanks for the review! I've removed the comments and pushed the updates. Please let me know if there's anything else that needs to be adjusted. @penguine-ip |
Summary
Fixes #2248.
check_llm_test_case_paramsonly rejectedNoneparams, so aretrieval_context(orcontext) of[""]bypassed validation and let metrics likeFaithfulnessMetricproduce a meaningless score.This treats a string-content list param that contains only empty/whitespace strings as a missing param and raises
MissingTestCaseParamsError, mirroring the existing emptyactual_outputcheck in the same function.Behavior
retrieval_context=[""]MissingTestCaseParamsError[" ", "\n"][](intentionally empty)["real ctx"]/["", "real"]Multimodal lists (containing
MLLMImage) are unaffected, since the check only triggers when every item is a blank string.Tests
Adds
tests/test_metrics/test_faithfulness_empty_retrieval_context.py(usesDummyModel, no API key). Verified the new tests fail without the fix and pass with it; existing param-validation tests still pass.🤖 Generated with Claude Code