Skip to content

Faithfullness metric returns 1.0 even if retrieval_context is empty #2248

Description

@omar-araboghli

Describe the bug
Faithfullness metric returns 1.0 even if retrieval_context is empty

To Reproduce
Steps to reproduce the behavior:

metric = FaithfulnessMetric(model="...")
test_case = LLMTestCase(
        input="xxx",
        expected_output="xxx",
        actual_output="xxx",
        retrieval_context=[""],  # this is a list of empty string
)
metric.measure(test_case)

I believe the required parameters check is being bypassed here and this might apply for any required param of any metric using the same check function. I only tested with above code snippet as of now.

Expected behavior
I expect an error or a 0.0 score.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions