Trouble using Faithfulness Metric with HHEM-2.1-Open

[awlevin]

[x] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question
Can't figure out how to use Faithfulness metric with HHEM-2.1-Open (https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/?h=faithfulness#example)

Code Examples

import asyncio
from ragas.dataset_schema import SingleTurnSample 
from ragas.metrics import FaithfulnesswithHHEM

async def main():

    sample = SingleTurnSample(
        user_input="When was the first super bowl?",
        response="The first superbowl was held on Jan 15, 1967",
        retrieved_contexts=[
            "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles."
        ]
    )

    scorer = FaithfulnesswithHHEM()
    result = await scorer.single_turn_ascore(sample)
    print(result)

if __name__ == "__main__":
    asyncio.run(main())

Additional context
Anything else you want to share with us?

Just keep getting this error: assert self.llm is not None, "LLM is not set" but I'm following the documentation exactly. I thought the LLM would be set under the hood when using this metric that pulls automatically from HuggingFace?

[sahusiddharth]

Hi @awlevin,

Yes, your understanding is correct, the FaithfulnesswithHHEM metric automatically selects the model from HuggingFace, which is used for doing classification.

However, under the hood, FaithfulnesswithHHEM requires a separate LLM to break the response into simpler statements before classification.

from ragas.dataset_schema import SingleTurnSample 
from ragas.metrics import FaithfulnesswithHHEM
from ragas.llms import LangchainLLMWrapper
from langchain_openai import OpenAI

evaluator_llm = LangchainLLMWrapper(OpenAI())


sample = SingleTurnSample(
        user_input="When was the first super bowl?",
        response="The first superbowl was held on Jan 15, 1967",
        retrieved_contexts=[
            "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles."
        ]
    )
scorer = FaithfulnesswithHHEM(llm=evaluator_llm)
await scorer.single_turn_ascore(sample)

I hope this helps