How to use TrueLens with Ollama and Langchain Conversational Rag

[El-Moghazy2]

I am using the following RAG:

`llm = ChatOllama(
model='llama3.1:8b-instruct-q8_0',
temperature=0)

def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)

prompt = PromptTemplate(template=prompt_template_str,
input_variables=['context', 'question'])

def construct_rag_chain(llm, prompt, retriever):
memory = ConversationBufferMemory(
memory_key="chat_history",
return_messages=True,
output_key='answer'
)
rag_chain = ConversationalRetrievalChain.from_llm(
llm,
retriever=retriever,
memory=memory,
return_source_documents=True
)

return rag_chain

rag_chain = construct_rag_chain(llm, prompt, retriever)`

It is not obvious to me how to use TruLens with to evaluate this chain given LiteLLM as a provider:

provider = LiteLLM( model_engine="ollama/llama3.1:8b-instruct-q8_0", api_base="http://localhost:11434" )

If I follow the documentation which uses openAI provider I get different errors as it is not compatible.

[dosubot[bot]]

Hello @El-Moghazy2! I'm Dosu, here to help you solve bugs, answer your questions, and guide you as you work on your project. I'm ready to assist you while we wait for a human maintainer. How can I help you today?

To use TruLens with your Conversational Retrieval Chain setup involving Ollama and Langchain, and evaluate it using LiteLLM as a provider, follow these steps:

1. Install Required Packages:
   Make sure you have the necessary packages installed:

   pip install trulens trulens-apps-langchain trulens-providers-litellm litellm==1.11.1 langchain==0.0.351

2. Import Modules:
   Import the required modules for your setup:

   from trulens.core import Feedback, TruSession
   from trulens.apps.langchain import TruChain
   from langchain.chains import ConversationalRetrievalChain
   from langchain.prompts import PromptTemplate
   import litellm

3. Initialize LiteLLM Provider:
   Set up the LiteLLM provider with the correct model engine and API base:

   provider = LiteLLM(
       model_engine="ollama/llama3.1:8b-instruct-q8_0", 
       api_base="http://localhost:11434"
   )

4. Define Feedback Function:
   Create a feedback function using LiteLLM's relevance method:

   relevance = Feedback(provider.relevance_with_cot_reasons).on_input_output()

5. Instrument the Chain:
   Use TruLens to record and evaluate the chain:

   session = TruSession()
   session.reset_database()
   tru_recorder = TruChain(rag_chain, app_name='ConversationalRAG', feedbacks=[relevance])
   with tru_recorder as recording:
       llm_response = rag_chain({'context': format_docs(docs), 'question': 'Your question here'})

6. Explore Results:
   You can view the results using the TruLens dashboard:

   from trulens.dashboard import run_dashboard
   run_dashboard(session)  # This opens a local Streamlit app to explore the results

This setup allows you to evaluate your Conversational Retrieval Chain using TruLens with LiteLLM as a provider, leveraging the RAG setup with Ollama and Langchain. The feedback function evaluates the relevance of the retrieved context using a chain of thought methodology ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[El-Moghazy2]

@dosu I still get this error RuntimeError: Failed to get property _chain_type.

[dosubot[bot]]

To resolve the RuntimeError: Failed to get property _chain_type error, ensure you have the trulens-providers-litellm package installed. You can do this by running:

pip install trulens-providers-litellm

Additionally, switch to using trulens.providers.langchain.endpoint instead of any deprecated modules you might be using. This change can help resolve compatibility issues and prevent the error ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[El-Moghazy2]

@dosu show me the modifications in the code

[dosubot[bot]]

To modify your code for using TruLens with the Conversational Retrieval Chain and LiteLLM, follow these steps:

1. Ensure Required Packages are Installed:
   Make sure you have the necessary packages installed:

   pip install trulens trulens-apps-langchain trulens-providers-litellm litellm==1.11.1 langchain==0.0.351

2. Import Necessary Modules:
   Update your imports to include TruLens and LiteLLM:

   from trulens.core import Feedback, TruSession
   from trulens.apps.langchain import TruChain
   from langchain.chains import ConversationalRetrievalChain
   from langchain.prompts import PromptTemplate
   import litellm

3. Initialize LiteLLM Provider:
   Set up the LiteLLM provider:

   provider = LiteLLM(
       model_engine="ollama/llama3.1:8b-instruct-q8_0", 
       api_base="http://localhost:11434"
   )

4. Define Feedback Function:
   Create a feedback function using LiteLLM's relevance method:

   relevance = Feedback(provider.relevance_with_cot_reasons).on_input_output()

5. Instrument the Chain with TruLens:
   Use TruLens to record and evaluate the chain:

   session = TruSession()
   session.reset_database()
   tru_recorder = TruChain(rag_chain, app_name='ConversationalRAG', feedbacks=[relevance])
   with tru_recorder as recording:
       llm_response = rag_chain({'context': format_docs(docs), 'question': 'Your question here'})

6. Explore Results:
   Use the TruLens dashboard to explore the results:

   from trulens.dashboard import run_dashboard
   run_dashboard(session)  # This opens a local Streamlit app to explore the results

These modifications will help you integrate TruLens with your Langchain setup using LiteLLM, allowing you to evaluate the Conversational Retrieval Chain effectively.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}