Trulens with LangGraph??

[carloss4dv]

Hi there i have a langgraph agent that i am looking forwar into evualting it with trulens.

Im new to this tool.

Any suggestions???

Thanks for developing this awesome tool!!!

[dosubot[bot]]

Hi there @carloss4dv! I'm Dosu, your friendly bot here to help you with bugs, answer questions, and guide you toward becoming a contributor while we wait for a human maintainer. Excited to assist you! 🚀

To evaluate a LangGraph agent using Trulens, you can follow these steps:

1. Import Required Libraries: Make sure to import the necessary modules from Langchain and Trulens.

   # !pip install trulens_eval langchain==0.0.283 yfinance google-search-results
   
   from langchain import LLMMathChain
   from langchain.agents import initialize_agent, Tool, AgentType
   from langchain.chat_models import ChatOpenAI
   from trulens_eval import TruSession, TruChain
   from trulens_eval.feedback import OpenAI as fOpenAI
   
   session = TruSession()

2. Set API Keys: Ensure your OpenAI and SERP API keys are set.

   import os
   os.environ["OPENAI_API_KEY"] = "sk-..."
   os.environ["SERPAPI_API_KEY"] = "..."

3. Create and Wrap the Application: Initialize the language model and math chain, then wrap it with Trulens.

   llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")
   llm_math_chain = LLMMathChain.from_llm(llm, verbose=True)
   
   tools = [
       Tool(
           name="Calculator",
           func=llm_math_chain.run,
           description="useful for when you need to answer questions about math"
       ),
   ]
   
   agent = initialize_agent(tools, llm, agent=AgentType.OPENAI_FUNCTIONS, verbose=True)
   tru_agent = TruChain(agent)

4. Run the Application: Execute the agent with the desired input and record the session.

   with tru_agent as recording:
       agent(inputs={"input": "how much is Euler's number divided by PI"})

5. Explore with TruLens Dashboard: Start the TruLens dashboard to explore the evaluation results.

   from trulens.dashboard import run_dashboard
   
   run_dashboard(session)

These steps will help you evaluate a LangGraph agent using Trulens by running the agent and exploring the results through the dashboard ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[dosubot[bot]]

To integrate Trulens into your custom workflow for testing a LangGraph agent using LLaMA3 and Chroma vector store, you can use the TruLlama class from the Trulens library. Here's how you can set it up:

1. Create your LlamaIndex application: This involves setting up your documents and creating a query engine.

   from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
   
   documents = SimpleDirectoryReader("data").load_data()
   index = VectorStoreIndex.from_documents(documents)
   
   query_engine = index.as_query_engine()

2. Define feedback functions: You can define feedback functions that utilize the specific context produced by the application's retriever.

   from trulens.providers.openai import OpenAI
   from trulens.core import Feedback
   import numpy as np
   
   # Select context to be used in feedback.
   from trulens.apps.llamaindex import TruLlama
   context = TruLlama.select_context(query_engine)
   
   # Use feedback
   f_context_relevance = (
       Feedback(provider.context_relevance_with_context_reasons)
       .on_input()
       .on(context)  # Refers to context defined from `select_context`
       .aggregate(np.mean)
   )

3. Wrap your application in a TruLlama recorder: This will provide logging and evaluation upon the application's use.

   from trulens.apps.llamaindex import TruLlama
   # f_lang_match, f_qa_relevance, f_context_relevance are feedback functions
   tru_recorder = TruLlama(query_engine,
       app_name='LlamaIndex',
       app_version="base",
       feedbacks=[f_lang_match, f_qa_relevance, f_context_relevance])
   
   with tru_recorder as recording:
       query_engine.query("What is llama index?")

This setup allows you to integrate Trulens into your workflow, providing instrumentation, logging, and evaluation capabilities for your LlamaIndex application ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[carloss4dv]

@dosu Thanks i am willing to use Ollama for the feedback recording i would like any suggestions about how to implement it, thanks.

[dosubot[bot]]

To implement feedback recording using Ollama with Trulens, you can follow these steps:

1. Setup and Import Required Libraries:
   Ensure you have the necessary libraries installed and imported:

   from trulens_eval import Feedback, Tru, TruChain
   from langchain.llms import Ollama
   from langchain.chains import LLMChain
   from langchain.prompts import PromptTemplate, ChatPromptTemplate, HumanMessagePromptTemplate
   import litellm
   from trulens_eval.feedback.provider import LiteLLM

2. Initialize the Session:

   tru = Tru()
   tru.reset_database()

3. Set Up the Ollama Model:

   ollama = Ollama(base_url="http://localhost:11434", model="llama2")

4. Create a Simple LLM Application:

   full_prompt = HumanMessagePromptTemplate(
       prompt=PromptTemplate(
           template="Provide a helpful response with relevant background information for the following: {prompt}",
           input_variables=["prompt"],
       )
   )
   chat_prompt_template = ChatPromptTemplate.from_messages([full_prompt])
   chain = LLMChain(llm=ollama, prompt=chat_prompt_template, verbose=True)

5. Initialize Feedback Function(s):

   litellm.set_verbose = False
   ollama_provider = LiteLLM(
       model_engine="ollama/llama3.1:8b", api_base="http://localhost:11434"
   )
   relevance = Feedback(
       ollama_provider.context_relevance_with_context_reasons
   ).on_input_output()

6. Instrument the Chain for Logging:

   tru_recorder = TruChain(
       chain, app_id="Chain1_ChatApplication", feedbacks=[relevance]
   )

7. Record and Display the Response:

   prompt_input = "What is a good name for a store that sells colorful socks?"
   with tru_recorder as recording:
       llm_response = chain(prompt_input)
   display(llm_response)

8. Retrieve and View Feedback:

   tru.get_records_and_feedback(app_ids=[])[0]

9. Log Human Feedback:
   You can also log human feedback by creating a mechanism for users to provide feedback, such as using buttons for thumbs up or thumbs down:

   from ipywidgets import Button, HBox, Label, VBox
   
   thumbs_up_button = Button(description="👍")
   thumbs_down_button = Button(description="👎")
   
   def update_feedback(human_feedback):
       session.add_feedback(
           name="Human Feedback",
           record_id=record.record_id,
           app_id=tru_app.app_id,
           result=human_feedback,
       )
   
   def on_thumbs_up_button_clicked(b):
       update_feedback(human_feedback=1)
       print("👍")
   
   def on_thumbs_down_button_clicked(b):
       update_feedback(human_feedback=0)
       print("👎")
   
   thumbs_up_button.on_click(on_thumbs_up_button_clicked)
   thumbs_down_button.on_click(on_thumbs_down_button_clicked)
   
   VBox([
       Label(record.main_input),
       Label(record.main_output),
       HBox([thumbs_up_button, thumbs_down_button]),
   ])

This setup allows you to record both automated and human feedback using the Ollama model with Trulens, and you can explore the results in a dashboard or directly in your notebook ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[sfc-gh-nvytla]

Hi @carloss4dv, hopefully the advice from Dosu has been helpful! I'm also linking our LangGraph Quickstart notebook, which you may find helpful.

Please feel free to let us know if you have any additional questions, thanks!