TruLens Feedback on Non-string Output

[viquangly]

I am building Agents using the Agno package. The response from an agent object is in the form of a RunResponse object, with the answer being stored in the content attribute, instead of a str.

How do I build a Feedback for this using provider.relevance_with_cot as an example?

[dosubot[bot]]

Hi @viquangly! I'm Dosu and I’m helping the trulens team.

You can build a Feedback for a RunResponse object by customizing the selector to extract the content attribute, since TruLens feedback functions (like provider.relevance_with_cot) expect a string by default but are flexible with selectors. Here’s how you can do it:

from trulens_eval import Feedback

# Assuming your RunResponse object has a .content attribute with the answer
f_relevance = Feedback(
    provider.relevance_with_cot, 
    name="Answer Relevance"
).on(lambda record: record.output.content)

This lambda tells TruLens to use the content attribute from your RunResponse object as the output for feedback evaluation. You can adapt this pattern for any custom output structure by changing the selector accordingly. This approach is supported by the modular design of TruLens feedback functions and selectors, which are intended to handle non-string outputs and custom agent frameworks like Agno. For more on this pattern, see the discussion and examples here and here.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[viquangly]

@dosu The code did not work. I got a ValueError: Expected a Lens but got '<function >' of type function

[dosubot[bot]]

The error happens because TruLens expects a Lens object, not a lambda, to select fields from your output. To extract the content attribute from your RunResponse, use a Lens path like this:

from trulens_eval import Feedback
from trulens_eval.utils.serial import Lens

f_relevance = Feedback(
    provider.relevance_with_cot,
    name="Answer Relevance"
).on(Lens().output.content)

This tells TruLens to use the output.content field for feedback evaluation, which matches your RunResponse structure. More details on this requirement are in the Feedback class implementation and Lens usage.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[viquangly]

@dosu When I check the record Feedback result, I only see a Futures object. How do I actually see the Feedback result?

[dosubot[bot]]

When you see a Futures object as the feedback result, it means the computation is running asynchronously and hasn't finished yet. To get the actual feedback value, you should use the wait_for_feedback_results() method on your record. This will block until all feedback computations are done and return a dictionary of feedback names and their results. For example:

results = record.wait_for_feedback_results()
for feedback_name, feedback_result in results.items():
    print(feedback_name, feedback_result.result)

This ensures you get the computed values, not just the Futures. If you have a raw Future object, you can also call .result() on it to block until the result is ready, but using wait_for_feedback_results() is the recommended approach in TruLens for feedback evaluation workflows.
Reference

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}