feat: multimodal content in evaluators#2876
Conversation
|
Mintlify preview ID generated: preview-multim-1772484891-f183e47 |
Kathryn May (katmayb)
force-pushed
the
multimodal-content-evaluators
branch
from
7b64f9a to
5cbf33a
Compare
|
Mintlify preview ID generated: preview-multim-1772486524-1b98060 |
|
Mintlify preview ID generated: preview-multim-1772643135-c02f520 |
Kathryn May (katmayb)
force-pushed
the
multimodal-content-evaluators
branch
from
8d4bb46 to
c785d38
Compare
|
Mintlify preview ID generated: preview-multim-1772661701-f0d5798 |
Kathryn May (katmayb)
changed the title
Kathryn May (katmayb)
requested a review
from Florence Morris (fjmorris)
as a code owner
|
Mintlify preview ID generated: preview-multim-1772724522-db00cdb |
|
Mintlify preview ID generated: preview-multim-1772725052-2c7077a |
| You can also create text-only evaluators that don't use attachments but evaluate the model's text output: | ||
|
|
||
| - OCR → text correction: Use a vision model to extract text from a document, then evaluate the accuracy of the extracted output. | ||
| - Speech-to-text → transcription quality: Use a voice model to transcribe audio to text, then evaluate the transcription against your reference. |
There was a problem hiding this comment.
Can we add a component on how it will take in any base64 components ?
Not all customers will use attachments to render traces, so we allow you to pass in any base64 format. For example, if part of their input / output is in base64 format, if they follow this guide here: https://docs.langchain.com/langsmith/log-multimodal-traces to set up their traces correctly, they will not only be able to visualize them in the UI, if i do variable mapping with {{input}} in the evaluator, they will also be passed correctly to the LLM that supports. I know this section is about atachments, but I think this may be worth calling out here.
We can use the images that I sent in the slack
| 1. Select **+ Evaluator** from the dataset page. | ||
| 1. In the **Template variables** editor, add a variable for the attachment(s) to include: | ||
| - For a single attachment type: Use the suggested variable name. All examples must have an attachment with this name. | ||
| - For multiple attachments or if attachment names vary across examples: Use the `All attachments` variable to include all available attachments for each example. |
There was a problem hiding this comment.
not entirely accurate in temrs of recommednation - I would say something like:
- if you want to include all attachmetns, use the {{attachments}} evaluator
- if you want to include a speicifc attachment, you can use the suggested variable name, such as {{attachment.file_name}}, this will map the file with file_name in the attachment list to passs it to the evaluator
(this is bc for 1 singel attachemnt, they can still do {{attachments}} which is probably easiest.
| - **Using attachments from traces**: Similar to [offline evaluations with attachments](/langsmith/evaluate-with-attachments), you can use attachments from your traces in online evaluations. Since your traces already include attachments logged via the SDK, you can reference them directly in your evaluator. To do so: | ||
|
|
||
| 1. In your evaluator configuration, click the file icon in the evaluator message where you want to add multimodal content. | ||
| 1. In the **Template variables** tab, add a variable for the attachment(s) to include: |
There was a problem hiding this comment.
update this section to mirror the above
|
Mintlify preview ID generated: preview-multim-1772734590-4b8b514 |
|
Mintlify preview ID generated: preview-multim-1772735417-638e527 |
2 participants
Add docs for multimodal content for UI in evaluators.
Preview
https://langchain-5e9cc07a-preview-multim-1772661701-f0d5798.mintlify.app/langsmith/evaluate-with-attachments#3-define-custom-evaluators