Multimodal RAG

[subnetdusk]

Hello everybody, big fan of this project.
I'd like to ask if multimodal RAG support is in the roadmap. Basically allowing to query the vector database using image embeddings from PDFs, retrieve the actual image file (via path in metadata) and pass it to the LLM. That would be a great addition. Ciao!

[pieroit]

Hi @subnetdusk thx and welcome

In v2 there is already the possibility to have multiple llms and embedders, with agents deciding what they want to use.
So having embedders for image, audio and other stuff should not be a problem. Also LLMs can take images as input because every message is made of blocks (included an image block).

A little more complicated is defining a uniform interface for memory, because query and filters can take on so many types and memory usage itself should be contextual (not necessarily the first step as in classic RAG systems).

Not sure, maybe the best choice is to leave memory completely to plugins, hooking agent_prompt_suffix so they can prepare a query as they whish and query as they whish, as long as they return a string to enrich the prompt.

Tell me yours! BTW I confirm this will be possible

[subnetdusk]

Hi Piero, (love your yt channel btw),
that's good news! Regarding the memory, personally I'd like to have a default implementation ('opinionated', as I saw Cheshire Cat AI is described elsewhere), while leaving alternative implementations to plugins. That would be great, to casual users and/or rapid prototyping. Not sure about more advanced users, maybe they would see it as a nuance nuisance idk.