Determining the right architecture for a prompt router and RAG system

[adgren]

I'm new to Semantic Kernel and have been tasked with porting our current RAG system to SK. We're using C#.

We work in the legal space and the current RAG system is pretty basic. It consists of:

1. A prompt router that classifies the user's question as either:
   • requiring data from our case document collection in our vector database OR
   • requiring data from our helpdesk collection in our vector database
2. Perform a similarity search in our vector database against the selected collection
3. Feed the results into an LLM to answer the question (The prompt for this part is a pretty involved)
4. Return the answer to the user.

I have built an agent which performs the similarity search using a ChatCompletionAgent, but I'm having trouble figuring out the correct way to build the prompt router and how to hook it up to this agent.

Any direction will be greatly appreciated.

Adam

[westey-m]

Hi @adgren, we have been working on a solution to help folks componentize this type of behavior and plug them into an agent. Note that this functionality is still experimental at the moment, so we may still make changes to the design. We are currently working hard on stabilizing hits though.

The idea is that we support a new type of component called an AI Context Provider. This component can be registered on a thread that is used with an agent. All messages added to the thread (both from the user and agent) are passed to the component, and also, before the underlying agent/llm is invoked, the component has an opportunity to inject additional context into the agent in the form of text instructions or functions.

We actually have an experimental implementation of a RAG AI Context Provider called TextSearchProvider that you can take a look at.
Note that we use the term Text Search (as in text based similarity search) for RAG in SK.
See https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/SemanticKernel.Core/Data/TextSearchBehavior/TextSearchProvider.cs

Here is a sample showing usage:
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Agents/ChatCompletion_Rag.cs

The sample is using a very simple and opinionated TextSearchStore for storing and retrieving data, however you do not need to use TextSearchStore, and may not want to, considering that you already have your own vector stores. If this is the case, I would recommend using the VectorStoreTextSearch class for proxying to the vector store of your choice and choosing your text fields. Both TextSearchStore and VectorStoreTextSearch implement the same text search abstraction ITextSearch, and the TextSearchProvider relies on ITextSearch, making it easy to pick which data source you want.
See https://learn.microsoft.com/en-us/semantic-kernel/concepts/text-search/text-search-vector-stores

If you are unable to rely on the experimental code we have, you can of course do the RAG pipeline outside of the agent, and just inject the search results with instructions into the agent on invocation via the AdditionalInstructions option.

I'm having trouble figuring out the correct way to build the prompt router

If you provide some more info on the specific area of building the prompt router that you would like advice on, I'd be happy to try and help.