RAG implementation tips?

[flintlouis]

I am trying to create an Agent that utilizes RAG using Microsoft.Extensions.AI IChatClient and Microsoft.SemanticKernel.Text TextChunker.SplitPlainTextParagraphs(). I followed the https://github.com/SteveSandersonMS/dotnet-ai-workshop tutorials.

What I would like to learn more about:

1. If you have a folder structure: DataTypes/Number.md. When uploading the Number.md file in chunks to the vector store, the parent folder name is not included in the vector. Is there a best practice to keep that information? Add it to the beginning of each chunk?
2. What is the best practice what to put in the instructions of the agent? My agent only does one search in the documentation while in the demo I see it does multiple searches.
3. I was wondering if there are other tutorials that go a bit more in dept on how to implement preprocessing for markdown files or other file types?

I hope this makes sense.
Thankyou

[moonbox3]

Adding @westey-m for visibility.

[westey-m]

@flintlouis some thoughts:

1. Yes, it can certainly be good to include additional context in the vector you are going to search for, but it will depend on whether that information will help match the user input. If not, then don't include it. If you just need a way to refer back to the source for citations, you can include the source file info in other fields on the same record in the DB.
2. Searching can take one of two forms. You can either search automatically before each agent invocation, or you can register a function to do search and allow the agent to choose whether to invoke that function. With the first option, you could inject the search results into the agent via additional instructions and this will allow you to search each time you invoke. With the second the number of invocations will be determined by the LLM/Agent.
3. KernelMemory has a Markdown chunker that you could look at: https://github.com/microsoft/kernel-memory/blob/main/extensions/Chunkers/Chunkers/MarkDownChunker.cs. It also has various loaders for reading different file types: https://github.com/microsoft/kernel-memory/tree/main/service/Core/DataFormats

Also check out the semantic kernel vector store connectors, which are implementing the Microsoft.Extensions.VectorData abstractions: https://learn.microsoft.com/en-us/semantic-kernel/concepts/vector-store-connectors. These make working with different vector stores and switching between them really easy.

I am also currently working on a feature that should make RAG with SK Agents much simpler and easier to use, which should release as experimental in the coming few of weeks.

[flintlouis]

Thankyou so much for taking the time to reply! Looking forward to your new feature. Is there anywhere I can subscribe to stay up to date when this new feature is released?

[westey-m]

Sure, we'll definitely publish a post about it on our blog when it's released:
https://devblogs.microsoft.com/semantic-kernel/

We also talk about any new features in our weekly office hours, which are free for anyone to join:
https://devblogs.microsoft.com/semantic-kernel/semantic-kernel-office-hours-recordings/

If you want to take a closer look in the mean time, this is the draft PR for the feature branch I'm working on: #11689
And more specifically the following PR into the feature branch contains the first part of the RAG story: #11774
Feel free to leave feedback on the PRs if you have any thoughts.

I'll be adding some samples soon too, so if it's a bit unclear right now how the components fit together that will help.

[flintlouis]

Thanks @westey-m
I have one more question you can maybe shed some light on. There is also Microsoft.Extensions.AI, which with you can also use to build agents and RAG with. I am trying to understand the correlation between the two, which one to use when. They seem to give the same functionality but in different ways. How to choose which one to use when?

[westey-m]

@flintlouis, Microsoft.Extensions.AI is an evolution of a subset of Semantic Kernel, specifically the AI Service abstractions that Semantic Kernel provides. The two are however interoperable. In fact, in many cases Semantic Kernel is already built on top of Microsoft.Extensions.AI. It is possible to convert between the abstractions too. E.g. SK provides the IChatCompletionService interface while Microsoft.Extensions.AI provides the IChatClient interface. It's possible to convert from one to the other using the AsIChatClient and AsChatCompletionService extensions methods.

See: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/SemanticKernel.Abstractions/AI/ChatCompletion/ChatCompletionServiceExtensions.cs#L123

[flintlouis]

Hi @westey-m, I have been playing around a bit with https://github.com/microsoft/kernel-memory/blob/main/extensions/Chunkers/Chunkers/MarkDownChunker.cs and I had another question for you if that is oke.
Before I was converting markdown files into plain text and then using the TextChunker.SplitPlainTextParagraphs(text) to chunk the plain text, assuming this worked a bit like semantic chunking. When giving the max token per chunk there it would stop whenever it thought the relevant paragraph was over.
When looking at the MarkDownChunker it doesn't look like it does that, am I correct? It just looks like it take the amount of max tokens specified. So it doesn't really do semantic chunking. Am I understanding this correctly?
Why should I not just use TextChunker.SplitMarkdownParagraphs() from SemanticKernel.Text?

Example what happens in my case:

Chunk1:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

### Title

Chunk2:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

Here it doesn't include the title with chunk2