Skip to content

Commit 7f35d29

Browse files
ilya-kolchinskyMichaelCliffordShreyanand
authored
RAG notebooks (#53)
* Add the first draft of the direct + agentic RAG demo. The demo should be run as a part of the Llama Stack Playground. * Added Jupyter notebooks for the RAG demos. * Update demos/rag_agentic/notebooks/Level1_foundational_RAG.ipynb Co-authored-by: Michael Clifford <mcliffor@redhat.com> * Update demos/rag_agentic/notebooks/Level1_foundational_RAG.ipynb Co-authored-by: Michael Clifford <mcliffor@redhat.com> * Update demos/rag_agentic/notebooks/Level3_agentic_RAG.ipynb Co-authored-by: Michael Clifford <mcliffor@redhat.com> * Update demos/rag_agentic/notebooks/Level3_agentic_RAG.ipynb Co-authored-by: Michael Clifford <mcliffor@redhat.com> * Changed the model to Granite for consistency. * Removed unneeded imports. * Highlighted Milvus as our preferred vector DB choice. * Update demos/rag_agentic/notebooks/Level3_agentic_RAG.ipynb Co-authored-by: Shrey <shanand@redhat.com> --------- Co-authored-by: Michael Clifford <mcliffor@redhat.com> Co-authored-by: Shrey <shanand@redhat.com>
1 parent 0ef5520 commit 7f35d29

2 files changed

Lines changed: 508 additions & 0 deletions

File tree

‎demos/rag_agentic/notebooks/Level1_foundational_RAG.ipynb‎

Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "45fc9086-93aa-4645-8ba2-380c3acbbed9",
6+
"metadata": {},
7+
"source": [
8+
"# Level 1: Foundational RAG\n",
9+
"\n",
10+
"This tutorial presents an example of executing queries with foundational (i.e., non-agentic) RAG in Llama Stack. It shows how the APIs provided by Llama Stack can be used to directly control and invoke all RAG stages, including indexing, retrieval and inference. \n",
11+
"For an agentic RAG tutorial, please refer to [Level3_agentic_RAG.ipynb](demos/rag_agentic/notebooks/Level3_agentic_RAG.ipynb).\n",
12+
"\n",
13+
"## Overview\n",
14+
"\n",
15+
"This tutorial covers the following steps:\n",
16+
"1. Connecting to a llama-stack server.\n",
17+
"2. Indexing a collection of documents in a vector DB for later retrieval.\n",
18+
"3. Executing the built-in RAG tool to retrieve the document chunks relevant to a given query.\n",
19+
"4. Using the retrieved context to answer user queries during the inference step.\n",
20+
"\n",
21+
"\n",
22+
"## Prerequisites\n",
23+
"\n",
24+
"Before starting, ensure you have a running instance of the Llama Stack server (local or remote) with at least one preconfigured vector DB. For more information, please refer to the corresponding [Llama Stack tutorials](https://llama-stack.readthedocs.io/en/latest/getting_started/index.html)."
25+
]
26+
},
27+
{
28+
"cell_type": "markdown",
29+
"id": "6db34e4b-ed29-4007-b760-59543d4caca1",
30+
"metadata": {},
31+
"source": [
32+
"## 1. Setting Up the Environment\n",
33+
"- Import the necessary libraries.\n",
34+
"- Define the settings for the RAG pipeline, including the Llama Stack server URL, inference and document ingestion parameters.\n",
35+
"- Initialize the connection to the server."
36+
]
37+
},
38+
{
39+
"cell_type": "code",
40+
"execution_count": 3,
41+
"id": "854e7cb4-aed9-4098-adc1-a66f4c9e6ce3",
42+
"metadata": {
43+
"tags": []
44+
},
45+
"outputs": [],
46+
"source": [
47+
"import os\n",
48+
"import uuid\n",
49+
"\n",
50+
"from llama_stack_client import Agent, AgentEventLogger, RAGDocument, LlamaStackClient\n",
51+
"\n",
52+
"# the server endpoint\n",
53+
"LLAMA_STACK_SERVER_URL = \"http://localhost:8321\"\n",
54+
"\n",
55+
"# inference settings\n",
56+
"MODEL_ID = \"ibm-granite/granite-3.2-8b-instruct\"\n",
57+
"SYSTEM_PROMPT = \"You are a helpful assistant. \"\n",
58+
"TEMPERATURE = 0.0\n",
59+
"TOP_P = 0.95\n",
60+
"\n",
61+
"# RAG settings\n",
62+
"VECTOR_DB_EMBEDDING_MODEL = \"all-MiniLM-L6-v2\"\n",
63+
"VECTOR_DB_EMBEDDING_DIMENSION = 384\n",
64+
"VECTOR_DB_CHUNK_SIZE = 512\n",
65+
"\n",
66+
"# For this demo, we are using Milvus Lite, which is our preferred solution. Any other Vector DB supported by Llama Stack can be used.\n",
67+
"VECTOR_DB_PROVIDER_ID = 'milvus'\n",
68+
"\n",
69+
"# initialize the inference strategy\n",
70+
"if TEMPERATURE > 0.0:\n",
71+
" strategy = {\"type\": \"top_p\", \"temperature\": TEMPERATURE, \"top_p\": TOP_P}\n",
72+
"else:\n",
73+
" strategy = {\"type\": \"greedy\"}\n",
74+
" \n",
75+
"# initialize the document collection to be used for RAG\n",
76+
"vector_db_id = f\"test_vector_db_{uuid.uuid4()}\"\n",
77+
" \n",
78+
"# initialize the server connection\n",
79+
"client = LlamaStackClient(base_url=os.environ.get(\"LLAMA_STACK_ENDPOINT\", LLAMA_STACK_SERVER_URL))"
80+
]
81+
},
82+
{
83+
"cell_type": "markdown",
84+
"id": "9203de51-f570-44ab-8130-36333a54888b",
85+
"metadata": {},
86+
"source": [
87+
"## 2. Indexing the Documents\n",
88+
"- Initialize a new document collection in the target vector DB. All parameters related to the vector DB, such as the embedding model and dimension, must be specified here.\n",
89+
"- Provide a list of document URLs to the RAG tool. Llama Stack will handle fetching, conversion and chunking of the documents' content."
90+
]
91+
},
92+
{
93+
"cell_type": "code",
94+
"execution_count": 6,
95+
"id": "8d81ffb2-2089-4cb8-adae-f32965f206c7",
96+
"metadata": {
97+
"tags": []
98+
},
99+
"outputs": [],
100+
"source": [
101+
"# define and register the document collection to be used\n",
102+
"client.vector_dbs.register(\n",
103+
" vector_db_id=vector_db_id,\n",
104+
" embedding_model=VECTOR_DB_EMBEDDING_MODEL,\n",
105+
" embedding_dimension=VECTOR_DB_EMBEDDING_DIMENSION,\n",
106+
" provider_id=VECTOR_DB_PROVIDER_ID,\n",
107+
")\n",
108+
"\n",
109+
"# ingest the documents into the newly created document collection\n",
110+
"urls = [\n",
111+
" (\"https://www.openshift.guide/openshift-guide-screen.pdf\", \"application/pdf\"),\n",
112+
" (\"https://www.cdflaborlaw.com/_images/content/2023_OCBJ_GC_Awards_Article.pdf\", \"application/pdf\"),\n",
113+
"]\n",
114+
"documents = [\n",
115+
" RAGDocument(\n",
116+
" document_id=f\"num-{i}\",\n",
117+
" content=url,\n",
118+
" mime_type=url_type,\n",
119+
" metadata={},\n",
120+
" )\n",
121+
" for i, (url, url_type) in enumerate(urls)\n",
122+
"]\n",
123+
"client.tool_runtime.rag_tool.insert(\n",
124+
" documents=documents,\n",
125+
" vector_db_id=vector_db_id,\n",
126+
" chunk_size_in_tokens=VECTOR_DB_CHUNK_SIZE,\n",
127+
")"
128+
]
129+
},
130+
{
131+
"cell_type": "markdown",
132+
"id": "d5639413-90d6-42ae-add4-6c89da0297e2",
133+
"metadata": {},
134+
"source": [
135+
"## 3. Executing Queries via the Built-in RAG Tool\n",
136+
"- Directly invoke the RAG tool to query the vector DB we ingested into at the previous stage.\n",
137+
"- Construct an extended prompt using the retrieved chunks.\n",
138+
"- Query the model with the extended prompt.\n",
139+
"- Output the reply received from the model."
140+
]
141+
},
142+
{
143+
"cell_type": "code",
144+
"execution_count": 24,
145+
"id": "0d39ab00-2a65-4b72-b5ed-4dd61f1204a2",
146+
"metadata": {
147+
"tags": []
148+
},
149+
"outputs": [
150+
{
151+
"name": "stdout",
152+
"output_type": "stream",
153+
"text": [
154+
"\n",
155+
"User> How to install OpenShift?\n",
156+
"inference> To install OpenShift, you can follow these steps:\n",
157+
"\n",
158+
"1. Download and install the OpenShift CLI (Command Line Interface) tool from the official Red Hat website.\n",
159+
"2. Create a new project in your OpenShift cluster using the `oc new-project` command.\n",
160+
"3. Install the OpenShift client tools on your system by running the `oc setup` command.\n",
161+
"4. Log in to your OpenShift cluster using the `oc login` command with your credentials.\n",
162+
"5. Create a new application in your project using the `oc new-app` command, specifying the image URL of the container you want to deploy.\n",
163+
"6. Once the application is created, you can access it through the OpenShift web console or by running the `oc expose` command.\n",
164+
"\n",
165+
"Alternatively, you can also use the odo tool provided by Red Hat to create and deploy applications on OpenShift. The odo tool allows you to create applications using \"Devfiles,\" which are YAML files that contain information about your application's programming language, dependencies, and other essential details.\n",
166+
"\n",
167+
"You can download the odo tool from the \"Help\" menu on the OpenShift web console through the \"Command line tools\" entry. Once installed, you can use the `odo init` command to create a new application using various programming languages, and then run the `odo push` command to build and deploy the container to the OpenShift container registry.\n",
168+
"\n",
169+
"Note: The above steps are general instructions and may vary depending on your specific environment and configuration. It's recommended to refer to the official Red Hat documentation for more detailed instructions and troubleshooting guides.\n",
170+
"User> Are employees based in California eligible for remote work?\n",
171+
"inference> Based on the provided knowledge search tool results, it appears that there are specific regulations and considerations for employers in California regarding remote work. According to the results, California has laws that require employers to reimburse employees for necessary business expenditures or losses incurred by the employee in direct consequence of the discharge of their duties, including internet usage, home electricity, computer equipment, and cell phone usage.\n",
172+
"\n",
173+
"Additionally, there are specific requirements for data security and privacy, as California has strong data protection laws such as the California Consumer Privacy Act (CCPA) and the California Privacy Rights Act (CPRA). Employers must safeguard sensitive information and comply with these regulations, which apply regardless of where the employee works.\n",
174+
"\n",
175+
"However, it is not explicitly stated in the provided results whether employees based in California are eligible for remote work. The results seem to focus more on the responsibilities and considerations for employers rather than the eligibility of employees for remote work.\n",
176+
"\n",
177+
"Therefore, without further information or clarification, I would say that the answer to the question \"Are employees based in California eligible for remote work?\" is not explicitly stated in the provided knowledge search tool results."
178+
]
179+
}
180+
],
181+
"source": [
182+
"queries = [\n",
183+
" \"How to install OpenShift?\",\n",
184+
" \"Are employees based in California eligible for remote work?\",\n",
185+
"]\n",
186+
"\n",
187+
"for prompt in queries:\n",
188+
" print(f\"\\nUser> {prompt}\")\n",
189+
" \n",
190+
" # RAG retrieval call\n",
191+
" rag_response = client.tool_runtime.rag_tool.query(content=prompt, vector_db_ids=[vector_db_id])\n",
192+
"\n",
193+
" # the list of messages to be sent to the model must start with the system prompt\n",
194+
" messages = [{\"role\": \"system\", \"content\": SYSTEM_PROMPT}]\n",
195+
"\n",
196+
" # construct the actual prompt to be executed, incorporating the original query and the retrieved content\n",
197+
" prompt_context = rag_response.content\n",
198+
" extended_prompt = f\"Please answer the given query using the context below.\\n\\nCONTEXT:\\n{prompt_context}\\n\\nQUERY:\\n{prompt}\"\n",
199+
" messages.append({\"role\": \"user\", \"content\": extended_prompt})\n",
200+
"\n",
201+
" # use Llama Stack inference API to directly communicate with the desired model\n",
202+
" response = client.inference.chat_completion(\n",
203+
" messages=messages,\n",
204+
" model_id=MODEL_ID,\n",
205+
" sampling_params={\n",
206+
" \"strategy\": strategy,\n",
207+
" },\n",
208+
" stream=True,\n",
209+
" )\n",
210+
" \n",
211+
" # print the response\n",
212+
" print(\"inference> \", end='')\n",
213+
" for chunk in response:\n",
214+
" response_delta = chunk.event.delta\n",
215+
" if isinstance(response_delta, TextDelta):\n",
216+
" print(response_delta.text, end='')\n",
217+
" elif isinstance(response_delta, ToolCallDelta):\n",
218+
" print(response_delta.tool_call, end='')"
219+
]
220+
},
221+
{
222+
"cell_type": "markdown",
223+
"id": "df6937a3-3efa-4b66-aaf0-85d96b6d43db",
224+
"metadata": {},
225+
"source": [
226+
"## Key Takeaways\n",
227+
"This tutorial demonstrates how to set up and use the built-in RAG tool for ingesting user-provided documents in a vector DB and later utilizing them during inference via direct retrieval. Please check out our [complementary tutorial]([Level3_agentic_RAG.ipynb](demos/rag_agentic/notebooks/Level3_agentic_RAG.ipynb) for an agentic RAG example."
228+
]
229+
}
230+
],
231+
"metadata": {
232+
"kernelspec": {
233+
"display_name": "Python 3 (ipykernel)",
234+
"language": "python",
235+
"name": "python3"
236+
},
237+
"language_info": {
238+
"codemirror_mode": {
239+
"name": "ipython",
240+
"version": 3
241+
},
242+
"file_extension": ".py",
243+
"mimetype": "text/x-python",
244+
"name": "python",
245+
"nbconvert_exporter": "python",
246+
"pygments_lexer": "ipython3",
247+
"version": "3.10.9"
248+
}
249+
},
250+
"nbformat": 4,
251+
"nbformat_minor": 5
252+
}

0 commit comments

Comments
 (0)