Skip to content

Commit c36c503

Browse files
minmin-intelpre-commit-ci[bot]
minmin-intel
and
pre-commit-ci[bot]
authored
Add SQL agent strategy (opea-project#975)
* initial code for sql agent llama Signed-off-by: minmin-intel <[email protected]> * add test for sql agent Signed-off-by: minmin-intel <[email protected]> * update sql agent test Signed-off-by: minmin-intel <[email protected]> * fix bugs and use vllm to test sql agent Signed-off-by: minmin-intel <[email protected]> * add tag-bench test and google search tool Signed-off-by: minmin-intel <[email protected]> * test sql agent with hints Signed-off-by: minmin-intel <[email protected]> * fix bugs for sql agent with hints and update test Signed-off-by: minmin-intel <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add readme for sql agent and fix ci bugs Signed-off-by: minmin-intel <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add sql agent using openai models Signed-off-by: minmin-intel <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bugs in sql agent openai Signed-off-by: minmin-intel <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * make wait time longer for sql agent microservice to be ready Signed-off-by: minmin-intel <[email protected]> * update readme Signed-off-by: minmin-intel <[email protected]> * fix test bug Signed-off-by: minmin-intel <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * skip planexec with vllm due to vllm-gaudi bug Signed-off-by: minmin-intel <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * debug ut issue Signed-off-by: minmin-intel <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * use vllm for all uts Signed-off-by: minmin-intel <[email protected]> * debug ci issue Signed-off-by: minmin-intel <[email protected]> * change vllm port Signed-off-by: minmin-intel <[email protected]> * update ut Signed-off-by: minmin-intel <[email protected]> * remove tgi server Signed-off-by: minmin-intel <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * align vllm port Signed-off-by: minmin-intel <[email protected]> --------- Signed-off-by: minmin-intel <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 6acefae commit c36c503

33 files changed

+2218
-64
lines changed

‎comps/agent/langchain/README.md‎

Lines changed: 15 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,11 @@ We currently support the following types of agents:
1111
1. ReAct: use `react_langchain` or `react_langgraph` or `react_llama` as strategy. First introduced in this seminal [paper](https://arxiv.org/abs/2210.03629). The ReAct agent engages in "reason-act-observe" cycles to solve problems. Please refer to this [doc](https://python.langchain.com/v0.2/docs/how_to/migrate_agent/) to understand the differences between the langchain and langgraph versions of react agents. See table below to understand the validated LLMs for each react strategy.
1212
2. RAG agent: use `rag_agent` or `rag_agent_llama` strategy. This agent is specifically designed for improving RAG performance. It has the capability to rephrase query, check relevancy of retrieved context, and iterate if context is not relevant. See table below to understand the validated LLMs for each rag agent strategy.
1313
3. Plan and execute: `plan_execute` strategy. This type of agent first makes a step-by-step plan given a user request, and then execute the plan sequentially (or in parallel, to be implemented in future). If the execution results can solve the problem, then the agent will output an answer; otherwise, it will replan and execute again.
14+
4. SQL agent: use `sql_agent_llama` or `sql_agent` strategy. This agent is specifically designed and optimized for answering questions aabout data in SQL databases. For more technical details read descriptions [here](src/strategy/sqlagent/README.md).
1415

1516
**Note**:
1617

17-
1. Due to the limitations in support for tool calling by TGI and vllm, we have developed subcategories of agent strategies (`rag_agent_llama` and `react_llama`) specifically designed for open-source LLMs served with TGI and vllm.
18+
1. Due to the limitations in support for tool calling by TGI and vllm, we have developed subcategories of agent strategies (`rag_agent_llama`, `react_llama` and `sql_agent_llama`) specifically designed for open-source LLMs served with TGI and vllm.
1819
2. For advanced developers who want to implement their own agent strategies, please refer to [Section 5](#5-customize-agent-strategy) below.
1920

2021
### 1.2 LLM engine
@@ -25,14 +26,16 @@ Agents use LLM for reasoning and planning. We support 3 options of LLM engine:
2526
2. Open-source LLMs served with vllm. Follow the instructions in [Section 2.2.2](#222-start-agent-microservices-with-vllm).
2627
3. OpenAI LLMs via API calls. To use OpenAI llms, specify `llm_engine=openai` and `export OPENAI_API_KEY=<your-openai-key>`
2728

28-
| Agent type | `strategy` arg | Validated LLMs (serving SW) | Notes |
29-
| ---------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
30-
| ReAct | `react_langchain` | [llama3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) (tgi-gaudi) | Only allows tools with one input variable |
31-
| ReAct | `react_langgraph` | GPT-4o-mini, [Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) (vllm-gaudi), | if using vllm, need to specify `--enable-auto-tool-choice --tool-call-parser ${model_parser}`, refer to vllm docs for more info |
32-
| ReAct | `react_llama` | [llama3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) (tgi-gaudi) | Recommended for open-source LLMs |
33-
| RAG agent | `rag_agent` | GPT-4o-mini | |
34-
| RAG agent | `rag_agent_llama` | [llama3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) (tgi-gaudi) | Recommended for open-source LLMs, only allows 1 tool with input variable to be "query" |
35-
| Plan and execute | `plan_execute` | GPT-4o-mini, [Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) (vllm-gaudi), [Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) (vllm-gaudi) | |
29+
| Agent type | `strategy` arg | Validated LLMs (serving SW) | Notes |
30+
| ---------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
31+
| ReAct | `react_langchain` | [llama3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) (tgi-gaudi) | Only allows tools with one input variable |
32+
| ReAct | `react_langgraph` | GPT-4o-mini, [Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) (vllm-gaudi), | if using vllm, need to specify `--enable-auto-tool-choice --tool-call-parser ${model_parser}`, refer to vllm docs for more info |
33+
| ReAct | `react_llama` | [llama3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) (tgi-gaudi) (vllm-gaudi) | Recommended for open-source LLMs |
34+
| RAG agent | `rag_agent` | GPT-4o-mini | |
35+
| RAG agent | `rag_agent_llama` | [llama3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) (tgi-gaudi) (vllm-gaudi) | Recommended for open-source LLMs, only allows 1 tool with input variable to be "query" |
36+
| Plan and execute | `plan_execute` | GPT-4o-mini, [Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) (vllm-gaudi), [Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) (vllm-gaudi) | Currently, due to some issues with guaided decoding of vllm-gaudi, this strategy does not work properly with vllm-gaudi. We are actively debugging. Stay tuned. In the meanwhile, you can use OpenAI's models with this strategy. |
37+
| SQL agent | `sql_agent_llama` | [llama3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) (vllm-gaudi) | database query tool is natively integrated using Langchain's [QuerySQLDataBaseTool](https://python.langchain.com/api_reference/community/tools/langchain_community.tools.sql_database.tool.QuerySQLDataBaseTool.html#langchain_community.tools.sql_database.tool.QuerySQLDataBaseTool). User can also register their own tools with this agent. |
38+
| SQL agent | `sql_agent` | GPT-4o-mini | database query tool is natively integrated using Langchain's [QuerySQLDataBaseTool](https://python.langchain.com/api_reference/community/tools/langchain_community.tools.sql_database.tool.QuerySQLDataBaseTool.html#langchain_community.tools.sql_database.tool.QuerySQLDataBaseTool). User can also register their own tools with this agent. |
3639

3740
### 1.3 Tools
3841

@@ -120,12 +123,12 @@ Once microservice starts, user can use below script to invoke.
120123
121124
```bash
122125
curl http://${ip_address}:9090/v1/chat/completions -X POST -H "Content-Type: application/json" -d '{
123-
"query": "What is the weather today in Austin?"
126+
"query": "What is OPEA project?"
124127
}'
125128
126129
# expected output
127130
128-
data: 'The temperature in Austin today is 78°F.</s>'
131+
data: 'The OPEA project is .....</s>' # just showing partial example here.
129132
130133
data: [DONE]
131134
@@ -210,4 +213,4 @@ data: [DONE]
210213
## 5. Customize agent strategy
211214

212215
For advanced developers who want to implement their own agent strategies, you can add a separate folder in `src\strategy`, implement your agent by inherit the `BaseAgent` class, and add your strategy into the `src\agent.py`. The architecture of this agent microservice is shown in the diagram below as a reference.
213-
![Architecture Overview](agent_arch.jpg)
216+
![Architecture Overview](assets/agent_arch.jpg)

‎comps/agent/langchain/agent_arch.jpg‎ renamed to ‎comps/agent/langchain/assets/agent_arch.jpg‎

File renamed without changes.

‎comps/agent/langchain/assets/sql_agent.png‎

246 KB
Loading

‎comps/agent/langchain/assets/sql_agent_llama.png‎

284 KB
Loading

‎comps/agent/langchain/log‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
[HumanMessage(content="what's the most recent album from the founder of ysl records?", id='cfde4aba-0464-4ad9-bd1c-d3fc40bbb46e'), AIMessage(content='', addi
2+
tional_kwargs={'tool_calls': [ChatCompletionOutputToolCall(function=ChatCompletionOutputFunctionDefinition(arguments={'query': 'founder of YSL Records'}, name
3+
e='duckduckgo_search', description=None), id='142bd66e-7ec6-4381-bcb6-2d0bd4fcecd3', type='function')]}, id='6430421d-2238-452b-9fbe-4d6cc8bc0cd8', tool_call
4+
s=[{'name': 'duckduckgo_search', 'args': {'query': 'founder of YSL Records'}, 'id': '142bd66e-7ec6-4381-bcb6-2d0bd4fcecd3', 'type': 'tool_call'}]), ToolMessa
5+
ge(content='Prosecutors allege the chart-topping artist\'s label Young Stoner Life Records also stands for Young Slime ... a YSL co-founder who reached a ple
6+
a deal in 2022 and claimed Young Thug was a part ... In 2016, Young Thug founded YSL Records, with its full name, "Young Stoner Life," perfectly representing
7+
the enigmatic persona of the label. YSL Records quickly became known for its ... The longest criminal trial in the State of Georgia\'s history has been plag
8+
ued by endless problems, including the recusal of not one but two judges in charge of the case. Young Thug and his lawyer ... Prosecutors say YSL - the acron
9+
ym for the artist\'s label, Young Stoner Life Records - also stands for Young Slime Life, an Atlanta-based street gang affiliated with the national Bloods ga
10+
ng. YSL Records calls its roster of artists the "Slime Family." ... YSL co-founder Walter Murphy entered a guilty plea on a single count of conspiracy to vio
11+
late the state\'s Racketeer Influenced ...', name='duckduckgo_search', id='1d9ebf6b-bf62-469a-8d45-365c0fa15bba', tool_call_id='142bd66e-7ec6-4381-bcb6-2d0bd
12+
4fcecd3'), HumanMessage(content='Retrieved document is not sufficient or relevant to answer the query. Reformulate the query to search knowledge base again.'
13+
, id='6f2fb9db-e5b4-4483-b69a-06122c996eb9'), AIMessage(content='', additional_kwargs={'tool_calls': [ChatCompletionOutputToolCall(function=ChatCompletionOut
14+
putFunctionDefinition(arguments={'query': 'YSL Records founder'}, name='duckduckgo_search', description=None), id='6d073578-5bf1-449e-8d32-4f3b5f999a06', typ
15+
e='function')]}, id='50a644af-3117-4513-be56-5fb4b16d0481', tool_calls=[{'name': 'duckduckgo_search', 'args': {'query': 'YSL Records founder'}, 'id': '6d0735
16+
78-5bf1-449e-8d32-4f3b5f999a06', 'type': 'tool_call'}]), ToolMessage(content='That same year, he founded the YSL record label, which the rapper has used to p
17+
ropel close friends and family members to industry success. The Birth Of YSL Records And Its Impact On The Music Industry In 2016, Young Thug founded YSL Rec
18+
ords, with its full name, "Young Stoner Life," perfectly representing the enigmatic persona of ... Young Thug, who runs the Young Stoner Life label, has been
19+
accused of co-founding the Young Slime Life Atlanta gang and violating the RICO act, among other charges. Here\'s what to know about the ... The rapper foun
20+
dead the record label Young Stoner Life in 2016 as an imprint of 300 Entertainment. YSL Records calls its roster of artists the "Slime Family." One of the YSL
21+
Records founder\'s charges includes conspiracy to violate the Racketeer Influenced and Corrupt Organizations Act (RICO).', name='duckduckgo_search', id='afc
22+
cee2f-f36f-4d86-8abd-8d05d733649d', tool_call_id='6d073578-5bf1-449e-8d32-4f3b5f999a06'), HumanMessage(content='Retrieved document is not sufficient or relev
23+
ant to answer the query. Reformulate the query to search knowledge base again.', id='22af0af8-92d0-4876-ad4e-16457c37f1f3'), AIMessage(content='', additional
24+
_kwargs={'tool_calls': [ChatCompletionOutputToolCall(function=ChatCompletionOutputFunctionDefinition(arguments={'query': 'latest album by Young Thug'}, name=
25+
'duckduckgo_search', description=None), id='2315fb35-11ec-4a6b-be86-405a1efef411', type='function')]}, id='189ba925-9f82-4e9e-bdab-6dcfc49a196e', tool_calls=
26+
[{'name': 'duckduckgo_search', 'args': {'query': 'YSL Records founder name'}, 'id': 'cddeca43-88e2-4279-90aa-4c0b58b8b3c4', 'type': 'tool_call'}, {'name': 'd
27+
uckduckgo_search', 'args': {'query': 'latest album by Young Thug'}, 'id': '2315fb35-11ec-4a6b-be86-405a1efef411', 'type': 'tool_call'}]), ToolMessage(content
28+
='In 2016, Young Thug founded YSL Records, with its full name, "Young Stoner Life," perfectly representing the enigmatic persona of the label. YSL Records qu
29+
ickly became known for its ... Thug is also the founder of Young Stoner Life Records. However, the rapper was arrested in May 2022 and charged with conspirin
30+
g to violate the state\'s Racketeer Influenced and Corrupt Organizations (RICO) Act, according to. Prosecutors alleged that YSL was a street gang connected t
31+
o a sleuth of crimes. Prosecutors say YSL - the acronym for the artist\'s label, Young Stoner Life Records - also stands for Young Slime Life, an Atlanta-bas
32+
ed street gang affiliated with the national Bloods gang. Trontavious Stephens, a co-founder of YSL, took the witness stand for hours Wednesday and Thursday.
33+
Stephens testified that he had a criminal record, but he said he did not commit crimes with ... Young Thug\'s arrest sent shockwaves through the rap communit
34+
y. On May 9, 2022, the YSL rapper was apprehended along with 27 other alleged gang members as part of a sprawling 56-count indictment ...', name='duckduckgo_
35+
search', id='153464f0-aee1-4097-aae6-191ffa372aa6', tool_call_id='cddeca43-88e2-4279-90aa-4c0b58b8b3c4'), ToolMessage(content='The discography of American ra
36+
pper Young Thug consists of three studio albums, two compilation albums, twelve self-released mixtapes, seven commercial mixtapes, three extended plays, and
37+
sixty-nine singles (including 71 as a featured artist).. In 2015, Thug released his debut mixtape, Barter 6, which reached number 22 on the Billboard 200. Hi
38+
s 2016 mixtape, I\'m Up matched the same position. Atlanta singer (and Thugger\'s rumored girlfriend) also released her own "From a Woman" Friday. Even behin
39+
d bars, Young Thug remains prolific, with the rapper dropping his new single "From a Man ... Mariah The Scientist\'s new song "From A Woman" isn\'t musically
40+
connected to "From A Man," though Mariah\'s cover art works as a flipside to Thug\'s. She sings about finding someone ... Future, Young Thug & More Join Mus
41+
tard On New Album \'Faith Of A Mustard Seed\': Stream. News | Jul 28, 2024, 11:00 AM PDT. HipHopDX brings you all the newest Young Thug albums, songs, and ..
42+
. Find top songs and albums by Young Thug including Lifestyle (feat. Young Thug & Rich Homie Quan), It\'s Up and more. ... JEFFREY to the country-ish sides o
43+
f Beautiful Thugger Girls—has cracked the mainstream, laying the groundwork for a new crop of fellow eccentrics like Lil Uzi Vert and Playboi Carti. In other
44+
words, Thug hasn\'t adjusted ...', name='duckduckgo_search', id='79667433-d46e-4536-a04b-842223008bdc', tool_call_id='2315fb35-11ec-4a6b-be86-405a1efef411')
45+
, HumanMessage(content='Retrieved document is not sufficient or relevant to answer the query. Reformulate the query to search knowledge base again.', id='2f1
46+
b3773-8837-4f14-82b5-0a8bbccf3948'), HumanMessage(content='I don’t know.', id='a08edb17-a41a-458f-903e-0f7e15908e3f')]

‎comps/agent/langchain/requirements.txt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
# used by microservice
22
docarray[full]
3-
4-
#used by tools
5-
duckduckgo-search
63
fastapi
74
huggingface_hub
85
langchain
6+
7+
#used by tools
8+
langchain-google-community
99
langchain-huggingface
1010
langchain-openai
1111
langchain_community

0 commit comments

Comments
 (0)