Skip to content

Commit 35bfeae

Browse files
github-actions[bot]github-actions[bot]
authored
Merge pull request #14 from unclecode/main
Auto PR from main to live
2 parents 6ca161d + 739aaa2 commit 35bfeae

11 files changed

+2533
-669
lines changed

‎README.md

+73-242
Large diffs are not rendered by default.

‎README_old.md

+319
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,319 @@
1+
# GroqCall.ai (I changed the name from FunckyCall to GroqCall)
2+
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1q3is7qynCsx4s7FBznCfTMnokbKWIv1F?usp=sharing)
3+
[![Version](https://img.shields.io/badge/version-0.0.1-blue.svg)](https://github.com/unclecode/groqcall)
4+
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
5+
6+
GroqCall is a proxy server provides function call for Groq's lightning-fast Language Processing Unit (LPU) and other AI providers. Additionally, the upcoming FuncyHub will offer a wide range of built-in functions, hosted on the cloud, making it easier to create AI assistants without the need to maintain function schemas in the codebase or or execute them through multiple calls.
7+
8+
## Motivation 🚀
9+
Groq is a startup that designs highly specialized processor chips aimed specifically at running inference on large language models. They've introduced what they call the Language Processing Unit (LPU), and the speed is astounding—capable of producing 500 to 800 tokens per second or more. I've become a big fan of Groq and their community;
10+
11+
12+
I admire what they're doing. It feels like after discovering electricity, the next challenge is moving it around quickly and efficiently. Groq is doing just that for Artificial Intelligence, making it easily accessible everywhere. They've opened up their API to the cloud, but as of now, they lack a function call capability.
13+
14+
Unable to wait for this feature, I built a proxy that enables function calls using the OpenAI interface, allowing it to be called from any library. This engineering workaround has proven to be immensely useful in my company for various projects. Here's the link to the GitHub repository where you can explore and play around with it. I've included some examples in this collaboration for you to check out.
15+
16+
<img width="150" src = "https://res.cloudinary.com/kidocode/image/upload/v1710148127/GroqChip-1-Die_lgi95d.jpg"/>
17+
18+
<img title="Powered by Groq" alt="Powered by Groq" width = "150" src="https://res.cloudinary.com/kidocode/image/upload/v1710142103/Stack_PBG_White_n6qdbj.svg">
19+
20+
21+
22+
## Running the Proxy Locally 🖥️
23+
To run this proxy locally on your own machine, follow these steps:
24+
25+
1. Clone the GitHub repository:
26+
```git clone https://github.com/unclecode/groqcall.git```
27+
28+
2. Navigate to the project directory:
29+
```cd groqcall```
30+
31+
3. Create a virtual environment:
32+
```python -m venv venv```
33+
34+
4. Activate virtual environment:
35+
```source venv/bin/activate```
36+
37+
5. Install the required libraries:
38+
```pip install -r requirements.txt```
39+
40+
6. Run the FastAPI server:
41+
```./venv/bin/uvicorn --app-dir app/ main:app --reload```
42+
43+
44+
## Using the Pre-built Server 🌐
45+
For your convenience, I have already set up a server that you can use temporarily. This allows you to quickly start using the proxy without having to run it locally.
46+
47+
To use the pre-built server, simply make requests to the following base URL:
48+
```https://groqcall.ai/proxy/groq/v1```
49+
50+
51+
## Exploring GroqCall.ai 🚀
52+
This README is organized into three main sections, each showcasing different aspects of GroqCall.ai:
53+
54+
- **Sending POST Requests**: Here, I explore the functionality of sending direct POST requests to LLMs using GroqCall.ai. This section highlights the flexibility and control offered by the library when interacting with LLMs.
55+
- **FuncHub**: The second section introduces the concept of FuncHub, a useful feature that simplifies the process of executing functions. With FuncHub, there is no need to send the function JSON schema explicitly, as the functions are already hosted on the proxy server. This approach streamlines the workflow, allowing developers to obtain results with a single call without having to handle function call is production server.
56+
- **Using GroqCall with PhiData**: In this section, I demonstrate how GroqCall.ai can be seamlessly integrated with other libraries such as my favorite one, the PhiData library, leveraging its built-in tools to connect to LLMs and perform external tool requests.
57+
58+
59+
```python
60+
# The following libraries are optional if you're interested in using PhiData or managing your tools on the client side.
61+
!pip install phidata > /dev/null
62+
!pip install openai > /dev/null
63+
!pip install duckduckgo-search > /dev/null
64+
```
65+
66+
## Sending POST request, with full functions implementation
67+
68+
69+
```python
70+
from duckduckgo_search import DDGS
71+
import requests, os
72+
import json
73+
74+
# Here you pass your own GROQ API key
75+
api_key=userdata.get("GROQ_API_KEY")
76+
header = {
77+
"Authorization": f"Bearer {api_key}",
78+
"Content-Type": "application/json"
79+
}
80+
proxy_url = "https://groqcall.ai/proxy/groq/v1/chat/completions"
81+
82+
83+
def duckduckgo_search(query, max_results=None):
84+
"""
85+
Use this function to search DuckDuckGo for a query.
86+
"""
87+
with DDGS() as ddgs:
88+
return [r for r in ddgs.text(query, safesearch='off', max_results=max_results)]
89+
90+
def duckduckgo_news(query, max_results=None):
91+
"""
92+
Use this function to get the latest news from DuckDuckGo.
93+
"""
94+
with DDGS() as ddgs:
95+
return [r for r in ddgs.news(query, safesearch='off', max_results=max_results)]
96+
97+
function_map = {
98+
"duckduckgo_search": duckduckgo_search,
99+
"duckduckgo_news": duckduckgo_news,
100+
}
101+
102+
request = {
103+
"messages": [
104+
{
105+
"role": "system",
106+
"content": "YOU MUST FOLLOW THESE INSTRUCTIONS CAREFULLY.\n<instructions>\n1. Use markdown to format your answers.\n</instructions>"
107+
},
108+
{
109+
"role": "user",
110+
"content": "Whats happening in France? Summarize top stories with sources, very short and concise."
111+
}
112+
],
113+
"model": "mixtral-8x7b-32768",
114+
"tool_choice": "auto",
115+
"tools": [
116+
{
117+
"type": "function",
118+
"function": {
119+
"name": "duckduckgo_search",
120+
"description": "Use this function to search DuckDuckGo for a query.\n\nArgs:\n query(str): The query to search for.\n max_results (optional, default=5): The maximum number of results to return.\n\nReturns:\n The result from DuckDuckGo.",
121+
"parameters": {
122+
"type": "object",
123+
"properties": {
124+
"query": {
125+
"type": "string"
126+
},
127+
"max_results": {
128+
"type": [
129+
"number",
130+
"null"
131+
]
132+
}
133+
}
134+
}
135+
}
136+
},
137+
{
138+
"type": "function",
139+
"function": {
140+
"name": "duckduckgo_news",
141+
"description": "Use this function to get the latest news from DuckDuckGo.\n\nArgs:\n query(str): The query to search for.\n max_results (optional, default=5): The maximum number of results to return.\n\nReturns:\n The latest news from DuckDuckGo.",
142+
"parameters": {
143+
"type": "object",
144+
"properties": {
145+
"query": {
146+
"type": "string"
147+
},
148+
"max_results": {
149+
"type": [
150+
"number",
151+
"null"
152+
]
153+
}
154+
}
155+
}
156+
}
157+
}
158+
]
159+
}
160+
161+
response = requests.post(
162+
proxy_url,
163+
headers=header,
164+
json=request
165+
)
166+
if response.status_code == 200:
167+
res = response.json()
168+
message = res['choices'][0]['message']
169+
tools_response_messages = []
170+
if not message['content'] and 'tool_calls' in message:
171+
for tool_call in message['tool_calls']:
172+
tool_name = tool_call['function']['name']
173+
tool_args = tool_call['function']['arguments']
174+
tool_args = json.loads(tool_args)
175+
if tool_name not in function_map:
176+
print(f"Error: {tool_name} is not a valid function name.")
177+
continue
178+
tool_func = function_map[tool_name]
179+
tool_response = tool_func(**tool_args)
180+
tools_response_messages.append({
181+
"role": "tool", "content": json.dumps(tool_response)
182+
})
183+
184+
if tools_response_messages:
185+
request['messages'] += tools_response_messages
186+
response = requests.post(
187+
proxy_url,
188+
headers=header,
189+
json=request
190+
)
191+
if response.status_code == 200:
192+
res = response.json()
193+
print(res['choices'][0]['message']['content'])
194+
else:
195+
print("Error:", response.status_code, response.text)
196+
else:
197+
print(message['content'])
198+
else:
199+
print("Error:", response.status_code, response.text)
200+
201+
```
202+
203+
## Schema-less Function Call 🤩
204+
In this method, we only need to provide the function's name, which consists of two parts, acting as a sort of namespace. The first part identifies the library or toolkit containing the functions, and the second part specifies the function's name, assuming it's already available on the proxy server. I aim to collaborate with the community to incorporate all typical functions, eliminating the need for passing a schema. Without having to handle function calls ourselves, a single request to the proxy enables it to identify and execute the functions, retrieve responses from large language models, and return the results to us. Thanks to Groq, all of this occurs in just seconds.
205+
206+
207+
```python
208+
from duckduckgo_search import DDGS
209+
import requests, os
210+
api_key = userdata.get("GROQ_API_KEY")
211+
header = {
212+
"Authorization": f"Bearer {api_key}",
213+
"Content-Type": "application/json"
214+
}
215+
216+
proxy_url = "https://groqcall.ai/proxy/groq/v1/chat/completions"
217+
218+
219+
request = {
220+
"messages": [
221+
{
222+
"role": "system",
223+
"content": "YOU MUST FOLLOW THESE INSTRUCTIONS CAREFULLY.\n<instructions>\n1. Use markdown to format your answers.\n</instructions>",
224+
},
225+
{
226+
"role": "user",
227+
"content": "Whats happening in France? Summarize top stories with sources, very short and concise. Also please search about the histoy of france as well.",
228+
},
229+
],
230+
"model": "mixtral-8x7b-32768",
231+
"tool_choice": "auto",
232+
"tools": [
233+
{
234+
"type": "function",
235+
"function": {
236+
"name": "duckduck.search",
237+
},
238+
},
239+
{
240+
"type": "function",
241+
"function": {
242+
"name": "duckduck.news",
243+
},
244+
},
245+
],
246+
}
247+
248+
response = requests.post(
249+
proxy_url,
250+
headers=header,
251+
json=request,
252+
)
253+
254+
if response.status_code == 200:
255+
res = response.json()
256+
print(res["choices"][0]["message"]["content"])
257+
else:
258+
print("Error:", response.status_code, response.text)
259+
260+
```
261+
262+
## Using with PhiData
263+
FindData is a favorite of mine for creating AI assistants, thanks to its beautifully simplified interface, unlike the complexity seen in the LangChain library and LlamaIndex. I use it for many projects and want to give kudos to their team. It's open source, and I recommend everyone check it out. You can explore more from this link https://github.com/phidatahq/phidata.
264+
265+
266+
```python
267+
from google.README import userdata
268+
from phi.llm.openai.like import OpenAILike
269+
from phi.assistant import Assistant
270+
from phi.tools.duckduckgo import DuckDuckGo
271+
import os, json
272+
273+
274+
my_groq = OpenAILike(
275+
model="mixtral-8x7b-32768",
276+
api_key=userdata.get("GROQ_API_KEY"),
277+
base_url="https://groqcall.ai/proxy/groq/v1"
278+
)
279+
assistant = Assistant(
280+
llm=my_groq,
281+
tools=[DuckDuckGo()], show_tool_calls=True, markdown=True
282+
)
283+
assistant.print_response("Whats happening in France? Summarize top stories with sources, very short and concise.", stream=False)
284+
285+
286+
```
287+
288+
## Contributions Welcome! 🙌
289+
I am excited to extend and grow this repository by adding more built-in functions and integrating additional services. If you are interested in contributing to this project and being a part of its development, I would love to collaborate with you! I plan to create a discord channel for this project, where we can discuss ideas, share knowledge, and work together to enhance the repository.
290+
291+
Here's how you can get involved:
292+
293+
1. Fork the repository and create your own branch.
294+
2. Implement new functions, integrate additional services, or make improvements to the existing codebase.
295+
3. Test your changes to ensure they work as expected.
296+
4. Submit a pull request describing the changes you have made and why they are valuable.
297+
298+
If you have any ideas, suggestions, or would like to discuss potential contributions, feel free to reach out to me. You can contact me through the following channels:
299+
300+
- Twitter (X): @unclecode
301+
- Email: [email protected]
302+
303+
### Copyright 2024 Unclecode (Hossein Tohidi)
304+
305+
Licensed under the Apache License, Version 2.0 (the "License");
306+
you may not use this file except in compliance with the License.
307+
You may obtain a copy of the License at
308+
309+
http://www.apache.org/licenses/LICENSE-2.0
310+
311+
Unless required by applicable law or agreed to in writing, software
312+
distributed under the License is distributed on an "AS IS" BASIS,
313+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
314+
See the License for the specific language governing permissions and
315+
limitations under the License.
316+
317+
I'm open to collaboration and excited to see how we can work together to enhance this project and provide value to the community. Let's connect and explore how we can help each other!
318+
319+
Together, let's make this repository even more awesome! 🚀

‎app/main.py

+27-3
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,14 @@
22
from fastapi.responses import HTMLResponse
33
from fastapi.templating import Jinja2Templates
44
from fastapi.staticfiles import StaticFiles
5+
from starlette.middleware.cors import CORSMiddleware
56
from starlette.requests import Request
67
from routes import proxy
78
from routes import examples
89
from utils import create_logger
910
import os
1011
from dotenv import load_dotenv
12+
1113
load_dotenv()
1214

1315
app = FastAPI()
@@ -16,31 +18,53 @@
1618
app.mount("/static", StaticFiles(directory="frontend/assets"), name="static")
1719
templates = Jinja2Templates(directory="frontend/pages")
1820

21+
22+
origins = [
23+
"*",
24+
]
25+
26+
app.add_middleware(
27+
CORSMiddleware,
28+
allow_origins=origins,
29+
allow_credentials=True,
30+
allow_methods=["*"],
31+
allow_headers=["*"],
32+
)
33+
34+
1935
@app.middleware("http")
2036
async def log_requests(request: Request, call_next):
2137
if "/proxy" in request.url.path:
2238
client_ip = request.client.host
23-
logger.info(f"Incoming request from {client_ip}: {request.method} {request.url}")
39+
logger.info(
40+
f"Incoming request from {client_ip}: {request.method} {request.url}"
41+
)
2442
response = await call_next(request)
2543
# logger.info(f"Response status code: {response.status_code}")
2644
return response
2745
else:
2846
return await call_next(request)
2947

48+
3049
app.include_router(proxy.router, prefix="/proxy")
3150
app.include_router(examples.router, prefix="/examples")
3251

52+
3353
@app.get("/", response_class=HTMLResponse)
3454
async def index(request: Request):
3555
return templates.TemplateResponse("index.html", {"request": request})
3656

57+
3758
# Add an get endpoint simple return the evrsion of the app
3859
@app.get("/version")
3960
async def version():
4061
return {"version": "0.0.1"}
4162

63+
4264
if __name__ == "__main__":
4365
import uvicorn
44-
# uvicorn.run("main:app", host=os.getenv("HOST"), port=int(os.getenv('PORT')), workers=1, reload=True)
45-
uvicorn.run("main:app", host=os.getenv("HOST"), port=int(os.getenv('PORT')), workers=1)
4666

67+
# uvicorn.run("main:app", host=os.getenv("HOST"), port=int(os.getenv('PORT')), workers=1, reload=True)
68+
uvicorn.run(
69+
"main:app", host=os.getenv("HOST"), port=int(os.getenv("PORT")), workers=1
70+
)

0 commit comments

Comments
 (0)