Skip to content

Commit 9224fa5

Browse files
committed
docs(ollama): more info and vision models with tool calling
1 parent 806dcc0 commit 9224fa5

File tree

1 file changed

+50
-1
lines changed

1 file changed

+50
-1
lines changed

docs/vendors.md

+50-1
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,63 @@
22

33
## Ollama
44

5-
For installation see: https://ollama.com/
5+
For installation see: https://ollama.com/. Then start
6+
[ollama server](https://github.com/ollama/ollama?tab=readme-ov-file#start-ollama) with
7+
`ollama serve` command.
68

79
```python
810
from langchain_community.chat_models import ChatOllama
911

1012
llm = ChatOllama(model='llava')
1113
```
1214

15+
### Configure ollama wth OpenAI compatible API
16+
17+
Ollama supports OpenAI compatible APIs (see [details](https://ollama.com/blog/openai-compatibility)).
18+
19+
> [!TIP]
20+
> Such setup might be more convenient if you frequently switch between OpenAI API and
21+
> local models.
22+
23+
To configure ollama through OpenAI API in `rai`:
24+
25+
1. Add `base_url` to [config.toml](../config.toml)
26+
27+
```toml
28+
[openai]
29+
simple_model = "llama3.2"
30+
complex_model = "llama3.2"
31+
...
32+
base_url = "http://localhost:11434/v1"
33+
```
34+
35+
### Example of setting up vision models with tool calling
36+
37+
In this example `llama3.2-vision` will be used.
38+
39+
1. Create a custom ollama `Modelfile` and load the model
40+
41+
> [!NOTE]
42+
> Such setup is not officially supported by Ollama and it's not guaranteed to be
43+
> working in all cases.
44+
45+
```shell
46+
ollama pull llama3.2
47+
echo FROM llama3.2-vision > Modelfile
48+
echo 'TEMPLATE """'"$(ollama show --template llama3.2)"'"""' >> Modelfile
49+
ollama create llama3.2-vision-tools
50+
```
51+
52+
3. Configure the model through OpenAI compatible API in [config.toml](../config.toml)
53+
54+
```toml
55+
[openai]
56+
simple_model = "llama3.2-vision-tools"
57+
complex_model = "llama3.2-vision-tools"
58+
...
59+
base_url = "http://localhost:11434/v1"
60+
```
61+
1362
## OpenAI
1463

1564
```bash

0 commit comments

Comments
 (0)