File tree 1 file changed +50
-1
lines changed
1 file changed +50
-1
lines changed Original file line number Diff line number Diff line change 2
2
3
3
## Ollama
4
4
5
- For installation see: https://ollama.com/
5
+ For installation see: https://ollama.com/ . Then start
6
+ [ ollama server] ( https://github.com/ollama/ollama?tab=readme-ov-file#start-ollama ) with
7
+ ` ollama serve ` command.
6
8
7
9
``` python
8
10
from langchain_community.chat_models import ChatOllama
9
11
10
12
llm = ChatOllama(model = ' llava' )
11
13
```
12
14
15
+ ### Configure ollama wth OpenAI compatible API
16
+
17
+ Ollama supports OpenAI compatible APIs (see [ details] ( https://ollama.com/blog/openai-compatibility ) ).
18
+
19
+ > [ !TIP]
20
+ > Such setup might be more convenient if you frequently switch between OpenAI API and
21
+ > local models.
22
+
23
+ To configure ollama through OpenAI API in ` rai ` :
24
+
25
+ 1 . Add ` base_url ` to [ config.toml] ( ../config.toml )
26
+
27
+ ``` toml
28
+ [openai ]
29
+ simple_model = " llama3.2"
30
+ complex_model = " llama3.2"
31
+ ...
32
+ base_url = " http://localhost:11434/v1"
33
+ ```
34
+
35
+ ### Example of setting up vision models with tool calling
36
+
37
+ In this example ` llama3.2-vision ` will be used.
38
+
39
+ 1 . Create a custom ollama ` Modelfile ` and load the model
40
+
41
+ > [ !NOTE]
42
+ > Such setup is not officially supported by Ollama and it's not guaranteed to be
43
+ > working in all cases.
44
+
45
+ ``` shell
46
+ ollama pull llama3.2
47
+ echo FROM llama3.2-vision > Modelfile
48
+ echo ' TEMPLATE """' " $( ollama show --template llama3.2) " ' """' >> Modelfile
49
+ ollama create llama3.2-vision-tools
50
+ ```
51
+
52
+ 3 . Configure the model through OpenAI compatible API in [ config.toml] ( ../config.toml )
53
+
54
+ ``` toml
55
+ [openai ]
56
+ simple_model = " llama3.2-vision-tools"
57
+ complex_model = " llama3.2-vision-tools"
58
+ ...
59
+ base_url = " http://localhost:11434/v1"
60
+ ```
61
+
13
62
## OpenAI
14
63
15
64
``` bash
You can’t perform that action at this time.
0 commit comments