Connection problems with ollama

[Sennar4]

Hello,
I have connection problems with my local LLMs on the GPU cluster.
My LLM specification looks like this:

api_key: ollama
base_url: http://172.17.0.1:12345/v1/
model: llama3.1:8b

The base_url should be correct because I had to choose a different port for Ollama. But I always get the error:

• Testing model: ollama
• Sending a message Hi
• Connection failed. Got error: Error code: 404 - {'error': {'message': 'model “llama3.1:8b” not found, try pulling it first', 'type': 'api_error', 'param': None, 'code': None}}

But when I look on the server it finds the models:

curl -X GET http://localhost:11434/v1/models
{“object”:“list”,“data”:[{“id”:“llama3.1:8b”,“object”:“model”,“created”:1736731098,“owned_by”:“library”},{“id”:"llama3. 2:1b“,”object“:”model“,”created“:1736730830,”owned_by“:”library“},{”id“:”nomic-embed-text:latest“,”object“:”model“,”created“:1736730124,”owned_by“:”library“},{”id“:”llama3. 2:latest“,”object“:”model“,”created“:1730220016,”owned_by“:”library“},{”id“:”sroecker/sauerkrautlm-7b-hero:latest“,”object“:”model“,”created“:1726276892,”owned_by“:”sroecker“},{”id“:”mistral:latest“,”object“:”model“,”created“:1725557151,”owned_by“:”library"}]}

Does anyone have any idea what the problem could be?

[sthpratik]

I am not clear on port you are running curl command on one port and connecting on different. Which app is on container or both app is in local

[ahadrez]

@Sennar4 if you're running the kotaemon RAG locally using Docker , you probably need replace http://localhost with http://host.docker.internal to correctly communicate with service on the host machine.
here's my example which worked fine for me.
api_key: ollama base_url: http://host.docker.internal:11434/v1/ model: deepseek-r1:7b