Difficulties when running local LLM backends

[av]

Tried raglite chainlit with a few models via Ollama:

OLLAMA_API_BASE=http://ollama:11434
CHAINLIT_HOST=0.0.0.0

raglite
      --llm ollama/llama3.1:8b
      --embedder ollama/mxbai-embed-large:latest
      chainlit

I tried following models:

• ollama/llama3.1:8b
• ollama/llama3.2:3b-instruct-q8_0
• ollama/mistral-small

Ollama context was set to 8k in all instances. All replies in chainlit UI were formatted like this:

[av]

Debugging further:

• Installed version of litellm is 1.56 despite 1.60 is set in the pyproject
  • Raglite installed with raglite[chainlit,pandoc,ragas]
• It appears that litellm has issues with tool calls when using ollama and ollama_chat provider with streaming enabled
• When trying to use Ollama via OpenAI-compatible adapter in litellm - the latter can't serve get_emebedding_dim and get_context_size from _litellm
• Using openrouter/openai/gpt-4o calls tools as intended

So, it doesn't appear there's a way to use RAGLite with a local Ollama for now

[lsorber]

Thanks for reporting this @av! We'll investigate and see what the best way to resolve this is.