ollama (self-hosted) support for AI Flow

[thefoxman88]

Can I request a self-hosted option for AI requests?
Hopefully others would like to avoid requesting book info to gemini/openAi about a questionable spicey books?

I have a simple setup with Ollama and Open-Webui and would love to share that resource with library-manager. Lets just hope the models required can be used with less then 12GB (RTX 3060)

[deucebucket]

Hey @thefoxman88! This is definitely on the roadmap.

Current state:

• We already support OpenRouter which gives you access to various models
• The AI is only used as a fallback when our 50M book database does not find a match
• Most lookups never hit AI at all (BookDB handles ~90% of queries)

Ollama support plan:
Adding Ollama would be straightforward since we already have the provider abstraction:

• New ai_provider: "ollama" option
• Configure ollama_url (default: http://localhost:11434)
• Model selection (llama3.2, mistral, etc.)

For a 12GB RTX 3060, you could run:

• llama3.2:3b (~2GB) - fast, good enough for metadata parsing
• mistral:7b (~4GB) - better accuracy
• llama3.1:8b (~5GB) - best balance

The task is pretty simple (parsing author/title from folder names), so even smaller models work well.

I will add this in an upcoming release. Would you prefer:

1. Just Ollama support, or
2. Generic "OpenAI-compatible API" support (works with Ollama, LM Studio, text-generation-webui, etc.)?

[deucebucket]

This has been implemented! 🎉

Ollama support is now available on the develop branch (PR #25 merged). Here's what's included:

Features:

• Ollama added as a third AI provider option in Settings → AI Setup
• Server URL configuration (default: http://localhost:11434)
• Model selector that fetches available models directly from your Ollama server
• Test Connection button to verify connectivity
• Works completely offline - no API keys or external calls needed

Recommended models for your RTX 3060 (12GB):

• llama3.2:3b - Fast, works great with 8GB+ VRAM
• llama3.2:1b - Even faster, works on 4GB+ VRAM
• mistral:7b - More capable, fits comfortably in 12GB
• phi3:mini - Microsoft's efficient small model

To try it now:
Pull the develop branch or use the Docker image tagged develop:

docker pull ghcr.io/deucebucket/library-manager:develop

Then in Settings → AI Setup, select "Ollama (Self-hosted - No API costs)" and configure your server URL.

Your spicy books stay private! 🌶️

This will be in the next stable release after some testing on develop.

[Jdiesel87]

All ollama models are showing up as undefined in library-manager. Tested with mistral:7b

[Jdiesel87]

Seeing this error in my logs

ERROR - Ollama: Model 'undefined' not found. Run: ollama pull undefined

[deucebucket]

This report exposed two real limitations in the current implementation: Ollama model handling still assumed specific fallback models/one response shape, and Library Manager had no generic llama.cpp/OpenAI-compatible provider.

A fix is now in draft PR #270: #270

It adds:

• llama.cpp and generic OpenAI-compatible API support (/v1/models and /v1/chat/completions)
• live model discovery from whichever AI provider is selected
• automatic selection only when the server exposes exactly one model
• editable model IDs when discovery is unavailable
• no hardcoded Ollama model fallback
• handling for Ollama name, model, and string model-list shapes (fixes undefined)
• reuse of transcript-only Skaldleita results with the selected AI fallback

Skaldleita remains the hosted metadata/audio-identification service. Users point only the new AI provider at their own local inference server. The PR is not merged or released yet; its checks are running.

[deucebucket]

Live validation is complete on PR #270, and all four GitHub checks now pass.

Verified against the actual local Bonsai deployment:

• native Ollama model discovery and inference
• raw llama.cpp /v1/models and /v1/chat/completions
• OpenAI-compatible inference using bonsai-clean:latest
• two-book batch output returning both items, not only the first
• Skaldleita transcript handoff to the selected local model with no second transcription
• desktop/mobile Settings and fresh-install setup wizard save/reload flows
• embedding-only Ollama models excluded from the selectable completion-model list

The stable host configuration used for testing was http://127.0.0.1:11500/v1 with bonsai-clean:latest. Do not configure Library Manager with the internal raw llama-server port because Ollama allocates that port dynamically. In Docker, 127.0.0.1 points at the Library Manager container, so use a host-reachable address such as host.docker.internal or the host LAN address instead.

The validation also found and fixed unbounded native generation, truncated multi-book JSON, permissive secrets.json permissions, and raw saved tokens being rendered into Settings HTML.