Support for distributed-llama?

[jkeegan]

I know the "o" in oterm stands for ollama, but I'm wondering if you'd consider making your tool compatible with distributed-llama?
( https://github.com/b4rtaz/distributed-llama )

The more systems it works with, the more popular it will be, and the more attention it can get from fans and developers.

Just a thought,
..Jeff

[ggozad]

Hey Jeff!
I have been wanting to do this, not just for distributed llama but others as well, but I wouldn't bet on it happening any time soon.
It takes considerable time and effort to maintain compatibility with different frameworks. For instance Ollama updated the python client recently and I haven't managed to put the time I need to update.
It's not just the chat API, it's the whole experience of editing parameters, pulling models and what not that is different from API to API. This is a solo pet project more or less, and I have to feed myself in the meantime :)

[jkeegan]

Understood, just wanted to add my voice to the chorus and give you feedback. What might make it easier than your Ollama python collision example is that the distributed-llama API is web-based - it's a subset (only the /v1/chat/completions endpoint) of the OpenAI API. So that might be a nice protocol division to keep things safe and isolated. I'm not that familiar with it yet TBH - just getting up-to-speed. I built my 4x Raspberry Pi 5 (8GB each, so 32GB total) cluster a month or so ago so I could play with models. (As such, since I'm ssh'ing into one of the boards to run commands like the api server etc, it'd be awesome to have a great curses interface as rich as yours). Although distributed-llama requires the number of nodes to be a power of 2, I'm not rulling out doubling the size of my cluster with another 4 boards, doubling the available memory and increasing the kinds of models I can run locally.
Thanks for replying (just saw it now) - have a great day!