connecting to hosted llm (NOT LOCAL & NOT OPENAI)

[squatchydev9000]

Anyone been able to connect to a remote server hosting an LLM that isn't openai and isn't local (ollama/litellm/openllm/etc)?

***without wrappers and workarounds? ***without using the openai api system?

[ekzhu]

LiteLLM supports OpenAI compatible API?

[squatchydev9000]

I cannot figure out how to use with a hosted open-sourced llm... do you know how?

And also, whats the point if it funnels through the openai api system? Doesn't that defeat the purpose of using open-sourced for anything other than saving compute costs?

This only gives openai and microsoft power over our production apps... they could simply alter the api to slow our stuff down, or absorb the data passing through it correct?

[ekzhu]

Many tools like vllm and litellm supports OpenAI compatible API so the API calls from your apps won't go to OpenAI or Microsoft. Regarding the API specification itself, I think it is in the best interest of the maintainer of the API specs (OpenAI in this case) to keep it stable for a healthy ecosystem. That said, we are also working towards a custom client layer which allows us to use local models directly without following the OpenAI's API specs (#831).

[squatchydev9000]

for a week I thought I had it talking to my remote server... and it was actually using chat-gpt3.5... lol

Looking forward to the custom client layer... but I am very apprehensive to trust microsoft and openai... (we all should be)

You guys (openai and MS) are pushing humanity over the cliff and into the abyss... rather than towards our potential... its a race to the bottom and I for one am terrified that if AI isn't available for everyone, everything will go to hell.