OpenAi compatible APi endpoint

[ali0une]

Hi there.

Very nice project!

Would it be possible to use an OpenAi compatible APi endpoint with a local LLM through LM Studio or text-generation-webui or llama.cpp?

i see llama.cpp has API Endpoints and even OAI-like API but i've never used them.

But i use LM Studio in server mode and it can expose an Open Api compatible endpoint like http://127.0.0.1:5000/v1

i'm pretty sure it's the same for text-generation-webui.

[regiellis]

Hi there.

Very nice project!

Would it be possible to use an OpenAi compatible APi endpoint with a local LLM through LM Studio or text-generation-webui or llama.cpp?

i see llama.cpp has API Endpoints and even OAI-like API but i've never used them.

But i use LM Studio in server mode and it can expose an Open Api compatible endpoint like http://127.0.0.1:5000/v1

i'm pretty sure it's the same for text-generation-webui.

Hey, I looked into this...because I am using the official python lib for my local calls...this should be supported out of the box. Just update the .env file to point to the http://[IP]:[PORT] and update the model you want to use. Should all work; I will be doing a test later to close this out...but you should have no issue as long as it is openAI compatible

[ali0une]

i'll have a look and let you know.
Thanks!

[ali0une]

So here my .env :

#https://developer.civitai.com/docs/getting-started/setup-profile#create-an-api-key
CIVITAI_TOKEN=xxxx
#/location/of/your/models/
MODELS_DIR=/whatever/models/

# http://localhost:11434 or http://host.docker.internal:11434
OLLAMA_API_BASE=http://localhost:5000/v1/ # same with http://localhost:5000
# Tuned to tinydolphin..as best I could
OLLAMA_MODEL=Meta-Llama-3.1-8B-Instruct
HTML_OUT=False # Some models require HTML conversion, some don't
TEMP=0
TOP_P=0

#OPENAI_MODEL=gpt-4o-mini
#OPENAI_API_KEY= # sk-proj

#GROQ_MODEL=llama-3.1-70b-versatile
#GROQ_API_KEY=#//

At launch the LM Studio server print this :

[2024-08-11 14:03:50.857] [INFO] [LM STUDIO SERVER] Verbose server logs are ENABLED
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] Success! HTTP server listening on port 5000
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] Supported endpoints:
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] ->	GET  http://localhost:5000/v1/models
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] ->	POST http://localhost:5000/v1/chat/completions
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] ->	POST http://localhost:5000/v1/completions
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] ->	POST http://localhost:5000/v1/embeddings     <------------ NEW!
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] Model loaded: local/Meta-Llama-3-8B-Instruct/Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] Logs are saved into /tmp/lmstudio-server-log.txt

And this if i issue a request with civitai-cli-manager and the .env above :

[2024-08-11 14:04:23.335] [ERROR] Unexpected endpoint or method. (POST /api/chat). Returning 200 anyway

i think we would need to have a specific configuration for openAi compatible APi like textgeneration-webui or LM Studio

[regiellis]

@ali0une
Strange...it should work as they both follow the OpenAI Standard...I will need to install LLMStudio server on my server and tested against it.