[feature] LocalAI endpoint

[nalbion]

I don't have any experience with it, but LocalAI might be more attractive for people working in environments where sending source code out to the interwebs is frowned upon.

LocalAI is a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families that are compatible with the ggml format. Does not require GPU.

• Text generation with llama.cpp, gpt4all.cpp and more
• OpenAI functions
• Embeddings generation for vector databases
• Download Models drectly from Huggingface

See also #69 by @mrgoonie

How do I utilize local.ai?

[jmikedupont2]

https://twitter.com/introsp3ctor/status/1708301615143256165?t=fuwFrgIAd7uZixTVZKfK6A&s=19 I have a fake openai running and some basic lollms running

[nalbion]

This may already be supported. Apparently you just need to set OPENAI_API_BASE=http://localhost:8080 and run LocalAI.

Oh that’s pretty sick, I better do more reading up on localai I hope it’s not hard to apply or is too expensive.. I’m trying to apply to Claude but I’m still such a beginner learner, I have an idea for a new type of reinforcement and recall learning tool for LLMs and I could really use the 100,000 tokens to test it out.. but they seem so high teir like they want actual corporations or businesses to apply.. I mean I want to eventually register as an independant startup but I need to learn more first but I always reach my token limit too quick. I just hope they find me worthy of Séance-ing with our soon to be gods of our own creation.. I mean if they say quantum computing will break RSA encryption so quick, and it’s that sophisticated, why aren’t they feeding THOSE DWAVE type quantum all these huge datasets and training and fine tuning into a quantum LLM which would be very very near sentience, perhaps indistinguishable. (More before I get dragged over the hot coals, I’m not a quantum computer expert frankly I can’t wrap my head around what the f it is,no thanks to Jodie rose who says it works by summoning things similar to the lovecraftian great old ones .. Like.. THEN TURN IT OFF?!😗😲 Sent from [Proton Mail](https://proton.me/mail/home) for iOS

…

On Tue, Oct 10, 2023 at 7:16 PM, Nicholas Albion ***@***.***(mailto:On Tue, Oct 10, 2023 at 7:16 PM, Nicholas Albion <<a href=)> wrote: This may already be supported. Apparently you just need to set OPENAI_API_BASE=http://localhost:8080 and run LocalAI. [image](https://user-images.githubusercontent.com/1866161/274101840-c3c97e13-70a4-416f-99a5-597d476a2400.png) — Reply to this email directly, [view it on GitHub](#92 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/A7I7VYDL4N3ZE7KHBBDU6D3X6XQPDAVCNFSM6AAAAAA4WERP2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJWGQ4TGNBRG4). You are receiving this because you commented.Message ID: ***@***.***>

[TomLucidor]

@jmikedupont2 sorry just found you through search, what are the feature differences between PrivateGPT, LocalAI, and LOLLMS? Also are Oobabooga and Koboldcpp/KoboldAI/SillyTavern just the same plain LLM access point without much features outside?