please add local llama support

[soragui]

using openai api is a little expensive to generate ,can add support for local llama ....

[nalbion]

@soragui how does Local Llama compare to LocalAI? (see issue #92)

[deepweather]

Nice request, highly support it! Being able to run gpt pilot with local models (maybe in the future fine tuned experts for different parts in the chain) and therefore have control over their instruction tuning would be wonderful.

@nalbion

I would suggest to use LocalAI over direct Llama, because this is much easier to maintain as LocalAI implements the OpenAI API specs.

[nalbion]

Thanks @deepweather

[nalbion]

@soragui @deepweather I don't suppose you'd be able to test this for us?

This may already be supported. Apparently you just need to set OPENAI_API_BASE=http://localhost:8080 and run LocalAI.

[adsanz]

I've tested this with a basic LocalAI set-up. It may be nice to add some tips so people don't go crazy. For example, users should set a big enough context size (above 1024) to allow gpt-pilot to work properly. Also I tested this with one of the vicuna 13b models from the gallery, it bugged quite a lot and usually didn't generate useful answers. If anybody wants me to try things out with LocalAI let me know and I'll get back here with results

[LeonOstrez]

@adsanz thats amazing! Thanks so much for testing it.

It would be very helpful if you could add to readme what you had to do for setup so we make it easier for others who want to try it out 🙏

[adsanz]

I'll commit a PR but I'd like to carry out a bit more testing, at the moment I didn't get anything useful out of local LLMs, GPT4 just blows them away. Also I noticed that on local LLMs it just goes straight to the user stories, and it does not make any questions. I will mention my findings in the PR as well if you like. @LeonOstrez

[Deng-Xian-Sheng]

Why is the local area not as cloud like, but openai is really too expensive

[Deng-Xian-Sheng]

I have a good idea that Intel BigDL may have an acceleration effect on local models, as Intel is promoting the local operation of large models. For example, the large model of 13B, optimized by BigDL for the 12th generation Intel, has 20 tokens per second

[Deng-Xian-Sheng]

How to start local now, is there README?Thanks