Local LLM, best models?

[rubin110]

I'm still having trouble getting some useful answers back from Copilot with my local LLM setup. If others can post the models they're having the best success with, that would be jelly. Thanks!

Edit: I should mention this is more an issue when dealing with embedding/RAG than doing simple "fix grammar" of selected texts prompts.

[Reaper151]

Agreed.
The result from Local LLMs is quite heavily hallucinated even with the Gemma 3-4B-IT with LM Studio, and 100k context length.
I've tried with Gemma 3-12B but my Mac mini M4 with 24 GB unified memory can't keep up with the required context length for my vault

[rubin110]

I should have been a little more specific with my original post. My troubles stem from the LLM hitting RAG.

For just general "fix grammar" prompts without RAG... I don't know how much text you're trying to throw at it, but I've found tuning the context length as low as I can manage provides more rational results.

Here's an interested post that goes into some of the specifics.

[Mainframework]

Try HugstonOne, it may solve your issue.
100% free & privacy friendly
Runs 100% locally, nothing leaves your computer. No hidden services, no HTTP, no server.
GGUF compatible – create charts on the fly
You can have the app for free in hugston.com Windows installer, msi or exe, cpu and gpu.
Very simple to use, download the app>select llm folder>select llm>load>ready.

I would love to hear your feedback.