Choosing best model that is free

[teberzin]

I was using chagpt and gemini subscriptions so far and I want to switch to local AI and obviously I want to try odysseus right now. But before I do I have some questions so here is quick question correct me if I'm mistaken.

Right now I either have chance to use local LLM or API to use. And API's a credit based and I need to pay some amount of money to get credit and I want to avoid it since I don't want to lost in credit calculation during usage of the model.

https://openrouter.ai has free tier 50 requests/day which is a thing but probably not enough so I will skip it as well and I need to use local llm to use on my computer. So while choosing the model I will use what should I consider?

[CoolJohn-lab]

The point of Odysseus is not to be a connector into an API model, yes there are some that you can use for free but the sort of data you will be exposing those free APIs to is deeply personal and likely to get you hacked or worse if you don't choose your API provider carefully.

Imagine some hacker was running a "Free" API endpoint that you could sign up for and use for nothing, why would they do that? To collect your data of course. So you start having it read all your emails and all of a sudden you are passing the details of every subscription, bank account, payment card that you have into this API. Of course they say your data is private and they will never look at it... sure ok.

Now imagine that the people that run OpenAI, Claude, whatever, imagine they are no better than some lowlife hacker running his little scam honeypot. The only difference between them and the hacker is the amount of money they have and how famous their model is... Now you see why all these API models have a generous "free" tier...

The correct way to run a free model is to run it on your own hardware. That way your personal data never leaves your own machine. This is what Odysseus is built to do, and it makes it super easy for you. Learn the cookbook, learn what capability your machine has.

If your machine isn't powerful enough to run anything, build a server. This thing is designed and intended to run on its own, standalone dedicated server and you connect into it via a web browser over your local network / wifi, or even from your phone over the internet. It's a lot of money up front to buy one, and it costs some money in electricity to run it, and it is a lot of work to build and configure it, but if you are serious about privacy and want to use the killer email features in this app, it is the only safe way to use it IMO.

I am actually surprised it lets you use API LLMs at all for some of the stuff it does... But I guess if it didn't, someone would hack it in anyway.

[razonjasky]

is local llm free or does it need credits as well?

[CoolJohn-lab]

Local LLM is free to use, it runs on your own computer using your own hardware and completely free software.

You will need a very large computer with a lot of memory and VRAM to make good use of it however. Don't expect to be able to run it on a laptop unless it's a crazy good MacBook pro with 32GB+ of RAM.

It is not "free" in the respect that it will work well on the computer you already have, it is "free" once you upgrade your computer to a sufficient spec to run it. You can create a dedicated server which any laptop can then connect into, or even your phone, and use as if it was Claude. That server will be free for you to use, but won't be cheap to build.

If you don't already have a gaming PC lying around, or better yet a Bitcoin mining rig, then you could buy something like a Mac Mini with 32GB unified memory and run it on that, or if you want to go hardcore you could buy a 128GB Nvidia DGX Spark and install it on that!!! The world is quite literally yours.

[razonjasky]

Hi, thanks for responding. Would it run on my MacBook air m5? It only has 16gb of ram, by would it run I mean just in a minimum way. Thanks