Distributed Self-Hosted LLM Model

[CoolJohn-lab]

After some time we would have maybe 2000-3000 users with dedicated servers, all around the world, all running Odysseus. I will be one of those users. But unless I am actively prompting it or it is running a background task, my server will sit mostly idle.

It might be nice if all Odysseus users could join a shared LLM pool, where complex tasks are farmed out to a very large model, which is sharded over the internet to multiple Odysseus instances, who all work on it together. This facility would be "opt in", and dynamically use my spare GPU and VRAM resources between a certain time period, so overnight basically.

I could set my "sleeping hours" in Odysseus to between say 23:30 and 07:00, which is when my electricity cost drops to off-peak and I don't mind my GPUs whirring away through the night, resolving someone else's prompts.

My continued participation is assured, since every token I process overnight gains me a token "credit", which I can then use to query this distributed mega-model myself.

This way, we could actually build out something to rival the frontier models with their dedicated data centers; a multi-trillion parameter model, sharded across hundreds or even thousands of distributed self-hosted server nodes. Similar to something like BitTorrent or TOR. It would probably respond quite slowly, but it could be really, really smart.

to distribute prompts we would need an orchestrator node capable of farming out prompts to distributed inferencing nodes, with each node running a shard or multiple shards of a very large model. The Orchestrator node would need to understand the full context of the user's prompt, in which case it would need to be run from the user's local Odysseus instance. It would send sharded inference reads and context KVs out to the distributed inferencing nodes it was communicating with.

We would trade a small amount of privacy in our prompts for this power. A user running a distributed inferencing shard would be able to peek into the memory of their GPUs to see what instructions the orchestrator node had been sending it, allowing someone to at least partially reconstruct the prompt, depending on what part of the distributed model they were hosting. So inference node operators would only see one small piece of a distributed prompt's context, not all of it. To mitigate this the inference nodes should be randomised and periodically switched between different distributed inferencing clusters, to clear local VRAM and make it harder to peek into other users' distributed prompt chains.