RFC: Inference

[feynon]

[PrimalPimmy]

[RFC] Linux Support via llama-cpp-python

1. Summary

This RFC proposes adding official Linux support to Tiles by implementing a Linux-specific Python backend using llama-cpp-python. It also proposes refactoring the Rust CLI runtime into a generic HTTP client that works transparently across both macOS and Linux, eliminating the need for duplicated OS-specific Rust code.

2. Motivation

Currently, the inference logic is heavily tied to Apple Silicon via the MLX framework. While the Python server has a routing mechanism to detect Linux and load a Linux backend, that backend is currently an empty stub.

To make the project truly cross-platform, we need a viable inference engine for Linux (and potentially Windows) that supports CPU, CUDA (and maybe Vulkan?) executions. llama.cpp (via its Python bindings) is the industry standard for this and would work perfectly for our local first approach.

3. Architecture & Proposed Changes

The implementation requires changes in two places: the Rust CLI and the Python Server.

A. Rust CLI Refactor (Platform Agnostic)

Currently, the codebase uses OS-specific flags to route macOS to its specific runtime.

• Proposal: Remove the OS-specific compile flags. Rename the macOS runtime module to something generic like "Server Runtime" or "HTTP Runtime". Both macOS and Linux will now use this exact same Rust code to spawn the Python server and connect to it locally.

B. Python Backend Implementation (llama-cpp-python)

• Dependency: Add llama-cpp-python as a platform-specific dependency for Linux in the Python package configuration.
• Linux Backend Routing: Implement the required API signatures (model loading, generating chat streams, OpenResponses API), maintaining the exact structural logic found in the macOS backend (e.g., handling SSE streams, computing metrics, and calling the reasoning extractor).
• Inference Runner: Create a llama-cpp runner class as a drop-in replacement for the MLX runner. It will wrap the llama.cpp library and implement token calculation methods, standard streaming generators, and harmony-specific streaming logic for reasoning models.

4. Alternate Solutions Considered

1. Pure Rust via llama-cpp-rs: Bypassing the Python server entirely on Linux.
   • Cons because it would require re implementing complex server-side logic like the Memory Agent, OpenResponses API, and harmony reasoning extraction natively in Rust, which is going to be hard to maintain.

5. Maintainer Feedback Requested

1. Model Format: llama.cpp requires .gguf files, while MLX uses .safetensors. We will need to adjust the model downloader or the model specs to ensure the correct format is pulled based on the target OS. What is the preferred UX for this?
2. Dependency Management: llama-cpp-python requires a C++ compiler to build from source if pre-built wheels aren't available for the user's specific CUDA/Vulkan setup. Should this be handled entirely by documentation, or should we attempt to provide pre-compiled environments? (Although this won't matter much for a generic linux user, I think. Need to check this)

References

https://github.com/ggml-org/ggml
https://github.com/abetlen/llama-cpp-python
https://github.com/ggml-org/llama.cpp

[madclaws]

@PrimalPimmy Overall Plan sounds good. Some things to note are,

• Currently the responses api are implemented in py server, but our plan is to implement the apis in rust itself (as from 0.4.4, we have a rust daemon too). And only do the actual inference from python. This is a future plan and you can carry on with your current plan as you won't be rewriting apis again for linux, just stating.

• Regarding the model format part, currently models are downloaded from the modelfile instruction. So either for default models we could keep multiple modelfiles for macos and others and depending on OS we just use it dynamically. For non default models, its always what users provide as CLI args either a modelfile or HF url (HF url is not supported yet). There was some discussion on embedding platform details in modelfile @feynon , could we use smthing like that?

• pre-compiled would be suitable imo considering Tiles dont want users to do the grunt work, but @feynon wdyt?

[feynon]

2. Yeah, but if you look at Ollama the Modelfile is platform agnostic.

"Ollama runs a Modelfile as a build specification, similar to a Dockerfile. The Modelfile itself is platform-agnostic. Platform differences are handled by the Ollama runtime and the model packaging system rather than the Modelfile."

3. Yeah pre-compiled is the way to go.

[madclaws]

Yeah, but if you look at Ollama the Modelfile is platform agnostic.

Does that mean if i do FROM gpt-oss-20b in ollama modelfile, it downloads and runs mlx one in mac and gguf in linux auto?
@feynon

[feynon]

Yeah, but if you look at Ollama the Modelfile is platform agnostic.

Does that mean if i do FROM gpt-oss-20b in ollama modelfile, it downloads and runs mlx one in mac and gguf in linux auto?

@feynon

I don’t think that’s how it works in Ollama. The FROM instruction just points to the model file location, which differs by platform. On macOS it would point to the safetensors weights, while on Linux it would point to a GGUF file. The CLI should recognize that both refer to the same model. It can determine this through the MIR field in the Modelfile spec that we plan to add.