Important
The RPC backend is currently in a proof-of-concept stage. It is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!
The RPC feature allows distributed LLM inference with llama.cpp by offloading computations to remote Mac Studio hosts.
- Git
- CMake
- Xcode Command Line Tools
- Metal SDK (included with macOS)
-
Clone the repository:
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp -
Build the RPC server with Metal backend:
mkdir build-rpc-metal cd build-rpc-metal cmake .. -DGGML_METAL=ON -DGGML_RPC=ON cmake --build . --config Release
-
Start the RPC server on each Mac:
./bin/rpc-server -p 50052
If doing multi-node compute, each node must be told its ip at start
./bin/rpc-server -H 10.0.0.1 -p 50052
-
Build llama.cpp with RPC:
mkdir build-rpc cd build-rpc cmake .. -DGGML_RPC=ON cmake --build . --config Release
-
Run llama.cpp with RPC (replace IP addresses with your Mac IPs):
./bin/llama-cli -m ../models/tinyllama-1b/ggml-model-f16.gguf -p "Hello, my name is" --repeat-penalty 1.0 -n 64 --rpc 192.168.1.10:50052,192.168.1.11:50052,192.168.1.12:50052 -ngl 99Or using descriptive tags:
./bin/llama-cli -m {path_to_model} -p {prompt} --repeat-penalty {penalty_value} -n {num_tokens} --rpc {ip1}:{port1},{ip2}:{port2},{ip3}:{port3} -ngl {num_gpu_layers}
- Keep the repository updated:
git pull origin master
- Rebuild after updates:
cd build-rpc cmake --build . --config Release
- RPC connection issues: Ensure firewalls allow traffic on the specified ports.
- Metal compilation errors: Update to the latest macOS and Xcode versions.
- Out of memory errors: Adjust model size or batch size based on available GPU memory.
- Use
-nglto balance GPU layers across machines for optimal performance. - Experiment with different batch sizes (
--batch-size) - Quantize (
-q) for larger models to reduce memory usage. - Currently working on performance tuning for llama-405, please contact or reach out if you are doing the same with M2 mac studios
- https://github.com/ggerganov/llama.cpp
- https://github.com/ggerganov/llama.cpp/tree/master/examples/rpc
- https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md
reach out on twitter if you have questions
-
Ensure you have the required dependencies:
pip install -r requirements.txt
-
Navigate to the llama.cpp directory:
cd path/to/llama.cpp -
Run the conversion script:
python3 convert_hf_to_gguf.py /path/to/input/model/ --outfile /path/to/output/model.gguf
Example:
python3 convert_hf_to_gguf.py /Volumes/PRO-G40/weights/llama-3.1/Meta-Llama-3.1-8B-Instruct/ --outfile /Volumes/PRO-G40/weights/llama-3.1/Meta-Llama-3.1-8B-Instruct/llama-3.1-model.gguf
-
Use the converted model with llama.cpp:
Note: Ensure you have sufficient disk space for the conversion process.