Skip to content

Latest commit

 

History

History
116 lines (90 loc) · 3.35 KB

README.md

File metadata and controls

116 lines (90 loc) · 3.35 KB

Building and Running llama.cpp with RPC Support on macOS

Overview

Important

The RPC backend is currently in a proof-of-concept stage. It is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!

The RPC feature allows distributed LLM inference with llama.cpp by offloading computations to remote Mac Studio hosts.

Prerequisites

  • Git
  • CMake
  • Xcode Command Line Tools
  • Metal SDK (included with macOS)

Building RPC Server

  1. Clone the repository:

    git clone https://github.com/ggerganov/llama.cpp.git
    cd llama.cpp
  2. Build the RPC server with Metal backend:

    mkdir build-rpc-metal
    cd build-rpc-metal
    cmake .. -DGGML_METAL=ON -DGGML_RPC=ON
    cmake --build . --config Release
  3. Start the RPC server on each Mac:

    ./bin/rpc-server -p 50052

    If doing multi-node compute, each node must be told its ip at start

    ./bin/rpc-server -H 10.0.0.1 -p 50052

Building llama.cpp with RPC Support

  1. Build llama.cpp with RPC:

    mkdir build-rpc
    cd build-rpc
    cmake .. -DGGML_RPC=ON
    cmake --build . --config Release
  2. Run llama.cpp with RPC (replace IP addresses with your Mac IPs):

    ./bin/llama-cli -m ../models/tinyllama-1b/ggml-model-f16.gguf -p "Hello, my name is" --repeat-penalty 1.0 -n 64 --rpc 192.168.1.10:50052,192.168.1.11:50052,192.168.1.12:50052 -ngl 99

    Or using descriptive tags:

    ./bin/llama-cli -m {path_to_model} -p {prompt} --repeat-penalty {penalty_value} -n {num_tokens} --rpc {ip1}:{port1},{ip2}:{port2},{ip3}:{port3} -ngl {num_gpu_layers}

Best Practices

  • Keep the repository updated:
    git pull origin master
  • Rebuild after updates:
    cd build-rpc
    cmake --build . --config Release

Troubleshooting

  1. RPC connection issues: Ensure firewalls allow traffic on the specified ports.
  2. Metal compilation errors: Update to the latest macOS and Xcode versions.
  3. Out of memory errors: Adjust model size or batch size based on available GPU memory.

Performance Optimization

  • Use -ngl to balance GPU layers across machines for optimal performance.
  • Experiment with different batch sizes (--batch-size)
  • Quantize (-q) for larger models to reduce memory usage.
  • Currently working on performance tuning for llama-405, please contact or reach out if you are doing the same with M2 mac studios

Useful Info

reach out on twitter if you have questions

Converting HuggingFace Models to GGUF Format

  1. Ensure you have the required dependencies:

    pip install -r requirements.txt
  2. Navigate to the llama.cpp directory:

    cd path/to/llama.cpp
  3. Run the conversion script:

    python3 convert_hf_to_gguf.py /path/to/input/model/ --outfile /path/to/output/model.gguf

    Example:

    python3 convert_hf_to_gguf.py /Volumes/PRO-G40/weights/llama-3.1/Meta-Llama-3.1-8B-Instruct/ --outfile /Volumes/PRO-G40/weights/llama-3.1/Meta-Llama-3.1-8B-Instruct/llama-3.1-model.gguf
  4. Use the converted model with llama.cpp:

Note: Ensure you have sufficient disk space for the conversion process.