Skip to content

About

No description, website, or topics provided.

Resources

Stars

4 stars

Watchers

2 watching

Forks

Latest commit

 

History

6 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 

Repository files navigation

llama.cpp RPC Setup Guide

This repository contains setup instructions for running llama.cpp with RPC support on different platforms:

Each README provides detailed instructions for:

  • Prerequisites
  • Building the RPC server
  • Building llama.cpp with RPC support
  • Running distributed inference
  • Best practices
  • Troubleshooting
  • Performance optimization
  • Converting HuggingFace models to GGUF format

Refer to the specific README for your platform to get started with distributed LLM inference using llama.cpp and RPC.

Important: The RPC backend is currently in a proof-of-concept stage. Use caution when deploying in production environments.

About

No description, website, or topics provided.

Resources

Stars

4 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors