llama.cpp RPC Multiple Machines Setup: The Complete Technical Guide

By TradeMAV LLC • Tested & Verified: October 2026 • 8 min read

Direct Answer

To run llama.cpp RPC across multiple machines, compile llama.cpp on all nodes with `-DGGML_RPC=ON`. Start the RPC listener daemon on each worker PC using `./rpc-server -H 0.0.0.0 -p 50052`. Finally, launch `llama-server` on your coordinator machine, passing `–rpc :50052,:50052` alongside your model flag `-m ` to shard layers across your LAN.

The open-source llama.cpp project includes a built-in Remote Procedure Call (RPC) backend that lets a primary machine offload transformer layers to auxiliary worker machines on your local network. This guide covers the exact commands, flags, and real-world gotchas required to establish a stable multi-machine cluster.

Step 1: Compile llama.cpp with the RPC Backend Enabled

Compilation Requirement: RPC support is disabled by default in standard llama.cpp release builds. You must pass `-DGGML_RPC=ON` to CMake during configuration on every single node (coordinator and all workers).

# 1. Clone repository
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# 2. Configure CMake with RPC (add -DGGML_CUDA=ON for Nvidia or -DGGML_METAL=ON for Mac)
cmake -B build -DGGML_RPC=ON

# 3. Build the binaries
cmake --build build --config Release -j

Step 2: Start the RPC Server Daemon on Worker Nodes

Worker Launch: Launch `rpc-server` on each computer contributing RAM or VRAM. By default, it listens on port 50052. On Apple Silicon Macs, always pass `-d MTL0` to restrict evaluation to the Metal device and prevent fatal BLAS graph compute aborts.

# On Linux or Windows worker (e.g., 192.168.1.101):
./build/bin/rpc-server -H 0.0.0.0 -p 50052

# On Apple Silicon Mac worker (e.g., 192.168.1.102):
./build/bin/rpc-server -H 0.0.0.0 -p 50052 -d MTL0

Step 3: Launch llama-server on the Coordinator Node

Coordinator Command: Start `llama-server` on your primary computer and supply the comma-separated IP and port list to the `–rpc` flag.

./build/bin/llama-server   -m models/qwen2.5-32b-instruct-q4_k_m.gguf   --rpc 192.168.1.101:50052,192.168.1.102:50052   --host 0.0.0.0 --port 8080 -c 4096

The 3 Major Gotchas in Raw llama.cpp RPC

1. Zombie RPC Servers Holding Port 50052

If an engine process crashes or stops unexpectedly, orphaned rpc-server processes frequently stay running in the background holding TCP port 50052. Subsequent connections fail silently. Always run taskkill /F /IM rpc-server.exe /T (Windows) or killall -9 rpc-server (Mac/Linux) before restarting.

2. Apple Silicon Memory Clamping

macOS limits Metal applications to roughly 70% of total unified memory. If your script attempts to allocate full RAM capacity over RPC, Metal panics with GGML_ASSERT([rsets->data count] == 0). You must clamp worker capacities to 70% of physical RAM.

3. Naive Proportional Splitting

If you connect a fast GPU alongside a slow CPU with 32 GB RAM, llama.cpp‘s default proportional split algorithm offloads a massive percentage of layers onto the slow CPU, crushing generation speed from 15 tok/s to under 2 tok/s. Sizing requires dynamic waterfilling.

Pillar Guide Reference

For the complete conceptual breakdown of distributed tensor sharding, read our core pillar article: How to Pool RAM from Old PCs: The Complete Guide. For security implications, see Is llama.cpp RPC Secure?.

Frequently Asked Questions

What port does llama.cpp RPC use?

By default, `rpc-server` listens on TCP port 50052, but you can override this with the `-p ` flag.

Can you use both GPU and CPU across RPC nodes?

Yes. If an RPC node has an Nvidia GPU, build with `-DGGML_CUDA=ON` to offload layers to its VRAM; if it is CPU-only, layers evaluate in system RAM.

Why is my RPC connection failing with connection refused?

Ensure the worker’s firewall allows inbound traffic on TCP port 50052, verify the worker bound to `0.0.0.0` or its local IP rather than `127.0.0.1`, and confirm no zombie rpc-server is holding the socket.

Written by TradeMAV LLC

TradeMAV LLC is an engineering and product team with a track record of funding, manufacturing, and shipping production hardware and software (including OptionMAV and TradeMAV). We developed RAMDeck to solve our own production AI memory constraints during the global DRAM shortage, creating a reliable way to orchestrate on-premise AI across mixed consumer hardware. Read our story on the Our Story page or explore our tools on GitHub.

Reserve a RAMDeck Hub on Indiegogo →