llama.cpp

What llama.cpp does

llama.cpp is the inference engine at the bottom of much of the local AI ecosystem. It runs language and vision models stored in the GGUF format, is written in plain C/C++ without external dependencies, and aims to run well on whatever hardware you have. Ollama, Jan, LocalAI and TextGen all build on it.

You do not need another app to use it, though. The llama-server program (also started as llama serve) loads a model, serves an API and opens a built-in web chat. The chat streams answers and shows reasoning for thinking models. You can branch a conversation by editing or regenerating a message, search past chats, and export or import them as JSON. It accepts images, PDFs, MP3 or WAV audio, and text files for models that can handle them. In router mode, which you get by starting the server without a model, the UI adds a model manager that loads and unloads models on demand and downloads new ones from Hugging Face.

Models and providers

llama.cpp runs GGUF models only. It can download them straight from Hugging Face (-hf user/model), and multimodal models use an extra projector file. It is a purely local engine, with no connections to OpenAI, Anthropic or other hosted providers. Instead, it acts as a provider for other software: llama-server offers OpenAI-compatible chat completions, responses and embeddings, plus an Anthropic-compatible Messages endpoint, so most chat apps and coding tools can use it.

For tools, the server can expose built-in tools that read and write local files, and it can expose tools from MCP servers it launches itself. Both appear in the web chat. There is no built-in web search.

Privacy and data

The web chat keeps conversations in your browser’s IndexedDB, not on the server. The server binds to 127.0.0.1 by default, and the documentation describes no telemetry. Lookups of model metadata on the Hugging Face Hub are off by default. Once you enable the file tools, the server can read and write files with its own permissions. In that case cross-origin access defaults to localhost, and for public deployments the README recommends an API key and a reverse proxy.

Running llama.cpp

You can install it with the script from llama.app, download pre-built binaries from GitHub releases, run the official Docker images, or build it with CMake. The server images come in CUDA 12, CUDA 13, ROCm, MUSA, Intel (SYCL) and Vulkan variants. The project notes that its GPU images are built by CI but not tested beyond the build.

The README states no hardware minimums. It documents what is supported instead:

• CPU: AVX, AVX2, AVX512 and AMX on x86; NEON on ARM; several RISC-V extensions. Partial GPU offload helps when a model is larger than your VRAM.
• Apple: Metal, enabled by default on macOS, plus Accelerate.
• NVIDIA: CUDA. AMD: HIP with ROCm. Intel GPUs: SYCL. Other GPUs: Vulkan.
• Other hardware: Moore Threads (MUSA), Ascend NPUs (CANN), Adreno (OpenCL), Snapdragon Hexagon, WebGPU and IBM Z. OpenVINO support is in progress.

The server handles parallel requests from several users, but its only access control is API keys. It has no user accounts.

Hosted version

The project offers no hosted service. The llama.cpp team and Hugging Face also publish small “Llama” desktop apps for Mac and Windows from llama.app. Those apps live in separate repositories and are not covered by this page.

Limitations

llama.cpp is configured through command-line flags, and choosing quantizations, context size and GPU layers takes some background knowledge. It supports GGUF models only. Chat history stays in one browser, there is no web search, and multi-user deployments need your own authentication in front of the server.

Who it suits

llama.cpp suits technically minded users who want the most direct control over local inference, people with unusual hardware that other runtimes do not support, and anyone who needs a small, scriptable model server. If you want a polished desktop experience, use one of the apps built on top of it.