LocalAI

What LocalAI does

LocalAI is a server you run on your own hardware to provide AI models through the same APIs that hosted services use. It goes well beyond chat. The same server can generate text, describe images, transcribe and synthesize speech, hold realtime voice conversations, generate images and video, and produce embeddings and rerankings. It does this by plugging in other engines as “backends”, such as llama.cpp, vLLM, whisper.cpp, diffusers and MLX, along with several C++ engines the project builds itself. Each backend is downloaded as a separate image only when a model needs it.

A web UI runs alongside the API. Its chat page keeps a list of conversations and accepts attached images, audio and documents. It can also dictate through a local transcription model, branch a conversation from any reply, open code in a side canvas and export chats as Markdown. Other pages manage the model gallery, run built-in agents that have tools and document knowledge bases, and handle users.

Models and providers

Models come from LocalAI’s gallery, from Hugging Face, from the Ollama registry or from any OCI registry. LocalAI is local by design: the chat runs models installed on the server. Individual agents can be pointed at an external OpenAI-compatible provider, but that is the exception. The main reason to run LocalAI is to be a provider yourself. It offers OpenAI-, Anthropic-, ElevenLabs- and Ollama-compatible APIs, so other chat apps, scripts and tools can use your hardware in place of a cloud service. MCP tools work in both the chat and the agents. Agents can search the web and run code through their configurable actions.

Privacy and data

Prompts and outputs stay on the machine running LocalAI unless you connect an agent to an outside service. Without authentication, the web UI stores chats in the browser. With user authentication turned on, each user’s history is also kept on the server, in SQLite by default or PostgreSQL if you configure it, and syncs across devices. One point matters for anyone exposing LocalAI on a network: if you configure neither API keys nor user accounts, the API and UI are open to anyone who can reach the port. Once authentication is configured, every route is private by default. The documentation pages we checked describe no telemetry.

Self-hosting LocalAI

The recommended install is a container, with images for CPU-only use, NVIDIA (CUDA 12 and 13), AMD (ROCm), Intel (oneAPI/SYCL), Vulkan and NVIDIA Jetson. Server binaries, a launcher app for macOS (unsigned) and Linux, and Kubernetes instructions are also available. No GPU is required. LocalAI detects your hardware and downloads matching backends. The GPU guide lists these requirements:

• NVIDIA: the NVIDIA container toolkit on the host.
• AMD: a ROCm 7 compatible GPU on Ubuntu 22.04 or 24.04, RHEL 9 or SLES 15, with amdgpu-dkms and ROCm 7.0 or newer installed, plus at least 100 GB of free disk space.
• Intel: the SYCL/oneAPI images.
• Apple Silicon: Metal through the macOS builds.

According to the FAQ, quantized 7B to 13B models take about 4 to 8 GB on disk, and models over 30B take 15 GB or more.

For teams, LocalAI has real multi-user features. These include accounts with admin and user roles, GitHub OAuth and OIDC single sign-on (for example Keycloak, Okta or Authentik), per-user API keys, quotas and usage statistics. Distributed mode spreads models across several machines using PostgreSQL and NATS.

Hosted version

The project offers no hosted service.

Limitations

The feature list is very large and changes quickly, so expect to read the documentation and tune model configuration files. The chat page works well, but web search and other tools come through agents or MCP servers rather than a simple switch. Security is opt-in: an unconfigured instance has no authentication. GPU containers need host drivers and container runtimes set up first.

Who it suits

LocalAI suits self-hosters and teams who want one private server that covers chat, speech and image work through OpenAI-compatible APIs, shared by several users with proper accounts. For a single person who just wants to chat with a local model on a laptop, a desktop app such as Jan is simpler.