LocalAI
LocalAI is an open-source, self-hosted AI engine that runs LLMs, vision, voice, image, and video models on any hardware, including CPU-only machines. It provides drop-in API compatibility with OpenAI, Anthropic, and ElevenLabs, and is designed for developers and organizations seeking privacy-preserving, on-premises AI inference.
🎬 LocalAI Introduction - Install, configure and build on your own AI platform · Richard Palethorpe
✨ Key features
- Runs any model: LLMs, vision, voice, image, video.
- No GPU required; supports NVIDIA, AMD, Intel, Apple Silicon, Vulkan.
- Drop-in API compatibility with OpenAI, Anthropic, ElevenLabs.
- Modular backends pulled on demand; install only what you need.
- Multi-user support with API key auth, quotas, role-based access.
- Built-in AI agents with tool use, RAG, MCP, and skills.
🎯 Use cases
- Self-hosted AI inference for privacy-sensitive applications.
- Run LLMs and other models on CPU-only or edge hardware.
- Build multi-user AI platforms with quotas and authentication.
- Deploy autonomous agents with tool use and RAG.
- Replace OpenAI/Anthropic APIs with a local drop-in server.
📦 Installation
🧰 Requirements: Runs on Linux, macOS, or Windows via Docker; no GPU required, but optional GPU acceleration supported. No API keys needed for local use.
Containers (Docker, podman, ...)
CPU only:
docker run -ti --name local-ai -p 8080:8080 localai/localai:latest
NVIDIA GPU:
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-13docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64-cuda-13AMD GPU (ROCm):
docker run -ti --name local-ai -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add=video localai/localai:latest-gpu-hipblas
Intel GPU (oneAPI):
docker run -ti --name local-ai -p 8080:8080 --device=/dev/dri/card1 --device=/dev/dri/renderD128 localai/localai:latest-gpu-intel
Vulkan GPU:
docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-gpu-vulkan
For macOS, download the DMG from the releases page and run `sudo xatt
🚀 Usage
Loading models
local-ai run llama-3.2-1b-instruct:q4_k_mlocal-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguflocal-ai run ollama://gemma:2blocal-ai run https://gist.githubusercontent.com/.../phi-2.yamllocal-ai run oci://localai/phi-2:latestTo work with a running LocalAI server from the terminal, start the built-in agent from another shell. It answers questions, reads your files and runs commands on your machine, asking you to approve anything that changes state. Inside a session, /models lists installed models and /model <name> switches between them. See the Terminal agent docs.
local-ai run llama-3.2-1b-instruct:q4_k_mlocal-ai chat --model llama-3.2-1b-instruct:q4_k_m❓ FAQ
Does LocalAI require a GPU?
No, LocalAI can run on CPU-only hardware. GPU acceleration is optional and supported for NVIDIA, AMD, Intel, Apple Silicon, and Vulkan.
How do I install LocalAI?
The quickest way is to use Docker: docker run -ti --name local-ai -p 8080:8080 localai/localai:latest for CPU-only. GPU-specific images are also available.
Can I use LocalAI as a drop-in replacement for OpenAI?
Yes, LocalAI offers drop-in API compatibility with OpenAI, Anthropic, and ElevenLabs APIs across all backends.
How do I load a model?
Use the local-ai run command with a model reference, such as local-ai run llama-3.2-1b-instruct:q4_k_m from the gallery, or from Huggingface, Ollama, or an OCI registry.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.