llama.cpp
llama.cpp is a C/C++ library and set of tools for running large language models (LLMs) and vision-language models (VLMs) locally with minimal setup and high performance. It supports a wide range of hardware including Apple Silicon, x86, RISC-V, and various GPUs. It is intended for developers who want to integrate LLM inference into their applications or run models locally.
🎬 Local AI just leveled up... Llama.cpp vs Ollama · Alex Ziskind
✨ Key features
- Plain C/C++ implementation with no dependencies
- Optimized for Apple Silicon via ARM NEON, Accelerate, and Metal
- Supports AVX, AVX2, AVX512, and AMX on x86
- Integer quantization from 1.5-bit to 8-bit for reduced memory
- Custom CUDA kernels for NVIDIA GPUs, plus HIP and MUSA support
- CPU+GPU hybrid inference for models larger than VRAM
🎯 Use cases
- Run LLMs locally on laptops or desktops
- Deploy an OpenAI-compatible API server for applications
- Build custom chat interfaces or CLI tools
- Perform inference on edge devices with limited resources
📦 Installation
🧰 Requirements: No specific runtime or OS requirements stated; can be built from source or run via Docker or pre-built binaries.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cppmakeAlternatively, download pre-built binaries from the releases page or use Docker (see docs/docker.md).
🚀 Usage
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUFllama serve -hf ggml-org/Qwen3.5-0.8B-GGUF⚠️ Good to know
The project is under active development; some backends like OpenVINO are marked as 'In Progress'.
❓ FAQ
What hardware is supported?
It supports a wide range: Apple Silicon, x86 with AVX/AVX2/AVX512, RISC-V, NVIDIA GPUs via CUDA, AMD GPUs via HIP, Intel GPUs via SYCL, and more via Vulkan, Metal, and other backends.
How can I install llama.cpp?
You can build from source, use Docker, download pre-built binaries from the releases page, or use the llama.app installer.
Can I run models larger than my GPU memory?
Yes, llama.cpp supports CPU+GPU hybrid inference to partially accelerate models larger than total VRAM capacity.
Does it support quantization?
Yes, it supports integer quantization from 1.5-bit to 8-bit for faster inference and reduced memory use.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.