vLLM
vLLM is a fast and easy-to-use library for LLM inference and serving, offering state-of-the-art serving throughput and efficient memory management via PagedAttention. It solves the problem of high-performance, cost-effective LLM deployment for developers and researchers. It is designed for anyone serving large language models in production or research.
🎬 What is vLLM? Efficient AI Inference for Large Language Models · IBM Technology
✨ Key features
- State-of-the-art serving throughput with continuous batching and PagedAttention.
- Flexible execution with CUDA/HIP graphs, torch.compile, and optimized kernels.
- Supports quantization formats like FP8, INT8, GPTQ, AWQ, GGUF, and more.
- Seamless integration with 200+ Hugging Face model architectures.
- OpenAI-compatible API server, plus Anthropic Messages API and gRPC.
- Distributed inference with tensor, pipeline, data, expert, and context parallelism.
🎯 Use cases
- Serve LLMs for production applications with high throughput and low latency.
- Run multi-modal models like LLaVA or Qwen-VL for vision-language tasks.
- Deploy embedding models for retrieval and search systems.
- Use speculative decoding to accelerate inference for autoregressive models.
- Serve multiple LoRA adapters efficiently for fine-tuned models.
📦 Installation
🧰 Requirements: Requires Python and pip or uv; supports NVIDIA, AMD, Intel GPUs, and CPUs; no API keys needed.
🚀 Usage
from vllm import LLM
llm = LLM(model="facebook/opt-125m")
output = llm.generate("Hello, my name is")
print(output)
For an OpenAI-compatible server, run:
vllm serve facebook/opt-125m
❓ FAQ
What hardware does vLLM support?
vLLM supports NVIDIA GPUs, AMD GPUs, Intel GPUs, and x86/ARM/PowerPC CPUs, with additional plugins for Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, and more.
Does vLLM support Hugging Face models?
Yes, vLLM seamlessly supports over 200 model architectures from Hugging Face, including decoder-only, MoE, multi-modal, embedding, and reward models.
How do I install vLLM?
You can install vLLM using uv pip install vllm or pip install vllm. For development, you can build from source.
Does vLLM provide an OpenAI-compatible API?
Yes, vLLM includes an OpenAI-compatible API server, as well as support for Anthropic Messages API and gRPC.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.