Jina-embeddings-v3
Jina-serve is a framework for building and deploying AI services that communicate via gRPC, HTTP, and WebSockets. It solves the problem of scaling AI services from local development to production while focusing on core logic. It is for developers who want to build, deploy, and scale AI services with native support for major ML frameworks and data types.
✨ Key features
- Native support for all major ML frameworks and data types
- High-performance service design with scaling, streaming, and dynamic batching
- LLM serving with streaming output
- Built-in Docker integration and Executor Hub
- One-click deployment to Jina AI Cloud
- Enterprise-ready with Kubernetes and Docker Compose support
🎯 Use cases
- Build gRPC-based AI services using models like StableLM
- Chain multiple AI services into pipelines using Flows
- Deploy scalable AI services to Kubernetes or Docker Compose
- Serve LLMs with token-by-token streaming responses
- Deploy AI services to Jina AI Cloud with a single command
📦 Installation
🧰 Requirements: Python environment with pip; optional GPU for some models; no API keys required for local use.
pip install jina
See guides for Apple Silicon and Windows.
🚀 Usage
from jina import Client
from docarray import DocList
from executor import Prompt, Generation
prompt = Prompt(text='suggest an interesting image generation prompt')
client = Client(port=12345)
response = client.post('/', inputs=[prompt], return_type=DocList[Generation])
❓ FAQ
What are the main layers of Jina-serve?
The three main layers are Data (BaseDoc and DocList), Serving (Executors and Gateway), and Orchestration (Deployments and Flows).
How can I scale a service locally?
You can use replicas for parallel processing, shards for data partitioning, and dynamic batching for efficient model inference.
How do I deploy a service to Kubernetes?
Use jina export kubernetes flow.yml ./my-k8s and then kubectl apply -R -f my-k8s.
Can I stream LLM outputs token by token?
Yes, you can implement a streaming executor that yields tokens as they are generated, and use the client's stream_doc method to receive them.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.