NVIDIA Molt
Molt is an agentic-first reinforcement learning (RL) framework for research, built on Ray, vLLM, and NVIDIA AutoModel. It provides a minimal PyTorch-native stack for fully-async, multimodal, multi-turn agentic RL at scale, targeting researchers who need a hackable, small codebase that scales to frontier MoE models.
✨ Key features
- Agentic-first API with Env.step() or ChatAgent.run()
- Fully-async runtime with Ray placement and vLLM engines
- PyTorch-native training with FSDP2 and NVIDIA AutoModel
- Single-actor RL topology with optional PPO critic
- Supports TP/EP/CP parallelism for MoE models
- Token-first contract for aligned data across rollout and training
🎯 Use cases
- Training language models with reinforcement learning from human feedback (RLHF)
- Developing multi-turn tool-calling agents
- Fine-tuning vision-language models (VLMs) with RL
- Running large-scale RL experiments on MoE models like DeepSeek-V3
- Implementing custom reward functions in Python for agentic tasks
📦 Installation
🧰 Requirements: Requires CUDA 13 and a compatible GPU (A100/H100/H200/B200/GB200); the container is recommended for older host drivers.
git clone https://github.com/NVIDIA-NeMo/labs-molt.git
cd labs-molt
pip install -e ".[vllm]" # local development only — the container bakes everything in
Or use the prebuilt container:
docker pull hijkzzz/molt:latest # or a pinned release: hijkzzz/molt:0.1.7
Or install from PyPI:
pip install "molt-rl[vllm]"
🚀 Usage
python3 -m molt.cli.train_rl_ray \
--actor.model_name_or_path /path/to/automodel \
--data.prompt_dataset /path/to/prompts.jsonl \
--data.input_key input \
--train.agent_path examples/python/agents/math.py \
--vllm.num_engines 2 \
--vllm.tensor_parallel_size 2 \
--rollout.batch_size 128 \
--train.batch_size 128 \
--train.micro_batch_size 1 \
--algo.advantage.estimator reinforce \
--algo.kl.init_coef 0 \
--fsdp.attn_implementation te \
--ckpt.output_dir ./ckpt/rl
⚠️ Good to know
The HF transformers path is a non-preferred fallback with limited support (text only, no CP/EP/TP), and the Muon optimizer is experimental with no consistent win over Adam.
❓ FAQ
What is the primary model backend for Molt?
NVIDIA AutoModel is the primary path, supporting native CP/EP/TP parallelism and custom MoE parallelization. The HF transformers path is a fallback with limited features.
Can I use Molt with CUDA 12?
No, Molt requires CUDA 13 because the git-pinned AutoModel is only compatible with the CUDA-13 torch build. If your host driver is older, use the container which includes a forward-compatibility layer.
What RL algorithms does Molt support?
Molt supports estimators like reinforce, reinforce_baseline, rloo, grpo, dr_grpo, gae (PPO), and on_policy_distill, with optional PPO critic and KL regularization.
How do I install Molt?
You can clone the repo and do an editable install, pull the prebuilt Docker container, or install from PyPI with pip install molt-rl[vllm].
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.