Train transformer language models with reinforcement learning.

★ 19,741⑂ 3,148👁 104⑃ 5,283● 2,914▤ 104
PythonApache-2.0Created 2020-03-27hf.co/docs/trl
Download: v1.18.0v1.18.0 ↓1.18.02026-11-13
+6
Daily
19,741
stars
3,148
forks
104
watchers
5,283
prs
2,914
issues
104
releases

README

TRL - Transformers Reinforcement Learning


    
        
        
    


 


    A comprehensive library to post-train foundation models



    
    
    
    


 🎉 What's New

⚡ AsyncGRPOTrainer is now stable: AsyncGRPOTrainer graduates to the stable API. It runs GRPO much faster by decoupling generation from training: a vLLM server streams rollouts while the trainer updates the model.

 Overview

TRL is a cutting-edge library designed for post-training foundation models using advanced techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). Built on top of the 🤗 Transformers ecosystem, TRL supports a variety of model architectures and modalities, and can be scaled-up across various hardware setups.

 Highlights

- Trainers: Various fine-tuning methods are easily accessible via trainers like SFTTrainer, GRPOTrainer, AsyncGRPOTrainer, DPOTrainer, KTOTrainer and more.

- Efficient and scalable:
  - Leverages 🤗 Accelerate to scale from single GPU to multi-node clusters using methods like DDP and DeepSpeed.
  - Full integration with 🤗 PEFT enables training on large models with modest hardware via quantization and LoRA/QLoRA.
  - Integrates 🦥 Unsloth for accelerating training using optimized kernels.

- Command Line Interface (CLI): A simple interface lets you fine-tune with models without needing to write code.

 Installation

 Python Package

Install the library using pip:

bash
pip install trl


 From source

If you want to use the latest features before an official release, you can install TRL from source:

bash
pip install git+https://github.com/huggingface/trl.git


 Repository

If you want to use the examples you can clone the repository with the following command:

bash
git clone https://github.com/huggingface/trl.git


 Quick Start

For more flexibility and control over training, TRL provides dedicated trainer classes to post-train language models or PEFT adapters on a custom dataset. Each trainer in TRL is a light wrapper around the 🤗 Transformers trainer and natively supports distributed training methods like DDP, DeepSpeed ZeRO, and FSDP.

 SFTTrainer

Here is a basic example of how to use the SFTTrainer:

python
from trl import SFTTrainer
from datasets import load_dataset

dataset = load_dataset("trl-lib/Capybara", split="train")

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,
)
trainer.train()


 GRPOTrainer

GRPOTrainer implements the Group Relative Policy Optimization (GRPO) algorithm that is more memory-efficient than PPO and was used to train Deepseek AI's R1. accuracy_reward needs math-verify, which you can install with pip install trl[math_verify].

python
from datasets import load_dataset
from trl import GRPOTrainer
from trl.rewards import accuracy_reward

dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=accuracy_reward,
    train

Top issues

Discussions

Latest release

v1.18.0 v1.18.0 2026-11-13
Features

 [!WARNING]
 The submodules of trl.experimental.async_grpo are gone. from trl.experimental.async_grpo.async_grpo_config import AsyncGRPOConfig, and imports from async_rollout_worker, packing, vllm_client or weight_transfer in that package, no longer work. Import the trainer from trl, the rollout loop from trl.trainer.async_grpo_trainer, and the vLLM client from trl.generation.async_vllm_client (renamed AsyncVLLMClient). from trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer still works, with a FutureWarning, until v2.0.0.

 :mortar_board: AsyncGRPOTrainer is n

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON