Skip to content
Markdown

TRL (Hugging Face)

Scope: Hugging Face's Transformer Reinforcement Learning library, covering HF-native SFT, preference optimization, online GRPO, experimental local async GRPO, vLLM rollouts, and PEFT.

Python blocks that import trl, peft, or accelerate are GPU reference templates. Each core method is paired with an executed numpy validation. Pin trl, Transformers, Accelerate, PEFT, and vLLM as one compatibility set.

What it is

TRL v1.8.0 is a post-training library built on the HF Trainer and Accelerate, with vLLM rollout for online methods and PEFT for LoRA or QLoRA. Its standard modes use a colocated vLLM engine or a separately launched vLLM server. The experimental AsyncGRPOTrainer adds a spawned rollout process, queued batches, policy versions, a maximum-staleness policy, and full NCCL weight synchronization. It is not a Ray or multi-cluster controller.

Trainer families include:

  • Supervised and offline preference: SFTTrainer, DPOTrainer, CPO, KTO, ORPO, and BCO.
  • Online policy optimization: GRPOTrainer, RLOOTrainer, OnlineDPOTrainer, XPOTrainer, NashMDTrainer, and PPOTrainer.
  • Reward and process models: RewardTrainer and PRMTrainer.
  • Experimental distillation: GKD, scalable external-teacher distillation, GOLD, MiniLLM, SDFT, and SSD trainers.

Why use it

  • Low setup cost. A trainer composes an HF model, dataset, configuration, and optional reward function without a cluster DSL.
  • HF ecosystem integration. It uses Transformers checkpoints, Datasets, Trainer callbacks, Accelerate, PEFT, and Hub workflows directly.
  • Strong small-to-medium path. It is a practical starting point for SFT, DPO, LoRA, and GRPO on one node or a modest Accelerate allocation.

When to use it (and when not)

  • Use TRL for HF-native SFT, preference training, reward modeling, LoRA, and online-RL experiments where the Trainer abstraction is sufficient.
  • Use the experimental async trainer when a local rollout process and bounded queue staleness fit the deployment. It does not supply elastic or multi-region orchestration.
  • Prefer Ray-based systems such as verl, slime, SkyRL, OpenRLHF, or NeMo-RL for large disaggregated fleets or Megatron-scale control.

Architecture

flowchart TB
  DS["datasets"] --> TR["HF Trainer family"]
  PEFT["PEFT adapters"] --- TR
  TR --> MODE{"Online rollout mode"}
  MODE -->|"server or colocate"| VS["vLLM rollout"]
  MODE -->|"experimental AsyncGRPO"| Q["Versioned rollout queue"]
  Q --> PROC["Spawned rollout process"]
  PROC -->|"full NCCL weight sync"| TR
  ACC["Accelerate: DDP, FSDP, or DeepSpeed"] -.-> TR

How to use it

Install, then pick a trainer. SFT is the baseline.

pip install trl                    # add extras as needed: pip install trl[vllm]

Reference template (needs trl and a GPU):

from datasets import load_dataset
from trl import SFTTrainer

dataset = load_dataset("trl-lib/Capybara", split="train")
trainer = SFTTrainer(model="Qwen/Qwen2.5-0.5B", train_dataset=dataset)
trainer.train()

Under the hood SFT is next-token cross-entropy over the completion tokens. The numpy block below reproduces that objective and asserts its endpoints, including a confidently-wrong adversarial case that must exceed the uniform baseline:

import numpy as np


def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)


def sft_token_loss(logits, targets):
    # SFT objective: mean next-token cross-entropy over completion tokens.
    p = softmax(np.asarray(logits, dtype=float))
    picked = p[np.arange(len(targets)), targets]
    return float(-np.log(picked).mean())


V, T = 5, 3
targets = np.array([1, 2, 3])

# Happy path: confident-correct logits drive the loss to ~0.
logits = np.full((T, V), -10.0)
logits[np.arange(T), targets] = 10.0
assert sft_token_loss(logits, targets) < 1e-3

# Reference: uniform logits give exactly ln(V) (max-entropy baseline).
assert abs(sft_token_loss(np.zeros((T, V)), targets) - np.log(V)) < 1e-12

# Adversarial: confidently WRONG predictions exceed the uniform baseline.
wrong = np.full((T, V), -10.0)
wrong[np.arange(T), (targets + 1) % V] = 10.0
assert sft_token_loss(wrong, targets) > np.log(V)

print("SFT cross-entropy: OK")

Executed output:

SFT cross-entropy: OK

DPO (offline preference, no rollouts) is the same shape with a binarized-preference dataset. Reference template (needs trl):

from datasets import load_dataset
from trl import DPOTrainer

ds = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
DPOTrainer(model="Qwen/Qwen3-0.6B", train_dataset=ds).train()

DPO replaces the reward model with a log-ratio between the policy and a frozen reference. The core loss, validated in numpy (init equals ln 2, monotone in the chosen log-prob, and a wrong-direction adversarial case):

import numpy as np


def dpo_loss(lp_chosen, lp_rejected, ref_chosen, ref_rejected, beta=0.1):
    # DPO implicit reward = policy-vs-reference log-ratio (arxiv 2305.18290).
    margin = (lp_chosen - ref_chosen) - (lp_rejected - ref_rejected)
    return float(-np.log(1.0 / (1.0 + np.exp(-beta * margin))))


# At init (policy == reference) the loss is exactly -log(0.5) = ln 2.
assert abs(dpo_loss(-2.0, -3.0, -2.0, -3.0) - np.log(2)) < 1e-12

# Raising the chosen log-prob must reduce the loss monotonically.
losses = [dpo_loss(lp, -5.0, -4.0, -5.0) for lp in [-6.0, -4.0, -2.0, 0.0]]
assert all(a > b for a, b in zip(losses, losses[1:]))

# Adversarial: optimizing the WRONG way (rejected above chosen) exceeds the ln 2 baseline.
assert dpo_loss(-5.0, -1.0, -4.0, -4.0) > np.log(2)

print("DPO loss: OK")

Executed output:

DPO loss: OK

How to integrate with it

TRL consumes the HF ecosystem directly: any datasets split, any transformers checkpoint, Trainer callbacks, and the Hub. Each trainer expects a specific dataset schema (prompt-completion for SFT, chosen/rejected for DPO, prompt-only for GRPO); the trainer's dataset-format guide is authoritative. Integration means composing three things into that Trainer: a reward function (online methods), PEFT adapters, and a vLLM rollout backend.

Custom reward functions (GRPO and the online family)

Online RL is GRPOTrainer plus a reward function (or a built-in). Config goes through the per-trainer *Config (a TrainingArguments subclass). Reference template (needs trl and vllm):

from datasets import load_dataset
from trl import GRPOTrainer, GRPOConfig

dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

def reward_len(completions, **kwargs):                # custom reward
    return [-abs(64 - len(c)) for c in completions]   # toward 64 tokens

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=reward_len,                           # or trl.rewards.accuracy_reward
    args=GRPOConfig(use_vllm=True, vllm_mode="server", bf16=True),
    train_dataset=dataset,
)
trainer.train()

GRPO is critic-free: it normalizes each reward against its own group of completions to the same prompt, so the reward function is the only learning signal you supply. The numpy block validates both reward_len and the group-relative advantage, checks it against a slow explicit-loop reference, and covers the zero-variance divide-by-zero guard:

import numpy as np


def reward_len(completions, target=64):
    # The page's GRPO reward: peak at `target` chars, linear falloff.
    return np.array([-abs(target - len(c)) for c in completions], dtype=float)


def grpo_advantages(rewards, eps=1e-8):
    # Group-relative, critic-free advantage (GRPO, DeepSeekMath arxiv 2402.03300).
    r = np.asarray(rewards, dtype=float)
    return (r - r.mean()) / (r.std() + eps)


# reward_len peaks exactly at the target length and is symmetric around it.
r = reward_len(["x" * n for n in [60, 64, 68]])
assert r.argmax() == 1 and r[0] == r[2] == -4.0

# Advantages are zero-mean; the best sample is positive.
adv = grpo_advantages([1.0, 2.0, 3.0, 10.0])
assert abs(adv.mean()) < 1e-9 and adv.argmax() == 3 and adv[3] > 0


# Reference equivalence: matches an explicit-loop implementation.
def ref_adv(xs):
    m = sum(xs) / len(xs)
    var = sum((x - m) ** 2 for x in xs) / len(xs)
    return [(x - m) / (var ** 0.5 + 1e-8) for x in xs]


assert np.allclose(adv, ref_adv([1.0, 2.0, 3.0, 10.0]))

# Adversarial: a zero-variance group must not divide by zero.
flat = grpo_advantages([5.0, 5.0, 5.0])
assert np.all(np.isfinite(flat)) and np.allclose(flat, 0.0)

print("GRPO advantages + reward_len: OK")

Executed output:

GRPO advantages + reward_len: OK

PEFT / LoRA adapters

Pass a LoraConfig via peft_config; QLoRA adds a 4-bit BitsAndBytesConfig on the model. Reference template (needs trl and peft):

from peft import LoraConfig
from trl import SFTTrainer

peft_config = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                         task_type="CAUSAL_LM")
SFTTrainer(model="Qwen/Qwen2.5-0.5B", train_dataset=dataset,
           peft_config=peft_config).train()

LoRA trains a scaled low-rank update (alpha/r)*B@A added to the frozen weight, so only the small A and B factors move. The numpy block confirms the update is genuinely rank-bounded, that merging is exact, and that a zero-init leaves the base weights bit-identical:

import numpy as np


def lora_delta(A, B, alpha, r):
    # LoRA update: scaled low-rank product added to frozen W (arxiv 2106.09685).
    return (alpha / r) * (B @ A)


rng = np.random.default_rng(0)
d, k, r, alpha = 8, 6, 2, 4
A = rng.standard_normal((r, k))
B = rng.standard_normal((d, r))
W = rng.standard_normal((d, k))

delta = lora_delta(A, B, alpha, r)
# The update is genuinely low rank: rank(delta) <= r << full rank.
assert np.linalg.matrix_rank(delta) <= r < min(d, k)

# Merging is exact: (W + delta) equals the explicit scaled sum.
assert np.allclose(W + delta, W + (alpha / r) * B.dot(A))

# Adversarial: zero-initialized B (LoRA's init) leaves the base weights bit-identical.
B0 = np.zeros((d, r))
assert np.array_equal(W + lora_delta(A, B0, alpha, r), W)

print("LoRA low-rank update: OK")

Executed output:

LoRA low-rank update: OK

Methods, rollouts, and the serving handoff

TRL implements the catalog in fine-tuning and post-training: supervised fine-tuning and LoRA (SFT and LoRA) via SFTTrainer; offline preference optimization (DPO, arxiv 2305.18290) via DPOTrainer; and the critic-free GRPO family (GRPO, DeepSeekMath) via GRPOTrainer, the method TRL highlights for reproducing DeepSeek-R1-style training. Reward-model training (RewardTrainer) feeds RLHF-style pipelines.

Online methods integrate vLLM purely to generate rollouts during training (server or colocate mode, see how to scale it). TRL is a training library: it does not serve models in production. To serve a trained or merged checkpoint, run it under vLLM/SGLang/Dynamo per the serving pages inference serving, serving open-weight models, disaggregated inference. Merge LoRA adapters first (model.merge_and_unload() / peft merge) before serving if you trained with PEFT.

How to run it in production

Separate generation from training (online methods)

For online methods, separate the generation and training GPUs: run a vLLM server on one set, train on the other.

CUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model Qwen/Qwen2.5-0.5B-Instruct
CUDA_VISIBLE_DEVICES=4,5,6,7 accelerate launch train.py   # GRPOConfig(use_vllm=True, vllm_mode="server")

vllm_mode="colocate" instead runs vLLM in the training process group (no separate server), trading isolation for fewer moving parts.

Experimental async GRPO and delta-sync status

TRL v1.8.0 ships AsyncGRPOTrainer under the experimental API. It overlaps a spawned rollout process with training, tags queued batches with policy versions, and can drop data beyond a configured staleness bound. Weight synchronization is still a full NCCL transfer. This provides local async execution without becoming a Ray-based, elastic, or multi-region rollout control plane.

Sparse Hub-bucket sync is not a released TRL feature. PR #5417 is a draft prototype: it keeps full BF16 CPU snapshots at both ends, emits int32 positions plus BF16 values as safetensors, writes a full anchor every ten syncs, and reconstructs full CPU tensors before calling vLLM's ordinary load_weights. It adds no tests, does not validate ordering, and its checksum setting is not wired into the payload. The published 405B figures are stated extrapolations. PR #5937 is an unmerged successor with a different design. Treat both as research branches, not configuration for v1.8.0; see delta weight sync.

Hardware and throughput

  • Accelerate + FSDP/DeepSpeed: shard parameters/optimizer state to fit larger models; HSDP (shard intra-node over NVLink, replicate inter-node over IB) for multi-node SFT/DPO (FSDP, DeepSpeed and ZeRO).
  • QLoRA: 4-bit base + LoRA adapters trains big models on modest GPUs; pair with bf16 and gradient checkpointing.
  • vLLM rollout on separate GPUs: keep generation off the training GPUs (CUDA_VISIBLE_DEVICES split) so PagedAttention throughput isn't fighting the optimizer for memory.
  • NCCL fabric: multi-node Accelerate collectives ride NCCL; on RDMA clusters set NCCL_IB_HCA, verify [GDRDMA] in NCCL_DEBUG=INFO (NCCL auto-selects the GDR cutoff from the detected PCIe topology; only force NCCL_NET_GDR_LEVEL=SYS if profiling shows that auto-selection is wrong), ACS off (performance tuning, networking fabric). Liger Kernel (use_liger_kernel=True) cuts memory and lifts throughput.

Recipes

1. SFT a small instruct model (single GPU) (reference template, needs trl):

from datasets import load_dataset
from trl import SFTTrainer
SFTTrainer(model="Qwen/Qwen2.5-0.5B",
           train_dataset=load_dataset("trl-lib/Capybara", split="train")).train()

2. DPO preference tuning (offline, no rollouts) (reference template, needs trl):

from datasets import load_dataset
from trl import DPOTrainer
DPOTrainer(model="Qwen/Qwen3-0.6B",
           train_dataset=load_dataset("trl-lib/ultrafeedback_binarized", split="train")).train()

3. GRPO with a vLLM server (online, 4+4 GPU split):

CUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model Qwen/Qwen2.5-0.5B-Instruct
CUDA_VISIBLE_DEVICES=4,5,6,7 accelerate launch grpo_train.py   # GRPOConfig(use_vllm=True, vllm_mode="server")

How to maintain it

  • Separate released and draft APIs. AsyncGRPOTrainer is shipped but experimental. Delta-weight-sync PRs #5417 and #5937 are unmerged as of 2026-07-20; pinning TRL v1.8.0 does not provide their flags or Hub protocol.

  • Pin the stack together: TRL v1.x moves fast and trainer signatures plus *Config fields shift across minor versions. Pin trl, transformers, accelerate, peft, vllm, and datasets as one set; the reference templates above target the v1.x API (mid-2026) and should be revalidated on upgrade.

  • Validate the dataset schema before long runs: a wrong format silently mis-trains (see Failure modes); check the trainer's dataset-format guide against a small sample first.
  • Merge LoRA before serving: a PEFT checkpoint served without merging runs as the base model; merge with model.merge_and_unload() (see the serving handoff above).
  • Monitor the training signals: reward, KL, and entropy for online methods; loss and eval for offline. Reward/KL/entropy drift is the leading indicator of GRPO collapse (observability).
  • Checkpoints and the Hub: the HF Trainer writes checkpoints on the save_steps cadence and can push_to_hub; resume long jobs with trainer.train(resume_from_checkpoint=...).

How to scale it

TRL scales via Accelerate, not Ray. One accelerate launch spans multi-GPU and multi-node with DDP (DDP), FSDP (FSDP), or DeepSpeed ZeRO (DeepSpeed and ZeRO), selected by an Accelerate config or a DeepSpeed JSON.

accelerate config                                  # choose FSDP or DeepSpeed, num processes/nodes
accelerate launch --config_file fsdp.yaml train.py
# or DeepSpeed ZeRO-3:
accelerate launch --use_deepspeed --deepspeed_config_file ds_z3.json train.py

Multi-node training is standard Accelerate/torchrun rendezvous on top of this. Standard online methods can scale generation separately by pointing the trainer at a vLLM server on separate GPUs; vllm_mode="colocate" keeps generation in the training process group. The experimental AsyncGRPOTrainer overlaps a spawned rollout process with training and bounds queue staleness, but it is not an elastic disaggregated cluster controller. Past a few nodes, the dedicated Ray stacks (verl, slime, SkyRL, OpenRLHF, NeMo-RL) provide broader online-RL orchestration.

Failure modes

  • GPU contention in online methods: colocating vLLM with training without splitting GPUs causes OOM/throughput collapse; use a server on separate devices or vllm_mode="colocate" deliberately.
  • Dataset format mismatch: each trainer expects a specific schema (prompt-completion vs chosen/rejected vs prompt-only); a wrong format silently mis-trains. Check the trainer's dataset-format guide.
  • Forgetting to merge LoRA: serving a PEFT checkpoint without merging adapters yields the base model's behaviour.
  • Expecting local async to provide Ray-scale control. AsyncGRPOTrainer has a spawned rollout process and queue, but no elastic multi-cluster scheduler; use a dedicated RL stack when that control plane is required.
  • GRPO instability: entropy/advantage collapse and KL drift still apply; monitor reward/KL/entropy (fine-tuning and post-training, observability, GRPO).

References

  • TRL PR #5417, draft Hub-bucket delta-sync prototype: https://github.com/huggingface/trl/pull/5417
  • Hugging Face delta-weight-sync article: https://huggingface.co/blog/delta-weight-sync

  • TRL PR #5937, unmerged successor sparse-sync design: https://github.com/huggingface/trl/pull/5937

  • TRL repo: https://github.com/huggingface/trl

  • TRL docs: https://huggingface.co/docs/trl/index · speeding up (vLLM): https://huggingface.co/docs/trl/main/en/speeding_up_training
  • TRL v1 blog: https://huggingface.co/blog/trl-v1 · co-located vLLM in TRL: https://huggingface.co/blog/vllm-colocate
  • Anyscale: Open Source RL Libraries for LLMs: https://www.anyscale.com/blog/open-source-rl-libraries-for-llms
  • GRPO (DeepSeekMath): https://arxiv.org/abs/2402.03300 · DPO: https://arxiv.org/abs/2305.18290 · LoRA: https://arxiv.org/abs/2106.09685

Related: RL Libraries · Fine-tuning · GRPO · DPO · SFT/LoRA · FSDP · DeepSpeed ZeRO · NeMo-RL · Async RL systems · Delta weight sync · Glossary