Markdown

RL libraries for LLMs (verl · slime · SkyRL · …)¶

Scope: comparison/selection overview and index for the open-source RL post-training libraries: how the systems are structured, which inference and training backends they use, how they orchestrate rollouts vs training, and how to choose one. This page maps the field and routes you to the right library; the detailed WHAT/WHY/WHEN/HOW for each library lives on its own dedicated page (see Focused pages below). The systems layer under the methods in fine-tuning and post-training, usually built on Ray (orchestration overview). Framing follows Anyscale's "Open Source RL Libraries for LLMs" survey (References).

The field moves monthly; treat the table as a map, not a spec. Verify a library's current backends and scale on its repo before committing.

Focused pages¶

Pick a library here, then open its page for the implementable detail (full WHAT/WHY/WHEN/HOW):

verl: use this when you need broad backends and high-performance RL, with a colocated default or the newer disaggregated separate_async path.
slime: use this when you want the Megatron + SGLang decoupled async stack (the one behind GLM), especially for MoE.
SkyRL: use this when you want to switch between colocated and disaggregated, or run flexible/agentic workloads.
OpenRLHF: use this when your focus is RLHF with reward models on DeepSpeed.
NeMo-RL: use this when you are on the NVIDIA stack or need async agentic training.
TRL: use this for HF-native simplicity, or to test its experimental local async GRPO path without a Ray control plane.
Tinker: use this when you want post-training without operating any GPUs; a managed training API (LoRA-only) with an open recipe library on top.
Molt: use this when the loop is agent/tool/multi-turn-shaped and you want a small (~8.6K LOC), single-actor, PyTorch-native AutoModel + FSDP2 stack you can read end-to-end.

For the library-by-library tradeoffs that drive this choice, see The landscape and Selection guidance below.

Paradigm: Generator + Trainer, colocated ↔ disaggregated¶

Every RL-for-LLM system decomposes into two components plus a controller:

Generator (rollout) runs inference (vLLM/SGLang) to sample completions and interacts with the environment / reward. Single-turn or multi-turn/agentic.
Trainer runs the policy update (PPO/GRPO/DPO, fine-tuning and post-training) on FSDP / DeepSpeed / Megatron.
Controller coordinates the two, usually Ray (orchestration overview).

The defining design axis is how generator and trainer share GPUs:

Colocated / tightly-coupled (for example, verl's default): rollout and training share GPUs, swapping or offloading between phases. This maximizes phase reuse but raises memory pressure.
Disaggregated / decoupled (for example, slime, SkyRL, AReaL, or verl separate_async): separate rollout and training pools enable async generation, straggler tolerance, and heterogeneous hardware, matching the disaggregation idea in serving (disaggregated inference).

Architecture: generator/trainer loop

flowchart LR
  subgraph Gen["Generator (rollout)"]
    ENG["Inference engine: vLLM / SGLang"]
    ENV["Environment / reward"]
  end
  subgraph Train["Trainer"]
    OPT["PPO / GRPO on FSDP / Megatron"]
  end
  Gen -->|"rollouts + rewards"| Train
  Train -->|"updated weights"| Gen
  CTRL["Controller: Ray"] -.-> Gen
  CTRL -.-> Train

Architecture: colocated vs disaggregated

flowchart TB
  subgraph Colo["Colocated: verl default"]
    G1["Rollout"] <-->|"offload / swap"| T1["Train"]
  end
  subgraph Disagg["Disaggregated: slime, SkyRL, verl separate_async"]
    G2["Rollout pool (SGLang)"] -->|"async rollouts"| T2["Train pool (Megatron)"]
    T2 -->|"weight sync"| G2
  end

The landscape (mid-2026)¶

Library	Origin	Trainer	Rollout	Orchestration	Coupling	Best for
verl	ByteDance	FSDP/FSDP2/Megatron	vLLM/SGLang	Ray	colocated default or disaggregated `separate_async`	large-scale, performance
slime	THUDM / Z.ai	Megatron only	SGLang	Ray	decoupled, async-first	high-perf MoE; powers GLM
SkyRL	UC Berkeley	FSDP2/Megatron/JAX	vLLM/OpenAI	Ray	colocated or disaggregated	flexible, agentic
OpenRLHF	community	DeepSpeed	vLLM	Ray	async + colocation	RLHF, reward models
NeMo-RL	NVIDIA	FSDP2/Megatron	vLLM/SGLang	Ray	async	agentic, NVIDIA stack
ROLL	Alibaba	FSDP2/Megatron	vLLM/SGLang	Ray	flexible	multi-purpose
AReaL	Ant Group	FSDP2/Megatron	vLLM/SGLang	optional Ray	async, interruptible	long rollouts, stragglers
TRL	Hugging Face	HF Trainer	vLLM/HF	none	server/colocate; experimental local async	HF-native and smaller-scale runs
Tinker	Thinking Machines	managed service (LoRA)	service sampling API	none (client loop)	training-as-a-service	post-training without a cluster
Verifiers	Prime Intellect	own + prime-rl	vLLM/OpenAI	none	—	multi-turn env, research
RAGEN	community	on verl (FSDP/Megatron)	vLLM/SGLang	Ray	on verl	multi-turn agentic
Molt	NVIDIA (NeMo Labs)	AutoModel + FSDP2	vLLM	Ray	async, single-actor	small codebase, agentic-first, frontier MoE

How they relate to the rest of this KB¶

slime is the RL stack behind the GLM models (serving open-weight models): Megatron training + SGLang rollouts, decoupled and async; opinionated and minimal.
verl is the high-performance default referenced in fine-tuning and post-training; colocated by default for throughput, with async/agentic modes added.
All of these juggle a rollout engine and a trainer as separate process groups, which is why Ray (orchestration overview) is the common controller.

Selection guidance¶

Choose by the row that matches your constraint, then open that library's page:

Maximum performance / scale → verl or slime.
Flexibility / research / agentic → SkyRL or Verifiers.
Multi-turn / environment-heavy → RAGEN, NeMo-RL, SkyRL, or Molt.
Small, forkable, agentic-first codebase → Molt.
Simplicity / single-node → TRL (fine-tuning and post-training).
RLHF with reward models → OpenRLHF.
No cluster at all / managed service → Tinker.

Agentic and environment support¶

Multi-turn / tool-using RL is where the libraries diverge most, because the rollout stops being a single generate call and becomes a loop with an environment in it (agentic RL).

The environment layer is a separate concern from the trainer. Prime Intellect's Verifiers (PrimeIntellect-ai/verifiers) is the environment/protocol library, not a trainer; it plugs into prime-rl and any OpenAI-compatible endpoint. RAGEN layers multi-turn agentic rollouts on verl. verl and NeMo-RL expose native agent loops; SkyRL targets flexible/agentic workloads directly.
Ask how each library handles the rollout token buffer. A multi-turn loop that re-renders messages and re-tokenizes between turns silently corrupts the importance ratio, because decode then encode is not injective. verl's AgentLoopOutput carries a response_mask ("1 for LLM generated token, 0 for tool response token") and TRL's GRPOTrainer concatenates token IDs without re-tokenizing, checking the chat template for prefix preservation at init. This is a correctness question, not a performance one: see Token-In, Token-Out.
The tool executor is part of the capacity plan. Sandboxes run alongside the GPUs and sit on the critical path of every trajectory (agent sandboxing and isolation).

Hardware & networking notes¶

The weight-sync from trainer to rollout happens every step or every few steps. It can use local device paths, NCCL over IB/RoCE, or checkpoint deltas over shared storage; measure the actual transfer and apply path (delta weight sync).
Colocated trades GPUs between phases via offload: memory pressure is the constraint; disaggregated keeps both hot but needs the interconnect to move weights/rollouts.
Run any of these on Ray via KubeRay on the GPU platform (orchestration overview, the Kubernetes platform); expose RDMA into the Ray workers.

Don't-miss checklist¶

Map the choice to the coupling you need: verl supports a colocated default and a narrower disaggregated path; slime and SkyRL remain async-first choices for heterogeneous or straggler-tolerant fleets.
Match rollout backend to the serving stack already in use (SGLang vs vLLM, inference serving).
Budget GPUs for rollout and training; rollouts often dominate wall-clock.
Reuse the platform's Ray/KubeRay + gang scheduler rather than a bespoke stack.
Monitor reward / entropy / KL for collapse regardless of library (observability).

Failure modes¶

Picking a colocated library then hitting memory limits that a disaggregated one would have avoided.
Rollout backend mismatched to the model's best engine (e.g. forcing vLLM where SGLang prefix caching wins).
Under-provisioned rollout pool → training GPUs idle (fine-tuning and post-training).
Treating any of these as turnkey: RL stability (entropy/KL) still needs active management.

Open questions & validation¶

Confirm each candidate library's current trainer/rollout backends and Ray dependency on its repo.
Validate the trainer↔rollout weight-sync path and bandwidth for the chosen coupling.
Benchmark a small GRPO run end-to-end before scaling; measure rollout vs train time split.

References¶

Anyscale — Open Source RL Libraries for LLMs: https://www.anyscale.com/blog/open-source-rl-libraries-for-llms
verl: https://github.com/verl-project/verl · AgentLoop docs: https://verl.readthedocs.io/en/latest/advance/agent_loop.html
slime (THUDM): https://github.com/THUDM/slime · docs: https://thudm.github.io/slime/
SkyRL (UC Berkeley): https://github.com/NovaSky-AI/SkyRL
OpenRLHF: https://github.com/OpenRLHF/OpenRLHF · NeMo-RL: https://github.com/NVIDIA-NeMo/RL
AReaL (Ant Group): https://github.com/areal-project/AReaL
Verifiers (Prime Intellect, environment/protocol layer): https://github.com/PrimeIntellect-ai/verifiers
TRL: https://github.com/huggingface/trl
Tinker cookbook: https://github.com/thinking-machines-lab/tinker-cookbook
Molt (NVIDIA NeMo Labs): https://github.com/NVIDIA-NeMo/labs-molt
Ray (controller): https://docs.ray.io/en/latest/