Markdown

verl (Volcano Engine RL)¶

Scope: ByteDance's RL post-training library and HybridFlow controller, including its colocated default and the newer disaggregated separate_async path.

Reference templates use real APIs, but the delta backend is newer than verl v0.8.0. Pin an exact commit and validate the selected trainer, engine, and config together.

What it is¶

verl is the open-source implementation of HybridFlow (EuroSys 2025), a hybrid-controller programming model for RL post-training. A single controller expresses rollout, reward, advantage, and update, while Ray worker groups execute the heavy compute. Training backends include PyTorch FSDP/FSDP2 and Megatron-LM; rollout engines include vLLM, SGLang, and HF Transformers. The algorithm catalog includes PPO, GRPO, GSPO, DAPO, RLOO, REINFORCE++, ReMax, and PRIME. See RL libraries for selection context.

Why use it¶

Broad backends and recipes. The same controller composes multiple trainers, inference engines, and policy-gradient variants.
Efficient colocation. The HybridEngine reshards an actor between training and generation layouts and reuses one GPU pool.
Independent pools when required. separate_async decouples training and rollout; the main-only delta_sharded backend can reduce supported SGLang weight transfers.

When to use it (and when not)¶

Use verl when large-scale RL needs a broad recipe ecosystem and either a colocated default or independently sized Ray pools.
Choose colocation when phase reuse and fast local synchronization dominate. Choose separate_async when rollout elasticity, memory isolation, or placement matters more.
Prefer slime for an opinionated Megatron plus SGLang async stack, or SkyRL for heavily agentic multi-turn loops. verl's sparse disaggregated path has a narrower support matrix than its established colocated path.

Architecture¶

flowchart TB
  CTRL["Single controller (HybridFlow)"]
  CTRL --> MODE{"Execution mode"}
  MODE -->|"hybrid_engine=true"| COLO["Colocated train and rollout pool"]
  MODE -->|"hybrid_engine=false; separate_async"| SPLIT["Independent train and rollout pools"]
  COLO --> ACT["Actor: FSDP, FSDP2, or Megatron"]
  SPLIT --> ACT
  COLO --> ROLL["Rollout: vLLM or SGLang"]
  SPLIT --> ROLL
  ROLL -->|"trajectories"| CTRL
  CTRL -->|"advantages and update"| ACT
  ACT -->|"full or delta weight sync"| ROLL
  RAY["Ray orchestration"] -.-> COLO
  RAY -.-> SPLIT

How to use it¶

pip install verl            # or build from source: pip install -e .
# (the NVIDIA/ROCm docker image is recommended for matched CUDA + vLLM/SGLang)

# Prepare a dataset (parquet), then run a PPO job on GSM8K:
PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
  data.train_files=$HOME/data/gsm8k/train.parquet \
  data.val_files=$HOME/data/gsm8k/test.parquet \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  actor_rollout_ref.rollout.name=vllm \
  trainer.n_gpus_per_node=1 trainer.nnodes=1 \
  trainer.total_epochs=15

Config is Hydra: every key is overridable on the CLI. main_ppo is the shared entrypoint for PPO and its critic-free variants.

How to develop with it¶

The sharded sparse protocol in PR #6974 snapshots each rank's local shard, offsets local changes into global flat positions, gathers the sparse payload, and checks each flush. This numpy model was executed and asserts exact reconstruction, the no-change boundary, and index corruption detection:

# verl_delta_shard_model.py: executed protocol model.
import hashlib
import numpy as np


before = np.arange(24, dtype=np.uint16)
after = before.copy()
after[[0, 9, 23]] ^= np.uint16(7)

position_parts = []
value_parts = []
for rank, (old_shard, new_shard) in enumerate(
    zip(np.split(before, 3), np.split(after, 3))
):
    local = np.flatnonzero(old_shard != new_shard).astype(np.int32)
    position_parts.append(local + rank * old_shard.size)
    value_parts.append(new_shard[local])

positions = np.concatenate(position_parts)
values = np.concatenate(value_parts)
digest = hashlib.sha256(positions.tobytes() + values.tobytes()).digest()

rebuilt = before.copy()
rebuilt[positions] = values
assert np.array_equal(rebuilt, after)
unchanged_parts = zip(np.split(before, 3), np.split(before.copy(), 3))
assert sum(np.count_nonzero(old != new) for old, new in unchanged_parts) == 0

bad_positions = positions.copy()
bad_positions[0] += 1
assert hashlib.sha256(bad_positions.tobytes() + values.tobytes()).digest() != digest
print("sharded_delta exact=True unchanged=0 corruption_detected=True")

Executed output:

sharded_delta exact=True unchanged=0 corruption_detected=True

GRPO is PPO without a critic, selected by the advantage estimator. Group sampling needs rollout.n > 1:

python3 -m verl.trainer.main_ppo \
  algorithm.adv_estimator=grpo \
  data.train_batch_size=1024 \
  actor_rollout_ref.rollout.name=vllm \
  actor_rollout_ref.rollout.n=8 \
  actor_rollout_ref.actor.use_kl_loss=True \
  algorithm.kl_ctrl.kl_coef=0.001

Custom reward: point custom_reward_function.path at a Python file exposing a scoring function (data_source, solution_str, ground_truth, extra_info) -> float. Datasets are parquet with a prompt column plus a reward_model field carrying the ground truth. See examples/grpo_trainer/ for ready scripts (e.g. run_qwen3_8b_fsdp.sh, run_qwen3_8b_megatron.sh).

How to scale it¶

The sharded delta backend merged after v0.8.0 and is available on main, not in that release. Its supported path is separate_async, hybrid_engine=False, SGLang rollout, BF16, NCCL, and FSDP1 or FSDP2 sharded on dimension 0. The first synchronization is dense; later synchronizations gather absolute int32 positions and replacement values from each rank. vLLM, TensorRT-LLM, Megatron actors, quantized rollout weights, and other shard dimensions remain outside the merged support matrix.

# Reference fragment for a pinned V1 separate_async or bundled one-step-off
# launcher after merge commit 903d90cc44ddbd06624a93e3aad7e15d92af5d99.
DELTA_ARGS=(
  actor_rollout_ref.hybrid_engine=False
  actor_rollout_ref.rollout.name=sglang
  actor_rollout_ref.actor.strategy=fsdp2
  actor_rollout_ref.rollout.checkpoint_engine.backend=delta_sharded
  +actor_rollout_ref.rollout.checkpoint_engine.engine_kwargs.delta_sharded.encoding=indices
)

The code accepts only indices. A bundled launcher comment also names deltas, but the executable assertion rejects it. The same launcher sets actor.fsdp_config.strategy=fsdp2, while the merged documentation says only the top-level actor.strategy selects FSDP2; follow the documented top-level key. Each parameter shard must contain fewer than 2^31 elements because positions are int32. See delta weight sync for recovery and integrity requirements.

Multi-node is Ray plus more workers. Start a head node, attach workers, then submit the same main_ppo command with trainer.nnodes raised:

ray start --head --port=6379 --num-gpus=8        # head
ray start --address=<HEAD_IP>:6379 --num-gpus=8  # each worker
python3 -m verl.trainer.main_ppo \
  trainer.nnodes=4 trainer.n_gpus_per_node=8 \
  actor_rollout_ref.actor.strategy=fsdp2 \
  actor_rollout_ref.actor.fsdp_config.param_offload=True

For very large models switch the strategy to megatron and set tensor/pipeline parallel sizes (tensor parallelism/pipeline parallelism). Colocation offloads optimiser/params to host between phases to free HBM for rollout.

Inference¶

verl does not serve traffic; the rollout step uses an inference engine (vLLM or SGLang) internally to sample completions. Choose it with actor_rollout_ref.rollout.name=vllm|sglang and tune rollout.gpu_memory_utilization, rollout.tensor_model_parallel_size, and prefix caching. For production serving of the trained model, export the checkpoint and serve via the standalone inference stack.

Fine-tuning¶

verl is a post-training (RL) library: GRPO/PPO/DAPO on top of an SFT or instruct base. It is the high-performance default referenced from the post-training overview; for the GRPO method itself see GRPO. Pair an SFT/LoRA warm-start (SFT and LoRA) with verl RL for the full recipe.

How to run it in production¶

Pin verl, the trainer backend, SGLang or vLLM, Ray, and CUDA in one image. For colocated runs, alert on phase-transition time, host-offload traffic, and rollout OOMs. For disaggregated runs, record policy version at generation and learner consumption, then enforce the allowed staleness window. With delta_sharded, export full-sync and delta-sync latency, changed-element density, checksum failures, and dense-fallback count; retain the full checkpoint engine as a recovery path.

Optimised hardware¶

Weight resync is the actor-to-rollout hot path. Colocation keeps it on-device or over NVLink/NVSwitch. The main-only delta_sharded path reduces bytes for a supported disaggregated SGLang and FSDP configuration, but adds CPU snapshots, sparse gathers, checksums, and receiver apply work.
Phase offload: param/optimiser offload to host (fsdp_config.param_offload, optimizer_offload) trades PCIe/CPU bandwidth for HBM headroom during rollout.
Megatron multi-node uses NCCL over InfiniBand/RoCE with GDR: set NCCL_IB_HCA; NCCL auto-selects the GDR distance cutoff from PCIe topology, so only override NCCL_NET_GDR_LEVEL (e.g. SYS) if profiling shows the wrong cutoff, and confirm [GDRDMA] in NCCL_DEBUG=INFO (networking fabric/performance tuning). Blackwell FP8/NVFP4 rollout precision can cut generation cost; verify reward parity against BF16 (the Blackwell platform).

How to maintain it¶

Upgrade the trainer and rollout engine together, then rerun a small reward-parity job and the exact weight-reconstruction test. Re-check Hydra keys against the selected entrypoint because the established main_ppo, V1 separate_async, and experimental one-step-off launchers do not expose identical configuration surfaces. For delta_sharded, audit the upstream documentation, bundled launcher, and executable assertions at the pinned commit; the current strategy and encoding inconsistencies demonstrate why a copied comment is not sufficient evidence.

Cookbook (common use cases)¶

1) GRPO on math (single node, 8 GPU):

python3 -m verl.trainer.main_ppo algorithm.adv_estimator=grpo \
  data.train_files=$HOME/data/gsm8k/train.parquet \
  actor_rollout_ref.model.path=Qwen/Qwen3-8B \
  actor_rollout_ref.rollout.name=sglang actor_rollout_ref.rollout.n=8 \
  trainer.n_gpus_per_node=8 trainer.nnodes=1

2) PPO with a critic (reward-model RLHF shape):

python3 -m verl.trainer.main_ppo algorithm.adv_estimator=gae \
  critic.model.path=$RM_PATH critic.ppo_micro_batch_size_per_gpu=4 \
  algorithm.kl_ctrl.kl_coef=0.001 actor_rollout_ref.rollout.name=vllm

3) Multi-node GRPO (4×8 on Ray): start the Ray head/workers as above, then submit (1) with trainer.nnodes=4 trainer.n_gpus_per_node=8 and actor_rollout_ref.actor.strategy=megatron for large models.

Failure modes¶

OOM during rollout is the signature colocation failure. Enable param/optimiser offload or lower rollout.gpu_memory_utilization.
GRPO needs rollout.n>1; n=1 silently degenerates (no group baseline). Watch reward/entropy/KL for collapse (GRPO).
Backend/version drift: vLLM, SGLang and Megatron move fast; mismatched CUDA/engine versions break the rollout, so pin via the maintained docker image and verify on the repo.
Reward function bugs dominate outcomes; unit-test the scorer on held-out strings before a full run.
Assuming every disaggregated backend supports sparse sync. PR #6974 is SGLang-only, BF16-only, and FSDP shard-dimension-0-only. Unsupported combinations must use their full checkpoint engine.
Treating a checksum as authentication. The per-flush XOR of torch.hash_tensor results catches accidental corruption but is not a cryptographic authenticity check.
Density surprises. The merged docs report 1% to 3% in tested dense-model runs, while a later PR discussion reports roughly 50% for another Qwen2.5 run. Measure payload density for the exact optimizer and model.

References¶

verl PR #6974, sharded delta weight synchronization: https://github.com/verl-project/verl/pull/6974
verl merged delta-weight-sync documentation: https://github.com/verl-project/verl/blob/903d90cc44ddbd06624a93e3aad7e15d92af5d99/docs/advance/delta_weight_sync.md
verl repo: https://github.com/verl-project/verl
verl docs (quickstart, GRPO): https://verl.readthedocs.io/en/latest/start/quickstart.html · https://verl.readthedocs.io/en/latest/algo/grpo.html
HybridFlow paper (EuroSys 2025): https://arxiv.org/abs/2409.19256
GRPO examples: https://github.com/verl-project/verl/tree/main/examples/grpo_trainer
Anyscale — Open Source RL Libraries for LLMs: https://www.anyscale.com/blog/open-source-rl-libraries-for-llms

Related: RL libraries · Post-training · GRPO · Ray · RL cluster bring-up recipe · slime · SkyRL · Delta weight sync · Glossary