Markdown

Asynchronous and disaggregated RL systems¶

Scope: the systems design that scales RL post-training of LLMs, splitting the rollout/generation workload from the policy-training workload and running them asynchronously, the off-policy staleness this introduces, and the truncated importance sampling that keeps it stable. The infrastructure companion to the algorithm in GRPO and the serving split in disaggregated inference.

Reference templates on real APIs; pin versions and validate before production use.

flowchart LR
  subgraph ACT["Actors (generation GPUs)"]
    G1["vLLM / SGLang rollout"]
  end
  subgraph LRN["Learners (training GPUs)"]
    T1["Policy-gradient update (FSDP / Megatron)"]
  end
  G1 -->|"completions + rewards"| Q["Rollout queue / buffer"]
  Q --> T1
  T1 -->|"weight sync (every N steps)"| G1
  T1 -.->|"rollouts from an older policy"| STALE["Off-policy staleness"]
  STALE -.->|"reweight, cap at C"| TIS["Truncated importance sampling"]
  TIS -.-> T1

What it is¶

A policy-gradient run is two very different workloads bolted together: generation (sampling completions from the current policy with an inference engine) and training (a gradient update on those completions). The theory of policy gradients assumes the data is on-policy: every completion is scored and trained on before the next update. In practice, exact on-policy execution is both impractically slow and technically impossible to synchronize perfectly, so modern systems run the two phases on separate GPU pools and let them overlap.¹ The convention, from the OLMo/open-RL tooling, is actors (GPUs dedicated to sampling) and learners (GPUs taking the RL steps), with a distributed library such as Ray passing rollouts from actors to learners and weights back the other way.¹

Why asynchrony¶

No idle compute. In a synchronous loop the trainer sits idle while the generator works and vice versa. Overlapping them keeps both pools busy, the same motivation as disaggregated inference, applied to training.
The straggler problem dominates reasoning RL. Reasoning models emit 10K-100K+ tokens per answer, so generation is the bottleneck. In a synchronous batch, one slow prompt (more tokens, more tool calls) leaves most of the allocation idle until it finishes.¹ Asynchrony (filling each training batch from the most-recently-completed rollouts across many generators) removes that barrier.
Multi-datacenter scale. Increasing the time between weight syncs makes the loop more off-policy but lets a run span datacenters, where tight synchronization is infeasible.¹

Off-policy staleness and the train-inference mismatch¶

Asynchrony buys throughput at the cost of being slightly off-policy: by the time a completion is trained on, the policy that generated it is a few updates stale. These systems are built on the premise that nearly on-policy data is good enough for stable learning.¹ Two distinct gaps appear:

Policy drift: the sampling policy pi_old lags the policy being updated pi_theta across the steps between weight syncs.
Engine mismatch: the inference engine (vLLM/SGLang) and the trainer compute token probabilities slightly differently, so even a "fresh" rollout is not exactly on-policy (GRPO notes this train-inference mismatch).

Both push the data away from the distribution the gradient assumes, and unbounded, both destabilize training.

Truncated importance sampling (TIS)¶

Importance sampling corrects for sampling from pi_old while optimizing pi_theta by reweighting each sample by the ratio rho = pi_theta / pi_old. Raw ratios have unbounded variance; a single large ratio can blow up the gradient. Truncated importance sampling caps the weight at min(rho, C) for a constant C, trading a small bias for bounded variance.¹² Unlike the bilateral clipping in PPO and CISPO (which constrains the ratio near 1 on both sides), TIS is a one-sided upper cap: the ratio may fall freely below 1 but is capped above at C to prevent extreme upweighting.¹ It is the correction that makes aggressively asynchronous and off-policy updates trainable.

Colocated vs disaggregated¶

	Colocated	Disaggregated
Layout	rollout and trainer share the same GPUs, phases alternate	separate actor and learner pools run concurrently
Sync	local, every step	over the network, every N steps
Best for	smaller models, simpler ops (TRL, verl colocate)	frontier scale, long rollouts, async (slime, verl)
Cost	memory pressure (both fit on one GPU; offload between phases)	weight-transfer bandwidth and staleness management

GRPO defaults to colocated in TRL; large-scale RL moves to a disaggregated, Ray-based actor/learner split.

# Disaggregated rollout: a dedicated vLLM server (actors) feeding the trainer (learners).
# TRL server mode keeps generation off the trainer GPUs.
CUDA_VISIBLE_DEVICES=0,1,2,3 trl vllm-serve --model Qwen/Qwen3-8B
# In GRPOConfig on the trainer GPUs: use_vllm=True, vllm_mode="server"

# verl (Ray): actor_rollout_ref groups the rollout engine, the actor (policy), and the
# reference; the trainer runs separately. Pin the verl release and verify keys on the repo.
python -m verl.trainer.main_ppo \
  actor_rollout_ref.rollout.name=vllm \
  actor_rollout_ref.rollout.mode=async \
  trainer.nnodes=4

The rollout fleet is a serving deployment¶

Once the actor pool is more than one engine, it stops being "the generator" and becomes a serving deployment with every ordinary concern of one, plus a twist that is specific to RL: rollouts are generated in groups, so a large fraction of concurrent requests share a long prefix by construction, and multi-turn rollouts return to the same growing prefix turn after turn.

Front the engines with one router, and keep admin traffic off it. Clients should hold a single URL whatever the topology behind it. Weight updates and health checks should address the engine directly rather than passing through the load balancer, because a weight update racing user traffic through a router is a race with no useful semantics.

Route on trajectory identity, not on request identity. Hashing a per-trajectory header so every turn of one rollout lands on the engine that already holds its KV converts the prefix reuse from accidental to designed (KV cache management, prompt caching). Round-robin remains the better choice when the number of concurrent trajectories is too small for hashing to spread evenly.

Balance on in-flight requests rather than scraped metrics. Queue-depth and cache-utilization scorers lag their scrape interval, and a burst of same-prefix requests is exactly what a group of rollouts is, so they concentrate the burst on whichever engine looked idle at the last scrape. An in-flight counter spreads it immediately.

The prefill/decode split follows rollout shape, not model size. Agentic rollouts grow their context every turn and are prefill-heavy; single-turn reasoning is short-prompt and decode-heavy. One published deployment guide recommends roughly 3:1 prefill-to-decode for agentic workloads (software engineering, theorem proving) and 1:2 for non-agentic ones (math, chat), and tells operators to watch the waiting-request count on both roles and add capacity to whichever queues. This is the disaggregated inference trade-off, parameterised by rollout shape.

Pool KV across nodes instead of per instance. When cache blocks are keyed by model, parallel rank, and content hash, with no instance identifier in the key, a prefix cached by one node is reusable by all of them over RDMA, so every node's DRAM contributes to one shared pool rather than to a private one. That changes the arithmetic of grouped rollouts substantially, and it requires an RDMA-capable fabric to be worth doing.

Two operational cautions from the same guide. Exporting per-token expert-routing decisions so the trainer can replay them (the MoE mismatch fix discussed under Molt and in the Composer 2 case below) is not free at the serving layer: the routing payload rides on every response and can be large enough to need a wider environment-server pool to stay parallel, and it has been reported as incompatible with CPU KV offload. And the KV transfer path used for disaggregated prefill and decode has been reported to segfault when the transfer library is installed from a prebuilt wheel against its bundled communication runtime, requiring a source build of that runtime. Neither is a reason to avoid the feature; both are reasons to prove the path under load before committing a multi-day run.

Production case: Composer 2¶

Composer 2 trained a 1.04-trillion-parameter, 32-billion-active-parameter MoE with an asynchronous single-epoch group policy-gradient system. Four decoupled services owned training, environments, inference, and evaluation. Ray futures and a central reconciler tracked policy versions, queued work, spilled state to NVMe, recovered failed tasks, and kept warm standby capacity across three GPU regions and four CPU regions.

Each trainer rank cached its previous S3 upload and published a sharded delta after every step. Rollout regions reconstructed the shared chain without a direct connection to the trainer. Compression, upload, download, and hot loading were pipelined, so rollout inference could continue until the final swap. Weight updates could occur during a rollout, which means one trajectory could span policy versions. The training path replayed MoE router choices with a plausibility filter to control numerical mismatch. This is an explicit systems choice: the algorithm and data model must tolerate versioned trajectories rather than pretending the transport preserves strict on-policy execution.

Weight sync and the interconnect¶

The recurring cost is moving updated weights from learners to actors. Colocated sync is local; disaggregated sync is a network or storage transfer that may use NVLink intra-node, IB/RoCE with GPUDirect RDMA inter-node, or shared object storage across regions (networking fabric, performance tuning). For NCCL, confirm [GDRDMA] in NCCL_DEBUG=INFO, set NCCL_IB_HCA, and keep ACS off for P2P; only force NCCL_NET_GDR_LEVEL=SYS when profiling disproves NCCL's topology choice. The longer the sync interval, the more off-policy the data and the more the correction must absorb.

Every option above assumes the transfer is small against a training step. Push the rollout fleet onto commodity GPUs behind ordinary internet links and it stops being: a 16 GB snapshot over a 100 Mbps per-worker downlink takes about 22 minutes, comparable to a whole step, and dissemination becomes a term in the fleet-sizing arithmetic rather than a pause to hide. That regime has its own capacity rule, broadcast topology, and cost model in policy dissemination for WAN rollout fleets.

Treat the transport as a configuration decision with a stated fallback rather than an implementation detail. A collective broadcast into the inference engines' memory is the fast default; a shared filesystem is the compatible one. Frameworks that offer both commonly fall back automatically in the cases where an in-memory broadcast cannot express the update, such as adapter-only training or a run with no inference server attached. The fallback is correct and much slower, so it belongs in the run's logged configuration: a run that silently took the filesystem path has a different staleness profile than the one you sized for.

Delta weight sync reduces bytes without claiming a universal density. PULSE measured about 99% unchanged BF16 elements per step in its controlled GRPO experiments and retained more than 98.5% sparsity at a tested sync interval of 32. Fireworks reported a different production point: an average 20.3 GiB delta for a 1024 GiB Composer 2 checkpoint. Both results require the exact optimizer, projection, and codec to be measured on the target workload.

Failure modes¶

Too off-policy: long sync intervals push pi_old far from pi_theta; gradients destabilize. Sync more often or rely on TIS, and watch KL.
Unbounded importance weights: skipping TIS in an async loop lets one large ratio dominate the update. Keep the cap on.
Straggler-bound generation: a synchronous batch idles on the longest rollout; move to async or pack sequences.
Actor/learner imbalance: too few generation GPUs starve the trainer; too few learners leave generators waiting. Profile and rebalance the split.
Weight-sync stall: a slow learner->actor transfer becomes the per-step bottleneck; put it on the fast fabric, not the management network.
Silent transport fallback: an adapter-only or server-less configuration drops the weight update onto the filesystem path; the run works and is slower and staler than planned. Log the resolved transport.
Round-robin over grouped rollouts: routing a group of same-prefix requests without prefix affinity re-prefills the shared prefix once per engine. Hash on a trajectory identifier instead.
Scrape-lagged load balancing: metrics-based scorers concentrate a burst of same-prefix requests on the engine that looked idle one scrape ago. Balance on in-flight counts.
Unproven KV transfer path: disaggregated prefill and decode depend on a KV transfer library whose prebuilt packaging has been reported to segfault; validate under load before a long run.

References¶

Cursor Research, Composer 2 Technical Report: https://cursor.com/resources/Composer2.pdf
PULSE, compute-visible weight sparsity and tested stale-sync intervals: https://arxiv.org/abs/2602.03839
Reinforcement Learning from Human Feedback (Nathan Lambert, Manning MEAP) — asynchronous RL systems (actors/learners, Ray + vLLM) and truncated importance sampling: https://rlhfbook.com
Asynchronous RLHF (off-policy generation/training): https://arxiv.org/abs/2410.18252
verl (Volcano Engine RL, Ray-based): https://github.com/verl-project/verl
slime (disaggregated, async RL): https://github.com/THUDM/slime
vLLM: https://docs.vllm.ai/en/latest/
Rollout-fleet inference deployment guide (single global router, trajectory-scoped consistent hashing, in-flight versus scraped scorers, shared KV pool keyed without an instance id, routed-expert export costs): https://github.com/PrimeIntellect-ai/prime-rl/blob/main/docs/inference.md
Prefill-to-decode ratios by rollout shape (3:1 agentic, 1:2 non-agentic) and the KV transfer library build requirement: https://github.com/PrimeIntellect-ai/prime-rl/blob/main/docs/advanced.md
Overlapped async semantics and the transports between trainer, orchestrator, and inference: https://github.com/PrimeIntellect-ai/prime-rl/blob/main/docs/overview.md
llm-d endpoint-picker routing: https://llm-d.ai
Mooncake distributed KV store: https://github.com/kvcache-ai/Mooncake

Reinforcement Learning from Human Feedback (Manning MEAP), 6.2.3 Asynchronous RL Systems and 6.2.4 Truncated Importance Sampling: exact on-policy execution is impractically slow, so actors (generation GPUs) and learners (training GPUs) run concurrently via Ray + vLLM on the premise that nearly on-policy data suffices; long reasoning rollouts make generation the straggler-bound bottleneck. ↩↩↩↩↩↩↩
Truncated importance sampling caps the per-sample ratio at min(rho, C) — a one-sided upper cap (versus PPO/CISPO bilateral clipping near 1) that bounds policy-gradient variance at the cost of a small bias. ↩