Markdown

RL-VLA³: triple-level asynchronous RL for VLA models¶

Scope: how to keep GPUs busy during reinforcement-learning post-training of Vision-Language-Action (VLA) models, whose RL loop threads through a physics simulator with variable step time. RL-VLA³ is a triple-level asynchronous architecture: asynchronous training and inference on disjoint GPUs, an asynchronous interaction policy with dynamic batching, and streaming micro-batch generation. Distilled from the JD Technology embodied-infrastructure report (Guo et al., 2026) and its companion RL-VLA³ paper (Sun et al., 2026). This is the VLA-specific counterpart to the KB's general async and disaggregated RL systems page, which covers the same overlap ideas for text-LLM RL.

The numpy block labelled core math is self-contained, runnable, and validated. It implements the dynamic-batching trigger, the sync-vs-async overlap bound, and the micro-batch gradient-equivalence proof that shows streaming generation costs no accuracy. Throughput multipliers are the source teams' measurements on their stacks and baselines; the Results section reconciles the different headline numbers honestly. Re-measure on your environment.

What it is¶

A VLA RL step has three stages that classically run in series: a rollout worker drives the policy through a simulator to generate a trajectory, an inference engine runs policy forward passes to pick actions, and an actor worker consumes trajectories to compute gradients and update the policy. Because a physics simulator's step time varies (contact-rich states are slower, environments desynchronize), a synchronous pipeline that waits for every environment and every rollout before training leaves GPUs idle waiting on the slowest component.

RL-VLA³ removes the waiting at three levels:

Asynchronous training and inference. Rollout workers and actor workers live on entirely different GPUs. A rollout worker drops each finished trajectory into a communication queue and immediately starts the next one on the current policy version, without waiting for its peers. The actor does not wait for all rollouts; once the queue holds a training batch, it pulls data and updates. Rollout generation and policy update overlap in time (the report calls this "computational masking").
Asynchronous interaction policy. Instead of stepping all parallel environments in lockstep and running one big inference batch, a dynamic batching scheduler forms inference batches on two triggers: a maximum batch size Bmax (flush when enough requests accumulate) or a maximum wait Tmax (flush when the oldest request has waited too long). Under load it naturally forms large batches for throughput; under light load or environment jitter it flushes early for latency.
Streaming generation. Rather than accumulating a full global training batch before any compute, the actor splits the global batch into micro-batches and starts forward/backward on each micro-batch as soon as it fills, accumulating gradients and doing one parameter update at the end. This removes the idle gap while the batch fills, with no change to the update math.

Why use it¶

Synchronous RL wastes the most-expensive resource. Serial rollout-then-inference-then-train leaves GPUs idle across the seams; the idle fraction grows with simulator-step variance. Overlapping the stages is the single largest throughput lever in simulator-in-the-loop RL.
Measured throughput gains are large. On LIBERO with π0.5, the report's asynchronous stack reached a 59.25% throughput improvement over the co-located baseline at 32 GPUs (Table 3); the companion RL-VLA³ paper reports up to 85.2% over synchronous training scaling from 8 to 256 GPUs.
Dynamic batching gives a throughput/latency dial. Bmax and Tmax let one knob-pair move between throughput-first (large batches under load) and latency-first (early flush under jitter) without code changes.
Streaming generation is a free idle-removal. Because size-weighted gradient accumulation over micro-batches is mathematically identical to one big-batch gradient (validated below), it removes the batch-fill idle gap at zero accuracy cost.

When to use it (and when not)¶

Use it for simulator-in-the-loop VLA RL where rollout and training can occupy separate GPUs and simulator-step time varies enough to starve a synchronous pipeline.
Asynchronous train/inference and streaming generation are almost always safe wins; they overlap work or remove idle without changing the update.
The asynchronous interaction policy is environment-dependent. It helps when environments run on CPU or desynchronize (LIBERO), but it can hurt when the simulator itself batches environment computation on GPU (ManiSkill), because splitting the batch into mini-batches for genuine env/rollout parallelism undercuts the simulator's batched efficiency (see Results).
Mind policy staleness. Fully asynchronous rollouts generate trajectories under a slightly older policy version than the one being updated; this is the standard off-policy-drift tradeoff of async RL. Bound the staleness and monitor policy performance, exactly as for text-LLM async RL.
Skip it when rollout is trivially cheap relative to training (rare in embodied RL), where the overlap buys little.

Architecture¶

The three levels are nested: train/inference overlap is the macro level (across rollout epochs), while dynamic batching and streaming generation operate at the micro level (within an epoch).

flowchart LR
  subgraph ROLLOUT["Rollout GPUs"]
    ENV["Simulator envs (variable step time)"] --> INF["Inference engine"]
    INF --> DB["Dynamic batch: flush on Bmax OR Tmax"]
  end
  DB -->|"trajectory queue"| Q["Communication pipe"]
  subgraph ACTOR["Actor GPUs"]
    Q --> MB["Split global batch into micro-batches"]
    MB --> GA["Forward/backward per micro-batch, accumulate grads"]
    GA --> UPD["One parameter update"]
  end
  UPD -->|"weight sync (next policy version)"| INF

Core math (runnable): dynamic batching, overlap bound, micro-batch equivalence¶

This numpy block implements the load-bearing pieces. (A) the dynamic-batching scheduler and its invariants (no batch exceeds Bmax, no request waits past Tmax, throughput-first under bursts, latency-first when sparse); (B) the sync-vs-async overlap speedup, bounded in [1, 2] and equal to 2 only when rollout and train times match; (C) streaming generation: size-weighted micro-batch gradient accumulation equals the full-batch gradient exactly, and naive equal-weight averaging does not. Run: python3 async_rlvla.py.

# async_rlvla.py -- RL-VLA^3 triple-level asynchrony, core math. numpy only.
import numpy as np

# ---------- (A) Dynamic batching: flush on Bmax OR Tmax ----------
def dynamic_batches(arrivals, Bmax, Tmax):
    """Return (flush_time, [indices]) batches. Flush when the queue hits Bmax OR the oldest
    queued request has waited Tmax. arrivals sorted ascending."""
    batches, queue = [], []
    for i, t in enumerate(arrivals):
        while queue and t - arrivals[queue[0]] >= Tmax:          # timeout on the oldest waiter
            k = min(Bmax, len(queue))
            batches.append((arrivals[queue[0]] + Tmax, queue[:k])); queue = queue[k:]
        queue.append(i)
        while len(queue) >= Bmax:                               # size trigger: emit full batches
            batches.append((t, queue[:Bmax])); queue = queue[Bmax:]
    while queue:                                                # drain tail at its timeout deadline
        k = min(Bmax, len(queue))
        batches.append((arrivals[queue[0]] + Tmax, queue[:k])); queue = queue[k:]
    return batches

Bmax, Tmax = 4, 20.0
burst = dynamic_batches([0.0] * 12, Bmax, Tmax)                 # 12 requests at once
assert [len(b[1]) for b in burst] == [4, 4, 4]                  # throughput-first: full batches
assert all(ft == 0.0 for ft, _ in burst)                       # no waiting under load

sparse_arr = [0.0, 100.0, 200.0]                               # inter-arrival > Tmax
sparse = dynamic_batches(sparse_arr, Bmax, Tmax)
assert [len(b[1]) for b in sparse] == [1, 1, 1]                # latency-first: flush alone
assert all(abs(ft - (sparse_arr[b[0]] + Tmax)) < 1e-9 for ft, b in sparse)

rng = np.random.default_rng(0)
arr = np.sort(rng.uniform(0, 500, size=200))                   # random stream: invariants hold
bts = dynamic_batches(list(arr), Bmax, Tmax)
assert sorted(i for _, idxs in bts for i in idxs) == list(range(len(arr)))  # each served once
for ft, idxs in bts:
    assert len(idxs) <= Bmax                                   # never exceed Bmax
    assert all(ft - arr[i] <= Tmax + 1e-9 for i in idxs)       # bounded latency <= Tmax

# ---------- (B) Async train/inference: overlap masks the shorter stage ----------
def async_speedup(R, T):                                       # rollout R, train T, disjoint devices
    return (R + T) / max(R, T)                                 # sync = R+T serial; async = max(R,T)
assert abs(async_speedup(5, 5) - 2.0) < 1e-9                   # balanced -> 2x (perfect masking)
assert async_speedup(9, 1) < 1.2                               # one stage dominates -> little to gain
for R, T in [(5, 5), (7, 3), (2, 8), (6, 6)]:
    assert 1.0 <= async_speedup(R, T) <= 2.0                   # never slower, never more than 2x

# ---------- (C) Streaming generation: micro-batch accumulation == full-batch gradient ----------
# Toy linear regression grad g(w) = (2/N) X^T (Xw - y). 97 samples in 4 UNEQUAL micro-batches.
d, Nf = 6, 97
X = rng.normal(size=(Nf, d)); y = rng.normal(size=Nf); w = rng.normal(size=d)
def grad(Xb, yb, w, N):                                        # contribution to the GLOBAL-mean gradient
    return (2.0 / N) * Xb.T @ (Xb @ w - yb)
full = grad(X, y, w, Nf)
Xs, ys = [X[k::4] for k in range(4)], [y[k::4] for k in range(4)]
acc = sum(grad(Xb, yb, w, Nf) for Xb, yb in zip(Xs, ys))       # size-weighted accumulation
assert np.allclose(acc, full)                                  # exact: streaming costs no accuracy
naive = np.mean([(2.0 / len(yb)) * Xb.T @ (Xb @ w - yb) for Xb, yb in zip(Xs, ys)], axis=0)
assert not np.allclose(naive, full)                            # equal-weight averaging is wrong for unequal chunks

print("PASS:",
      f"burst={[len(b[1]) for b in burst]} sparse={[len(b[1]) for b in sparse]} |"
      f" speedup(5,5)={async_speedup(5,5):.2f} speedup(7,3)={async_speedup(7,3):.2f} |"
      f" microbatch grad exact (err {np.abs(acc-full).max():.1e})")

How it works¶

The macro level decouples the two GPU pools. Rollout workers run the current policy against the simulator and enqueue trajectories; the actor dequeues once a batch has accumulated and updates. Because neither pool blocks on the other, rollout of step n+1 overlaps the training of step n. The overlap bound in the core-math block captures the ceiling: the async per-step time is max(R, T) instead of the synchronous R + T, so the speedup is (R + T) / max(R, T), which is 2x when rollout and training take equal time and shrinks toward 1x when one stage dominates. The cost is policy staleness: enqueued trajectories were generated under a slightly older policy, the standard off-policy drift that async RL systems manage with staleness bounds and periodic weight sync.

The micro level fills the two pools. Dynamic batching keeps the inference engine fed without lockstep: the Bmax-or-Tmax trigger (validated to bound both batch size and per-request latency) forms large batches under load and flushes early under jitter. Streaming generation keeps the actor busy: splitting the global batch into micro-batches and accumulating size-weighted gradients is exactly equal to the full-batch gradient (validated), so the actor starts computing the moment the first micro-batch fills, with zero accuracy cost. The adversarial half of that check matters in practice: accumulation must be size-weighted, because averaging per-micro-batch means over unequal chunks silently reweights the update.

How to use it¶

Reference sketch of the dynamic-batching inference loop (conceptual; a real system uses an async server and a GPU inference engine):

# REFERENCE TEMPLATE (conceptual; not executed). The scheduler's math is validated in numpy above.
import time, queue

def batching_loop(requests, engine, Bmax, Tmax):
    pending, deadline = [], None
    while True:
        try:
            timeout = None if deadline is None else max(0.0, deadline - time.monotonic())
            req = requests.get(timeout=timeout)                 # block until next request or the flush deadline
            if not pending:
                deadline = time.monotonic() + Tmax              # start the Tmax clock on the first waiter
            pending.append(req)
            if len(pending) >= Bmax:                            # size trigger
                engine.infer_batch(pending); pending, deadline = [], None
        except queue.Empty:                                     # Tmax trigger: oldest request waited long enough
            if pending:
                engine.infer_batch(pending); pending, deadline = [], None

Place rollout and actor on disjoint GPUs so their compute genuinely overlaps; co-locating them reintroduces the serialization you are trying to remove.
Tune Bmax to the inference engine's efficient batch size and Tmax to the acceptable per-step latency; the core-math invariants guarantee no request waits longer than Tmax.
Use size-weighted gradient accumulation for streaming generation, never an equal-weight average of per-micro-batch means.

How to develop with it¶

Build on an existing RL framework's placement abstraction. The report compares against RLinf's co-located and disaggregated placements; the async strategy is layered on top of a disaggregated placement, so start from a framework that already separates rollout and actor roles.
Profile your simulator before enabling asynchronous interaction. Whether the async interaction policy helps depends on whether your environments are CPU-stepped/desynchronized (helps) or GPU-batched (may hurt); measure both before committing.
Add a staleness monitor early. Track the policy-version gap between rollout and update; it is the knob that trades throughput for on-policy fidelity.

How to run it in production¶

Bound and alert on policy staleness, not just throughput. A run can look fast while quietly drifting off-policy; watch task-success rate alongside samples/second.
Rate-match the rollout fleet to the actor. Too few rollout workers starve the actor; too many pile up stale trajectories. This is the same sizing problem as the KB's rollout fleet sizing runbook.
Expect sublinear scaling at large GPU counts. The report found near-linear scaling from 8 to 24 GPUs on LIBERO+π0.5, degrading from 24 to 128 and further from 128 to 256, as inter-worker communication grows. Plan capacity against the measured curve, not the ideal line.
Keep the interaction policy switchable per environment. Because it can help or hurt depending on the simulator, make asynchronous interaction a flag you can disable for GPU-batched environments.

How to maintain it¶

Keep the core-math assertions as unit tests. The Bmax/Tmax invariants and the size-weighted gradient equivalence are library-independent and catch a broken scheduler or a mis-weighted accumulation immediately.
Re-tune Bmax/Tmax after an engine or model change. The efficient batch size and acceptable wait shift with the inference engine and model size.
Re-validate the interaction policy after a simulator upgrade. A change in how the simulator batches environment computation can flip the async interaction policy from a win to a loss (the ManiSkill effect).
Re-check the staleness bound after fleet resizing. Changing the rollout-to-actor ratio changes trajectory age; re-verify policy performance holds.

Results¶

From the report's Table 3 (throughput, higher is better; "Increase%" is versus the co-located baseline). The verifiable headline is +59.25% over co-located on LIBERO+π0.5 at 32 GPUs. The report's own abstract separately states "the throughput improvement reaches 126.67%" after "further optimization through a decoupling strategy," in the same sentence as a 256-GPU validation claim, a scale Table 3 does not cover; that number does not reconstruct from any combination of Table 3's 8/16/32-GPU cells against either baseline, so treat it as a distinct, undisclosed data point rather than a recomputation of Table 3 under a different baseline. Cite the verifiable +59.25% for anything you need to defend, not the abstract's 126.67% or the companion paper's 85.2% (see below), unless you can trace the exact comparison behind them:

Configuration	LIBERO+π0.5 (8/16/32 GPU)	LIBERO+GR00T (8/16/32)	ManiSkill+π0 (8/16/32)
Co-located (baseline)	289 / 548 / 704	372 / 680 / 1126	133 / 232 / 370
Disaggregated (1:1)	163 / 308 / 457	221 / 410 / 730	61 / 151 / 257
+ Train Async	230 / 441 / 737	244 / 477 / 951	127 / 245 / 436
+ Rollout Async	370 / 687 / 1041	434 / 816 / 1620	69 / 139 / 275
+ Streamer	383 / 713 / 1121	440 / 816 / 1592	72 / 141 / 280
Increase % vs co-located	+32.5 / +30.3 / +59.25	+18.3 / +20.0 / +44.0	-4.46 / +5.33 / +17.84

Two honest readings: (1) Train Async and Streamer help nearly everywhere; the async interaction policy (Rollout Async) is the swing factor. (2) On ManiSkill it costs 4.46% at 8 GPUs because ManiSkill batches environment computation on GPU and the mini-batch partition undercuts that, but the loss shrinks and turns into a +17.84% gain by 32 GPUs as scale amortizes the overhead. This is the concrete example behind the environment-dependence caveat above.

Failure modes¶

Unbounded policy staleness. Without a staleness cap, rollouts drift far off-policy and training destabilizes; the throughput looks great right up until the policy collapses.
Async interaction on a GPU-batched simulator. Enabling the interaction policy on an environment like ManiSkill can reduce throughput at small scale; keep it switchable.
Equal-weight micro-batch averaging. Averaging per-micro-batch gradient means over unequal-size micro-batches silently reweights examples and is not the full-batch gradient; always accumulate size-weighted.
Rollout/actor rate mismatch. An under- or over-sized rollout fleet either starves the actor or floods it with stale data (rollout fleet sizing).
Assuming linear scaling. Communication overhead makes scaling sublinear past a couple dozen GPUs; sizing to the ideal line over-provisions.

References¶

Guo et al., "Thousand-GPU Large-Scale Training and Optimization Recipe for AI-Native Cloud Embodied Intelligence Infrastructure" (JD Technology, 2026): https://arxiv.org/abs/2603.11101
Sun et al., "RL-VLA³: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Training" (companion paper): https://arxiv.org/abs/2602.05765
Yu et al., RLinf (the placement/baseline framework): https://arxiv.org/abs/2509.15965
Zang et al., RLinf-VLA (unified VLA+RL framework): https://arxiv.org/abs/2510.06710
LIBERO benchmark: https://github.com/Lifelong-Robot-Learning/LIBERO
ManiSkill (GPU-parallel simulation): https://github.com/mani-skill/ManiSkill