Markdown

Policy dissemination for WAN rollout fleets¶

Scope: running the rollout half of RL post-training on GPUs that sit behind ordinary internet links instead of a datacenter fabric, where shipping each new policy snapshot to the fleet takes minutes rather than seconds. Covers the capacity rule that decides fleet size once dissemination latency enters the arithmetic, the broadcast topology that keeps the trainer uplink from becoming the bottleneck, the staleness budget that pays for both, and what the cost saving actually is. This is the wide-area case of async and disaggregated RL systems; when both pools sit on the same fabric, size them with rollout fleet sizing instead, which assumes weight sync is not the binding constraint.

The three Python blocks below were executed with the standard library and their pasted output is verbatim. They are models of the published operating points, not measurements of a running system. The framework this page draws its numbers from is not open source: the study publishes no repository, and its authors' public org ships Parallax, Lattica, and Symphony but no implementation of the RL layer.¹² Treat the mechanisms as reproducible designs, not as software you can install.

What it is¶

RL post-training runs two workloads against each other: a learner that takes optimizer steps, and a rollout fleet that samples completions and scores them. Rollout is forward passes plus reward evaluation, so it does not need the learner's memory capacity or its interconnect, and it dominates wall-clock time. That makes rollout the obvious thing to move onto cheaper hardware.

Moving it across a WAN breaks an assumption every colocated design makes silently. When the learner publishes a new policy, the fleet has to receive it before it can generate under it. Inside a datacenter that transfer is seconds over RDMA or NVLink, small enough to ignore. Over the public internet with a 16 GB snapshot and a 100 Mbps per-worker downlink, it is roughly 22 minutes, comparable to an entire training step. Dissemination stops being an implementation detail and becomes a first-class term in every sizing decision.

Three mechanisms make that workable, and they are coupled:

Bounded staleness as a budget, not an artifact. The learner accepts trajectories generated under any policy version no older than S steps, and publishes a snapshot every kappa steps rather than every step. This is the temporal slack that lets generation, dissemination, and training overlap. The published configuration is kappa = S - 1 with kappa >= 2, so the smallest usable budget is S = 3.¹
A capacity rule with the broadcast in it. Given the staleness budget, the required aggregate rollout throughput follows in closed form, along with a hard feasibility gate that no amount of hardware clears.
A broadcast topology that does not route every byte through the trainer. Workers relay snapshot chunks to each other, so the fleet's aggregate bandwidth carries the fan-out instead of the trainer's uplink.

Why use it¶

The rollout tier can be an order of magnitude cheaper per GPU-hour. Commodity cards at roughly $0.35/hr against datacenter accelerators at $2.90 to $3.06/hr is the entire economic argument.¹ Everything else on this page exists to make that substitution survive the network.
Rollout does not need the fabric that training needs. No gradient all-reduce crosses the WAN. The only cross-domain traffic is snapshots outbound and finished trajectories inbound, both of which tolerate latency that a collective would not.
Quality holds at moderate lag. Across five math benchmarks at 4B and 8B, distributed rollout under a bounded budget matched a colocated verl baseline: mean scores of 33.64 against 34.13 at 4B and 35.75 against 35.30 at 8B, differences inside run-to-run variance.¹ The staleness budget is what costs quality, not the fact that the workers are remote.
Capacity is elastic in a way a training cluster is not. Rollout workers are independent and stateless between snapshots, so the pool can grow, shrink, and tolerate individual failures without a gang-scheduled restart.

When to use it (and when not)¶

Use it when the rollout fleet genuinely cannot sit next to the learner: capacity is only available on a different provider or continent, or the price gap between tiers is large enough to pay for the engineering. The pattern is also the right shape when rollout hardware is heterogeneous and priced differently per node, because worker selection becomes a cost decision rather than a homogeneity assumption.

Do not use it when:

kappa * T_train <= T_bcast. The overlap is infeasible and no fleet size fixes it. Raise kappa (and therefore S), or cut the payload, before provisioning anything.
The recipe cannot tolerate S >= 3. Strict or near-strict on-policy objectives do not have the slack this design spends. Check the algorithm's staleness tolerance first, as in async RL systems.
You already have the bandwidth. If trainer and rollout pools share a fabric, weight sync is not your bottleneck and this page's machinery is overhead. Size with rollout fleet sizing.
You need the run to finish sooner. This pattern trades wall clock for dollars. It does not make training faster, and in the published comparison it made each step slower.
The learner itself must be distributed across sites. That is a different problem with different failure modes; see geo-distributed training placement and DiLoCo.

Architecture¶

flowchart TB
  subgraph DC["Datacenter (stable, expensive)"]
    L["Learner: optimizer step every T_train<br/>publishes snapshot every kappa steps"]
    B["Replay buffer<br/>admits version v >= v_t - S"]
  end
  subgraph WAN["Rollout fleet (commodity, priced per node)"]
    S1["seed 1"] --> W1["worker"] --> W2["worker"]
    S2["seed 2"] --> W3["worker"] --> W4["worker"]
  end
  L -->|"snapshot chunks, uplink B0<br/>floor(B0 / Bw) parallel chains"| S1
  L --> S2
  W2 -->|"version-tagged trajectories"| B
  W4 --> B
  B --> L
  L -.->|"scheduler: activate cheapest workers<br/>by rho = cost / throughput"| WAN

Each worker holds one upstream parent and at most one downstream child, so fan-out is 1 everywhere and no node schedules multiple outbound flows. Workers relay chunks the moment they arrive and switch to the new policy version as soon as their own copy finishes installing, which means a worker starts producing fresh rollouts without waiting for the workers behind it.

How to use it: size the pool against the broadcast¶

Continuous learner utilization requires that one publication period of kappa training steps covers both the broadcast and the generation of the trajectories consumed in it. Rearranging gives the aggregate throughput the active pool must sustain, plus a feasibility gate that fires before any fleet size is computed.¹

# rollout_capacity.py -- validated. Aggregate rollout capacity needed to keep a
# centralized learner saturated when policy dissemination is NOT free.
#
# Overlap condition: one publication period of k training steps must cover both
# the broadcast and the generation of the k*R trajectories consumed in it:
#
#     k * T_train  >=  T_bcast + k*R / sum(mu_i)
#
# Rearranged, with a hard feasibility gate: if the broadcast does not fit inside
# k training steps, NO amount of rollout capacity removes the learner bubble.
import math


def min_pool_throughput(t_train: float, t_bcast: float, kappa: int, batch: int) -> float:
    """Aggregate rollouts/s the active pool must sustain. Raises if infeasible."""
    slack = kappa * t_train - t_bcast
    if slack <= 0:
        raise ValueError(
            f"infeasible: broadcast {t_bcast:.0f}s does not fit in kappa*T_train "
            f"= {kappa * t_train:.0f}s; raise kappa (and S), or cut T_bcast"
        )
    return kappa * batch / slack


def workers_needed(t_train, t_bcast, kappa, batch, mu_worker, gamma=1.1) -> int:
    """Worker count at a uniform per-worker rate, with a gamma overprovision."""
    return math.ceil(gamma * min_pool_throughput(t_train, t_bcast, kappa, batch) / mu_worker)


def bubble_ratio(t_train: float, t_bcast: float, kappa: int, batch: int, mu_pool: float) -> float:
    """Learner idle fraction: what one publication period fails to cover."""
    need = t_bcast + kappa * batch / mu_pool
    idle = max(0.0, need - kappa * t_train)
    return idle / (idle + kappa * t_train)


# Operating point from the 8B WAN run: ~1500 s/step, 128x16 = 2048 trajectories
# per step, dissemination measured at 1437 s, S = 3 so kappa = S - 1 = 2.
T_TRAIN, T_BCAST, BATCH, KAPPA = 1500.0, 1437.0, 2048, 2
mu_min = min_pool_throughput(T_TRAIN, T_BCAST, KAPPA, BATCH)
print(f"slack per period   = {KAPPA * T_TRAIN - T_BCAST:>8.0f} s of {KAPPA * T_TRAIN:.0f} s")
print(f"required pool rate = {mu_min:>8.3f} rollouts/s")

# Per-worker rate back-solved from the two fleet sizes the source reports (8
# workers at gamma=1.0, 9 at the default gamma=1.1), not measured here.
MU_WORKER = 0.33
for gamma in (1.0, 1.1):
    print(f"workers @ gamma={gamma}: {workers_needed(T_TRAIN, T_BCAST, KAPPA, BATCH, MU_WORKER, gamma)}")

# Staleness buys capacity: a larger kappa amortizes the same broadcast.
print("\nkappa  slack(s)  pool rate  workers(gamma=1.1)")
for k in (2, 3, 5, 8):
    r = min_pool_throughput(T_TRAIN, T_BCAST, k, BATCH)
    print(f"{k:>5}  {k * T_TRAIN - T_BCAST:>8.0f}  {r:>9.3f}  {workers_needed(T_TRAIN, T_BCAST, k, BATCH, MU_WORKER):>16}")

# Adversarial: the gate must fire, not silently return a huge fleet.
try:
    min_pool_throughput(t_train=1500.0, t_bcast=3100.0, kappa=2, batch=BATCH)
except ValueError as exc:
    print(f"\ninfeasible case -> {exc}")
else:
    raise AssertionError("feasibility gate did not fire")

# Adversarial: capacity past the requirement cannot drive the bubble below zero,
# and capacity below it leaves a bubble the extra workers were bought to remove.
assert bubble_ratio(T_TRAIN, T_BCAST, KAPPA, BATCH, mu_pool=mu_min) == 0.0
assert bubble_ratio(T_TRAIN, T_BCAST, KAPPA, BATCH, mu_pool=10 * mu_min) == 0.0
starved = bubble_ratio(T_TRAIN, T_BCAST, KAPPA, BATCH, mu_pool=0.7 * mu_min)
print(f"bubble at 70% of required capacity = {starved:.3f}")
assert 0.0 < starved < 1.0

# Adversarial: as T_bcast -> 0 the rule collapses to plain rate matching, R/T_train,
# independent of kappa. Broadcast latency is the only reason kappa appears at all.
for k in (2, 8):
    assert abs(min_pool_throughput(T_TRAIN, 0.0, k, BATCH) - BATCH / T_TRAIN) < 1e-9
print(f"T_bcast=0 -> {BATCH / T_TRAIN:.3f} rollouts/s for every kappa (plain rate matching)")

Executed output:

slack per period   =     1563 s of 3000 s
required pool rate =    2.621 rollouts/s
workers @ gamma=1.0: 8
workers @ gamma=1.1: 9

kappa  slack(s)  pool rate  workers(gamma=1.1)
    2      1563      2.621                 9
    3      3063      2.006                 7
    5      6063      1.689                 6
    8     10563      1.551                 6

infeasible case -> infeasible: broadcast 3100s does not fit in kappa*T_train = 3000s; raise kappa (and S), or cut T_bcast
bubble at 70% of required capacity = 0.183
T_bcast=0 -> 1.365 rollouts/s for every kappa (plain rate matching)

Three things to take from this. First, the last assertion is the whole point of the page: with T_bcast at zero the rule degenerates to R / T_train, ordinary rate matching with no dependence on kappa at all. The staleness budget only earns its keep because the broadcast is slow.

Second, the published operating point spends more than half its overlap window on the network: 1437 s of dissemination inside a 3000 s window leaves 1563 s of slack for generating 4096 trajectories. That is why the fleet is sized the way it is, and it is a fragile place to sit.

Third, the returns on staleness flatten quickly. Going from kappa = 2 to kappa = 3 removes two workers; going from 5 to 8 removes none. Past the point where the broadcast is amortized, extra staleness buys nothing and costs stability.

How to develop with it: get the trainer uplink out of the critical path¶

The naive push-to-all is fine while the trainer's uplink B0 exceeds N * Bw. Past that, each worker's share falls to B0 / N and broadcast time grows linearly with fleet size, so every worker you add to fix a capacity shortfall makes the dissemination term worse. The fix is to make the fleet carry its own fan-out: instantiate floor(B0 / Bw) parallel chains and have each worker relay chunks to exactly one downstream child at line rate.

# broadcast_topology.py -- validated. What the trainer uplink costs you when the
# rollout fleet is on a WAN, and what peer forwarding does and does not fix.
#
# Three regimes for shipping one snapshot of size Snap to N workers:
#   star-unlimited  every worker pulls at its own downlink cap Bw (a floor, not buildable)
#   star-limited    the trainer uplink B0 is split N ways once N*Bw > B0
#   chain-pipelined floor(B0/Bw) parallel store-and-forward chains, fan-out 1 per hop
import math

MB, Mb_PER_MB = 1_000_000, 8  # decimal MB; link rates are in Mbps


def snapshot_mb(params: float, bytes_per_param: int = 2) -> float:
    return params * bytes_per_param / MB


def star_unlimited(snap_mb: float, bw_mbps: float) -> float:
    """Every worker saturates its own downlink. The floor no topology beats."""
    return snap_mb * Mb_PER_MB / bw_mbps


def star_limited(snap_mb: float, b0_mbps: float, bw_mbps: float, n: int) -> float:
    """Trainer pushes to all N at once; each gets min(Bw, B0/N)."""
    return snap_mb * Mb_PER_MB / min(bw_mbps, b0_mbps / n)


def chain_pipelined(snap_mb: float, b0_mbps: float, bw_mbps: float, n: int,
                    chunk_mb: float = 64.0) -> float:
    """floor(B0/Bw) chains; each hop relays chunk-by-chunk at line rate Bw.

    Chain depth adds pipeline fill (one chunk per extra hop), not another copy of
    the snapshot, so time is near-flat in N until the chains themselves run out.
    """
    chains = max(1, math.floor(b0_mbps / bw_mbps))
    depth = math.ceil(n / chains)
    fill = (depth - 1) * chunk_mb * Mb_PER_MB / bw_mbps
    return star_unlimited(snap_mb, bw_mbps) + fill


SNAP = snapshot_mb(8.2e9)  # 8B-class policy, bf16
BW = 100.0                 # per-worker downlink cap, Mbps
print(f"snapshot {SNAP / 1000:.1f} GB, per-worker downlink {BW:.0f} Mbps")
print(f"floor (star-unlimited)  = {star_unlimited(SNAP, BW):.0f} s   <- no topology beats this")
print(f"source's measured T_bcast = 1437 s "
      f"({100 * (1437 / star_unlimited(SNAP, BW) - 1):+.0f}% over the floor)\n")

print(" N   star-limited(300M)  star-limited(800M)  chains(300M)  chains(800M)")
for n in (4, 6, 9, 20):
    print(f"{n:>2}   {star_limited(SNAP, 300, BW, n):>17.0f}   {star_limited(SNAP, 800, BW, n):>17.0f}"
          f"   {chain_pipelined(SNAP, 300, BW, n):>11.0f}   {chain_pipelined(SNAP, 800, BW, n):>11.0f}")

# Adversarial: while the uplink is not yet the binding constraint the two
# topologies must agree; peer forwarding is not free throughput, only relief.
assert star_limited(SNAP, 800, BW, 4) == star_unlimited(SNAP, BW)
assert math.isclose(chain_pipelined(SNAP, 800, BW, 4), star_unlimited(SNAP, BW))

# Adversarial: no topology goes below the per-worker downlink floor.
for n in (1, 9, 100):
    for b0 in (300, 800, 10_000):
        assert chain_pipelined(SNAP, b0, BW, n) >= star_unlimited(SNAP, BW) - 1e-9

# Star degrades linearly in N once split; chains stay within a small factor.
big_star, big_chain = star_limited(SNAP, 300, BW, 20), chain_pipelined(SNAP, 300, BW, 20)
print(f"\nat N=20, B0=300 Mbps: star {big_star:.0f} s vs chains {big_chain:.0f} s "
      f"({big_star / big_chain:.1f}x)")
assert big_star / big_chain > 5

# The floor is a payload problem, not a topology problem: only sending fewer
# bytes moves it. A 1% delta of the same snapshot is the lever.
for density in (1.0, 0.05, 0.01):
    print(f"payload {density:>5.0%} of snapshot -> floor {star_unlimited(SNAP * density, BW):>6.0f} s")
assert star_unlimited(SNAP * 0.01, BW) < 0.02 * star_unlimited(SNAP, BW)

Executed output:

snapshot 16.4 GB, per-worker downlink 100 Mbps
floor (star-unlimited)  = 1312 s   <- no topology beats this
source's measured T_bcast = 1437 s (+10% over the floor)

 N   star-limited(300M)  star-limited(800M)  chains(300M)  chains(800M)
 4                1749                1312          1317          1312
 6                2624                1312          1317          1312
 9                3936                1476          1322          1317
20                8747                3280          1343          1322

at N=20, B0=300 Mbps: star 8747 s vs chains 1343 s (6.5x)
payload  100% of snapshot -> floor   1312 s
payload    5% of snapshot -> floor     66 s
payload    1% of snapshot -> floor     13 s

The model is built independently from link rates and payload size, and it lands on the published measurements from both directions: the 1312 s floor sits 10% under the 1437 s the study reports, and the star-limited case at 9 workers on a 300 Mbps uplink predicts 3936 s against the roughly 4000 s ceiling its broadcast figure shows.¹ That agreement is also evidence for the assumption behind the calculation, which the paper never states: that the snapshot is a full bf16 copy of the policy.

The topology fixes scaling, not the floor. Peer forwarding turns a broadcast that degrades linearly in fleet size into one that is nearly flat, which is exactly what lets the pool grow to satisfy the capacity rule. It cannot go below the time it takes one worker to pull the payload over its own link. That floor moves only by sending fewer bytes, which is what delta weight sync is for: at the 1% to 5% densities measured there, the same 22-minute transfer becomes 13 to 66 seconds and the entire feasibility problem disappears. A WAN rollout design that ships full snapshots has left its largest lever unused.

There is a second way to sidestep the uplink, and it is worth knowing before building chains. Cursor's Composer 2 fanned weights out to three GPU regions through S3 rather than a peer topology, letting object storage absorb the fan-out (async RL systems). Peer forwarding is the answer when you are paying for a trainer uplink and want the fleet's aggregate bandwidth for free; object storage is the answer when egress is cheaper than the engineering. Compare the two on price before assuming a relay tree is required.

How to maintain it: budgets, scoring, and faults¶

Set the staleness budget from measured stability, not from what the capacity rule wants. The published sweep on an 8B policy under standard GRPO found reward within roughly 5% of the synchronous baseline for S <= 6, and divergence at S = 11.¹ With kappa = S - 1, that is a usable range of kappa from 2 to 5, which the capacity table above shows is exactly where the returns flatten. Treat S = 11 as outside the envelope for an unmodified objective; algorithm-level corrections such as trust-region methods for stale data extend that range, but they are a separate change with their own validation burden.³

Rank workers by unit throughput cost, not by throughput. The scheduling key is rho_i = c_i / mu_i, dollars per rollout per second, and the greedy activation is to sort ascending and take the cheapest prefix that clears the capacity target. On a homogeneous pool this collapses to counting nodes. On a mixed pool it is the difference between a correct fleet and an expensive one: the published ablation that replaced cost-aware activation with uniform random selection kept the same nine machines and the same zero learner wait, and still paid 15% more per step.¹

Keep a small overprovision instead of a fast failure detector. The reported default is gamma = 1.1, about 10% spare capacity, and it absorbs single-worker faults with no reconfiguration at all. Stripping it to gamma = 1.0 and injecting faults produced recovery in 240 to 312 s with bubble-ratio increases under 3%.¹ Broadcast-stage faults cost more than rollout-stage ones and are the asymmetry to design around: losing one relay stalls policy installation for its entire downstream segment, so a single failure in an 8-worker fleet cost 16.45% of throughput rather than the 12.5% one worker represents.

Fault	Throughput drop	Recovery	Bubble increase
Broadcast relay slows 0.5x	8.94%	312 s	1.56%
Broadcast relay drops out	16.45%	312 s	2.63%
Rollout worker slows 0.5x	6.80%	240 s	0.97%
Rollout worker drops out	12.50%	240 s	1.88%

Source's fault injection at gamma = 1.0 with 8 workers.¹ The ordering, not the absolute values, is the transferable result.

Re-measure T_train and T_bcast when the pool changes. Both are inputs to the capacity rule and both drift: T_bcast with the fleet's link quality and size, T_train with sequence length and batch composition. Measure T_bcast as the time until some target fraction of workers have installed and can generate, not until the last one has, or a single slow tail node dictates your fleet size.

How to run it in production: the cost accounting¶

This pattern is a cost play, and the arithmetic deserves more scrutiny than a headline percentage. Offloading changes two things at once, a cheaper rollout tier and a longer step, so only their product matters.

# rollout_offload_cost.py -- validated. Whether moving rollout onto cheap remote
# GPUs actually saves money, and how much slower a step you can afford first.
#
# Offloading trades two things at once: a cheaper rollout tier, and a longer
# wall-clock step (WAN dissemination and slower workers are not free). Only the
# product matters, so the break-even is a wall-clock budget, not a price ratio.
PRICE = {"A100-80GB": 3.06, "H100": 2.90, "RTX-5090": 0.35}  # USD/hr, mid-2026 public list


def hourly(fleet: dict[str, int]) -> float:
    return sum(PRICE[gpu] * n for gpu, n in fleet.items())


def cost_per_step(fleet: dict[str, int], step_seconds: float) -> float:
    return hourly(fleet) * step_seconds / 3600


def max_affordable_step(base_fleet, base_step, new_fleet) -> float:
    """Longest step the offloaded layout can take and still cost no more."""
    return base_step * hourly(base_fleet) / hourly(new_fleet)


# Case 1: offload against a colocated trainer. The baseline's 8 GPUs run both
# rollout and training; the offloaded learner needs only 4, so this bundles a
# halved learner with the rollout move. The two effects are not separable here.
BASE = ({"A100-80GB": 8}, 1508.2)
OFFLOAD = ({"A100-80GB": 4, "RTX-5090": 8}, 1631.2)
print(f"colocated : ${hourly(BASE[0]):>6.2f}/hr, {BASE[1]:.0f} s/step -> ${cost_per_step(*BASE):.2f}/step")
print(f"offloaded : ${hourly(OFFLOAD[0]):>6.2f}/hr, {OFFLOAD[1]:.0f} s/step -> ${cost_per_step(*OFFLOAD):.2f}/step")
print(f"saving {100 * (1 - cost_per_step(*OFFLOAD) / cost_per_step(*BASE)):.1f}% per step, "
      f"while each step takes {100 * (OFFLOAD[1] / BASE[1] - 1):+.1f}% longer")
print(f"break-even step length: {max_affordable_step(BASE[0], BASE[1], OFFLOAD[0]):.0f} s "
      f"(actual {OFFLOAD[1]:.0f} s)\n")

# Case 2: offload against a system that already disaggregates rollout. Same
# trainer both sides, so the entire difference is the rollout tier's price.
PRIME = {"H100": 8 + 10}
ECHO = {"H100": 8, "RTX-5090": 76}
HOURS = 1914 / hourly(PRIME)
print(f"already-disaggregated : ${hourly(PRIME):.2f}/hr x {HOURS:.4f} h = ${hourly(PRIME) * HOURS:,.0f}")
print(f"commodity rollout tier: ${hourly(ECHO):.2f}/hr x {HOURS:.4f} h = ${hourly(ECHO) * HOURS:,.0f}")
print(f"saving {100 * (1 - hourly(ECHO) / hourly(PRIME)):.1f}% -- and the two implied run "
      f"lengths are identical, so this is the price ratio, not a systems result")

# The published totals reproduce exactly from the rates at one shared run length.
assert round(hourly(PRIME) * HOURS) == 1914 and round(hourly(ECHO) * HOURS) == 1826
assert abs(1826 / hourly(ECHO) - HOURS) < 1e-9

# Adversarial: a price-ratio win is bought by worker count, so it flips. Find
# the fleet size where the cheap tier stops being cheap.
break_even = (hourly(PRIME) - hourly({"H100": 8})) / PRICE["RTX-5090"]
print(f"\nbreak-even fleet: {break_even:.0f} commodity workers (deployed: 76)")
for n in (76, 83, 84):
    delta = 100 * (1 - hourly({"H100": 8, "RTX-5090": n}) / hourly(PRIME))
    print(f"  {n:>3} workers -> {delta:+.1f}% vs the disaggregated baseline")
assert hourly({"H100": 8, "RTX-5090": 84}) > hourly(PRIME)

# Adversarial: at equal wall clock the cheap tier always wins, so a comparison
# that holds wall clock fixed cannot measure a scheduler. Give the offloaded
# side the slower step it actually runs and the 4.6% headline disappears.
for slower in (1.00, 1.05, 1.10):
    print(f"same fleets, offloaded step {slower:.0%} of baseline -> "
          f"{100 * (1 - hourly(ECHO) * slower / hourly(PRIME)):+.1f}%")
assert hourly(ECHO) * 1.10 > hourly(PRIME)

Executed output:

colocated : $ 24.48/hr, 1508 s/step -> $10.26/step
offloaded : $ 15.04/hr, 1631 s/step -> $6.81/step
saving 33.6% per step, while each step takes +8.2% longer
break-even step length: 2455 s (actual 1631 s)

already-disaggregated : $52.20/hr x 36.6667 h = $1,914
commodity rollout tier: $49.80/hr x 36.6667 h = $1,826
saving 4.6% -- and the two implied run lengths are identical, so this is the price ratio, not a systems result

break-even fleet: 83 commodity workers (deployed: 76)
   76 workers -> +4.6% vs the disaggregated baseline
   83 workers -> -0.1% vs the disaggregated baseline
   84 workers -> -0.8% vs the disaggregated baseline
same fleets, offloaded step 100% of baseline -> +4.6%
same fleets, offloaded step 105% of baseline -> -0.2%
same fleets, offloaded step 110% of baseline -> -4.9%

Read the two cases separately, because they answer different questions and the study's own appendix says they should not be conflated.¹

Against a colocated trainer the saving is real and large, but it is not purely a rollout result. A third of the cost comes off, and the reason is visible in the rates: the offloaded layout runs a 4-GPU learner where the baseline ran 8, because the baseline's GPUs were doing generation too. That is a legitimate consequence of disaggregating, but it means the published 33% to 36% bundles halving the learner with moving rollout offsite, and the study never ablates the two apart. Note also the direction of the wall clock: each step got 8.2% longer. The break-even calculation is the useful artifact here, because it tells you the real budget: at these prices the offloaded layout could afford a 2455 s step and still break even, so a 1631 s step leaves substantial headroom for a worse network than the one tested.

Against a system that already disaggregates rollout, the reported margin is a price ratio and nothing more. Both sides run the same 8-GPU H100 trainer, and the published totals of $1,914 and $1,826 divide by their fleet rates to give the identical run length of 36.6667 hours, to four decimal places. A comparison that holds wall clock fixed on both sides cannot measure a scheduler, a broadcast topology, or a provisioning rule; it measures $26.60/hr of commodity GPUs against $29.00/hr of datacenter ones. The study presents this experiment as isolating its system mechanisms from the architectural shift, and on its own numbers it does not do that.

Two consequences for anyone planning a deployment on this basis. The margin is bought with worker count, so it inverts: 83 commodity workers is break-even and 84 is a loss. And it assumes the two systems finish in the same time, which a WAN deployment against an intranet one generally will not: a step just 5% slower erases the entire 4.6%.

Cost the run, not the GPU-hour. Rental prices move, and this whole pattern rests on a price gap that is a market condition rather than a property of the hardware (cloud and cost). Re-derive the break-even at your own quotes before committing, and include egress: object-storage fan-out and per-worker downloads are billable on most providers and appear in none of the published figures.

Failure modes¶

Broadcast exceeds the publication window. kappa * T_train <= T_bcast and the learner stalls every period regardless of fleet size. Adding workers makes it worse under a star topology, since each new worker cuts everyone's share of B0. Raise kappa, cut the payload, or stop.
Fleet sized without the dissemination term. Plain rate matching (R / T_train) undersizes the pool whenever T_bcast is material, and the symptom is a persistent bubble that survives adding the workers the naive formula asked for.
Staleness raised to fix a capacity shortfall. The capacity rule makes S look like free headroom. Past the measured stability envelope it is not, and the failure is a diverging run many steps after the config change, not an error at provisioning time.
A slow relay silently stalls its downstream segment. Fan-out-1 chains make each worker a single point of failure for everything behind it, which is why broadcast faults cost more than their share of throughput. Rank relays on measured transfer speed and success rate, demote degraded ones to rollout-only, and rebuild the affected segment rather than the whole topology.
T_bcast measured at the tail. Defining dissemination as complete when the last worker installs lets one bad link dictate the fleet size for everyone. Measure at a target fraction and let the tail catch up.
Trajectory backlog aged out of the budget. Workers keep generating under whatever policy they hold, so a fleet that outruns the learner produces trajectories that are discarded on admission. Wasted spend that looks like healthy utilization on the worker side; watch admitted-versus-generated, not generated alone (rollout fleet sizing).
Full snapshots shipped when a delta would do. The single largest lever on the binding constraint, unused. See delta weight sync.
Untrusted workers assumed honest. A commodity or volunteer pool can return fabricated trajectories, and nothing in this design detects that. Rollout verification is a separate problem with its own partial answers and known gaps (P2P transport).

References¶

ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning (arXiv 2602.02192, v5 26 May 2026) — the capacity rule, peer-forwarding broadcast, cost-aware activation, and every measured figure quoted above: https://arxiv.org/abs/2602.02192
ECHO: Decoupling Inference and Training for Large-Scale RL Alignment on Heterogeneous Swarms (arXiv 2508.05387) — the predecessor system: https://arxiv.org/abs/2508.05387
Parallax: Efficient LLM Inference Service over Decentralized Environment (arXiv 2509.26182) — the rollout-serving layer: https://arxiv.org/abs/2509.26182
Parallax source: https://github.com/GradientHQ/parallax
INTELLECT-2 (arXiv 2505.07291) — fully decentralized RL with Shardcast checkpoint distribution, the CDN-style alternative to peer chains: https://arxiv.org/abs/2505.07291
prime-rl, the open implementation of that lineage: https://github.com/PrimeIntellect-ai/prime-rl
AReaL (arXiv 2505.24298) — asynchronous RL with bounded staleness inside one datacenter: https://arxiv.org/abs/2505.24298
StreamRL (arXiv 2504.15930) — disaggregated stream generation: https://arxiv.org/abs/2504.15930
Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs? (arXiv 2510.01161) — trust-region methods that extend the usable staleness range: https://arxiv.org/abs/2510.01161

ECHO-2, arXiv 2602.02192v5. Capacity rule and feasibility gate in section 3.3; kappa = S - 1 with kappa >= 2 in sections 3.3 and 4.1; per-worker rate rho_i = c_i / mu_i and gamma = 1.1 in sections 3.4 and 4.3; peer-forwarding chains N_ch = floor(B0 / Bw) in section 4.2; T_bcast = 1437 s and the cost ablation in Table 2; staleness sweep in section 5.3; benchmark parity in Table 5; fault injection in Table 4 and appendix D; the Prime-RL cost comparison and its own caution against conflating the two margins in appendix E.3. The paper reports "4B to 32B" in its abstract and conclusion; the 32B result is appendix-only, against a different baseline, at a margin roughly seven times smaller than the 4B/8B headline. ↩↩↩↩↩↩↩↩↩↩↩
Checked 2026-08-04: the paper contains no repository link, and the GradientHQ organization publishes parallax, lattica, symphony, and logits, with no ECHO or ECHO-2 repository. ↩
Zheng et al., arXiv 2510.01161, propose M2PO for training under large staleness. Layering such a method changes the objective, so the stability envelope quoted here (standard GRPO) no longer applies and has to be re-measured. ↩