Markdown

Delta weight sync for RL¶

Scope: sparse and byte-delta protocols for synchronizing policy weights between disaggregated RL trainers and rollout engines. This page separates the BF16 sparsity result from transport-specific implementations, validates an exact overwrite round-trip, and records the support boundaries in vLLM, verl, slime, and the unreleased TRL prototypes.

These interfaces changed during 2026. Pin the trainer and inference-engine commits together, verify an exact reconstruction test, and retain a full-snapshot recovery path. The Python example below was executed with numpy. Framework snippets are unexecuted reference templates.

What it is¶

An RL trainer updates an FP32 master copy, then casts or projects parameters into the representation used for rollout. A small FP32 update can leave the BF16 projection unchanged. PULSE measured about 99% unchanged BF16 elements per step across four model families in a controlled GRPO experiment; the worst reported condition remained above 98%. The hidden FP32 updates are not discarded: they accumulate in the trainer and can cross a BF16 rounding boundary later.

Delta sync compares two consecutive rollout representations and sends only their differences. A sparse overwrite contains positions and absolute replacement values. A byte-delta protocol can instead encode changed checkpoint bytes with XOR or overwrite records. The receiver reconstructs the trainer's rollout view, not the trainer's FP32 master weights or optimizer state.

The density is workload-dependent. PULSE's production examples averaged 108 MB per sync for a 14 GB BF16 model, while Fireworks reported an average 20.3 GiB delta for a 1024 GiB Composer 2 checkpoint. These are separate systems and operating points, not a universal 1% rule.

Why use it¶

Less transfer on constrained links. PULSE reported more than 100x payload reduction in its two production workloads. Fireworks reported 94% less cross-region transfer over a 50-step example that included full anchors.
Exact rollout reconstruction. Absolute overwrite values avoid the cumulative arithmetic drift of additive deltas. Exactness is relative to the transmitted rollout representation.
Independent resource pools. Trainer and rollout fleets can use different regions or fabrics when the update stream fits object storage or a shared filesystem.
Shorter inference pauses. Compression, upload, download, and reconstruction can run while inference continues; only the final weight swap must block generation.

Transfer reduction is not the same as end-to-end training speedup. The optimizer, model conversion, object-store visibility, receiver apply, and policy-staleness budget can remain dominant.

When to use it (and when not)¶

Use delta sync when repeated full checkpoints are a measured bottleneck between separate trainer and rollout pools.
Prefer an in-memory sparse transport on a single low-latency fabric. Prefer checkpoint-byte deltas when fleets are separated by a filesystem, object store, or runtime-layout boundary.
Keep full sync when observed delta density, encoding overhead, or reconstruction time approaches the full-checkpoint cost.
Do not assume a BF16 result applies to FP8, INT4, or another quantizer. Measure the canonical byte density after quantization and verify the engine's required layout.
Colocated jobs may still need weight conversion or broadcast, but delta encoding only helps if that path is material in a profile.

Architecture¶

flowchart LR
  MASTER["FP32 trainer state"] --> VIEW["Rollout representation"]
  VIEW --> DIFF["Compare with pinned snapshot"]
  DIFF --> PAYLOAD["Positions plus values, or byte records"]
  PAYLOAD --> VERIFY["Version and integrity checks"]
  VERIFY --> APPLY{"Receiver path"}
  APPLY -->|"live tensors"| ENGINE["Sparse in-engine apply"]
  APPLY -->|"canonical checkpoint"| DISK["Patch local checkpoint and reload"]
  ENGINE --> ROLL["Versioned rollout policy"]
  DISK --> ROLL
  ANCHOR["Full snapshot anchor"] -.-> VERIFY

The implementations requested for this review expose different contracts:

Implementation	Status on 2026-07-20	Contract and boundary
PULSESync	paper protocol	BF16 positions and overwrite values, gap encoding, zstd, signed manifests, hashes, and a full anchor every 50 syncs in the production design
vLLM sparse NCCL	released in v0.23.0; current API checked at v0.25.1	transport/apply primitive only; current backend is `sparse_nccl`, with TP=1 and PP=1, contiguous runtime-format parameters, flat int32 positions, and same-dtype values
verl `delta_sharded`	merged after verl v0.8.0; main-only	disaggregated `separate_async`, SGLang, BF16, FSDP1/FSDP2 `Shard(0)`, NCCL, int32 absolute positions; first sync is dense
slime delta over disk	main-only after slime v0.3.0	non-colocated Megatron trainer, disk transport, canonical checkpoint bytes, patched SGLang `/pull_weights`, then the ordinary disk weight loader
TRL PR #5417	draft, unmerged	Hub-bucket safetensors prototype with two CPU BF16 snapshots and periodic anchors; it is not part of TRL v1.8.0

How to use it¶

The protocol below compares exact bit patterns, gap-encodes changed positions, hashes both indices and values, and applies absolute overwrites. It also exercises an unchanged tensor and corruption in each payload component.

# delta_roundtrip.py: executed exact sparse-overwrite model.
import hashlib
import numpy as np


def encode_delta(current, snapshot):
    positions = np.flatnonzero(current.view(np.uint32) != snapshot.view(np.uint32))
    if positions.size == 0:
        return np.empty(0, np.uint16), current[positions]
    gaps = np.empty(positions.size, np.int64)
    gaps[0] = positions[0]
    gaps[1:] = np.diff(positions) - 1
    assert gaps.max() < 2**16
    return gaps.astype(np.uint16), current[positions]


def apply_delta(snapshot, gaps, values):
    positions = (gaps.astype(np.int64) + 1).cumsum() - 1
    rebuilt = snapshot.copy()
    rebuilt[positions] = values
    return rebuilt


def checksum(gaps, values):
    payload = gaps.tobytes() + values.view(np.uint8).tobytes()
    return hashlib.sha256(payload).digest()


rng = np.random.default_rng(0)
element_count = 1_000_000
snapshot = rng.standard_normal(element_count).astype(np.float32)
current = snapshot.copy()
changed = rng.choice(element_count, int(0.02 * element_count), replace=False)
current[changed] = rng.standard_normal(changed.size).astype(np.float32)

gaps, values = encode_delta(current, snapshot)
expected_checksum = checksum(gaps, values)
rebuilt = apply_delta(snapshot, gaps, values)
assert np.array_equal(rebuilt.view(np.uint32), current.view(np.uint32))

empty_gaps, empty_values = encode_delta(snapshot, snapshot)
assert np.array_equal(apply_delta(snapshot, empty_gaps, empty_values), snapshot)

bad_gaps = gaps.copy()
bad_gaps[0] ^= np.uint16(1)
assert checksum(bad_gaps, values) != expected_checksum
bad_values = values.copy()
bad_values[0] = np.nextafter(bad_values[0], np.float32(np.inf))
assert checksum(gaps, bad_values) != expected_checksum

wire_bytes = gaps.nbytes + values.nbytes
print(
    f"reduction={snapshot.nbytes / wire_bytes:.1f}x "
    "bit_identical=True corruption_detected=True"
)

Executed output:

reduction=33.3x bit_identical=True corruption_detected=True

For slime's current disk implementation, the executable argument validator requires delta mode, disk transport, and non-colocated rollout:

# Reference template from slime main at commit ea9819f88caa5e043eb8aea992b0969ffe79aa8e.
python3 train.py \
  --update-weight-mode delta \
  --update-weight-transport disk \
  --update-weight-disk-dir /shared/fs/delta-updates \
  --update-weight-local-checkpoint-dir /local/nvme/rollout-ckpt \
  --update-weight-delta-encoding overwrite \
  --update-weight-delta-checksum xxh3-128

xor is the smaller apply-once encoding; overwrite is idempotent and safer when an object-store notification or pull can be retried. The supported checksums are xxh3-128, blake3, and adler32.

How to develop with it¶

Define the protocol before optimizing its codec:

Assign every update a monotonically increasing policy version and base version.
Hash the positions, values, and reconstruction target. A hash of values alone cannot detect a corrupted index.
Advance the sender snapshot only at the protocol's defined commit point. If publication and apply are not atomic, retain the base needed for retry.
Specify cold start and recovery. PULSE publishes full anchors every 50 syncs; the TRL draft defaults to 10; the Fireworks 50-step example uses 25. slime does not schedule anchors automatically.
Measure element density, encoded bytes, publish latency, apply latency, and rollout policy age separately.

The vLLM sparse NCCL engine does not discover changes or provide versioning, compression, checksums, anchors, or recovery. Its caller must produce flat int32 indices and matching values for a named, contiguous runtime parameter. The current stable implementation applies them with index_copy_ and rejects tensor- or pipeline-parallel world sizes above one.

verl PR #6974 builds a framework protocol around sparse SGLang updates. The merged code accepts only the indices encoding, although one bundled launcher comment also names deltas; follow the executable assertion until that upstream inconsistency is resolved. Its main-only reference shape is:

# Reference template for verl main after merge commit 903d90cc44ddbd06624a93e3aad7e15d92af5d99.
actor_rollout_ref:
  hybrid_engine: false
  rollout:
    name: sglang
    checkpoint_engine:
      backend: delta_sharded
      engine_kwargs:
        delta_sharded:
          encoding: indices
  actor:
    strategy: fsdp2

How to maintain it¶

Pin framework and engine commits as one compatibility unit. vLLM's original PR used update_kind="sparse_flat"; current stable releases expose a separate WeightTransferConfig(backend="sparse_nccl") path.
Re-run an exact reconstruction test after optimizer, dtype, quantizer, sharding, parameter-name, or inference-kernel changes.
Preserve a known-good full checkpoint and test the cold-start path. A chain that only works from an operator's long-lived cache is not recoverable.
Treat non-cryptographic checksums as accidental-corruption detectors. verl's XOR of torch.hash_tensor results is not authentication.
Audit upstream docs against code. slime's external-rollout page still recommends delta + nccl, while its current argument validator rejects that combination.

How to run it in production¶

Composer 2 is a production case for the disaggregated design. Its 1.04-trillion-parameter, 32-billion-active-parameter MoE RL system separated training, environments, inference, and evaluation services. Ray futures, a central reconciler, policy versions, NVMe spill, and warm standbys coordinated three GPU regions and four CPU regions. Each training rank cached its previous upload and published sharded deltas to shared S3; rollout clusters reconstructed the chain without direct trainer connectivity. Compression, upload, download, and hot loading were pipelined so the trainer did not block.

Composer 2 also allowed a trajectory to span policy versions because weights could change mid-rollout. The system replayed MoE router choices with a plausibility filter to preserve training and inference parity. Delta transport therefore belongs inside a versioned staleness and numerical-parity design, not as an isolated bandwidth codec.

For slime, use the shipped SGLang patch that provides /pull_weights; only the final update_weights_from_disk load is an ordinary engine operation. Gate cross-host visibility with --custom-update-weight-post-write-path on the publisher and --sglang-custom-pull-weights-pre-read-hook on the reader. Force a full version after any ambiguous or failed apply. For retry-heavy storage, prefer overwrite records because replaying an XOR delta can revert sections that were already applied.

TRL's Hub-bucket workflow remains a prototype. PR #5417 reconstructs a full CPU snapshot at the receiver before calling vLLM's full load_weights, has no payload checksum or ordering validation, and contains no added tests. Its published 405B payload and pause figures are extrapolations, not measurements. Do not configure those draft flags as though they were released TRL APIs; track PR #5417 and its unmerged successor PR #5937.

The other lever: optimizing the full refit path¶

Delta encoding reduces the bytes. A separate body of work reduces the time spent moving whatever bytes you send, and it matters because the two are complementary: if your refit is dominated by per-tensor launch overhead rather than payload size, sending 100x fewer bytes will not help you.

NeMo-RL's non-colocated refit work is the clearest published account. Refit there is the step that synchronizes inference-GPU weights with the updated training parameters, and it decomposes into three phases: gathering sharded policy parameters across training GPUs, broadcasting the gathered weights to generation workers, and loading the new weights into the inference models. Three optimizations were applied.¹

Topology-aware NCCL broadcast. Expanding the communication group to include additional policy-worker GPUs enables multiple NCCL rings, so several inter-node links carry traffic simultaneously rather than one ring serializing on a single path. The reported theoretical ceiling on their multi-node system is 400 GB/s.
Coalesced collectives and batched updates. Packing many model tensors into contiguous buffers before broadcast reduces kernel-launch overhead. The discussion notes this "particularly benefit[s] models with numerous small tensors like Mixture-of-Experts architectures", which is exactly the regime the executed block below characterizes.
Overlapping phases. Multiple CUDA streams overlap parameter gathering with weight broadcasting, because each operation on its own underutilizes available bandwidth.

Reported end-to-end refit times: Qwen3 30B-A3B from 15 s to 1.5 s (10x), Qwen3 235B-A22B from 23 s to 5 s (4.6x), and DSV3 from 70 s to 14 s (5x).

The distribution of those speedups is itself informative, and the block below makes the mechanism explicit.

# refit_cost.py — validated: why coalescing pays off on MoE and not on dense
# checkpoints, why overlap is capped at 2x, and what a 10x refit is worth.
import numpy as np

REFIT = {"Qwen3 30B-A3B": (15.0, 1.5), "Qwen3 235B-A22B": (23.0, 5.0), "DSV3": (70.0, 14.0)}
speedups = {k: before / after for k, (before, after) in REFIT.items()}
assert abs(speedups["Qwen3 30B-A3B"] - 10.0) < 1e-9
assert abs(speedups["Qwen3 235B-A22B"] - 4.6) < 1e-9
assert abs(speedups["DSV3"] - 5.0) < 1e-9
# The SMALLEST model got the LARGEST speedup: the original bottleneck was
# per-tensor overhead, not bytes on the wire.
assert min(REFIT, key=lambda k: REFIT[k][0]) == "Qwen3 30B-A3B"
assert speedups["Qwen3 30B-A3B"] > speedups["DSV3"] > speedups["Qwen3 235B-A22B"]

def transfer_time(n, sz, bw, launch):                 # one collective per tensor
    return n * (launch + sz / bw)

def coalesced_time(n, sz, bw, launch, bucket):        # pack into contiguous buffers
    return int(np.ceil(n / bucket)) * launch + n * sz / bw

BW, LAUNCH = 200e9, 8e-6                              # 200 GB/s effective, 8 us per launch
MOE_N, MOE_SZ = 40_000, 64 * 1024                     # many small expert tensors
naive  = transfer_time(MOE_N, MOE_SZ, BW, LAUNCH)
packed = coalesced_time(MOE_N, MOE_SZ, BW, LAUNCH, bucket=512)
assert naive / packed > 20
assert (MOE_N * LAUNCH) / naive > 0.95                # launch overhead IS the cost
DENSE_N, DENSE_SZ = 400, 64 * 1024 * 1024             # few large tensors
d_naive  = transfer_time(DENSE_N, DENSE_SZ, BW, LAUNCH)
d_packed = coalesced_time(DENSE_N, DENSE_SZ, BW, LAUNCH, bucket=64)
assert 1.0 < d_naive / d_packed < 1.05                # same fix, ~2% instead of ~24x
assert (naive / packed) / (d_naive / d_packed) > 20
crossover = LAUNCH * BW                               # per-tensor bytes == one launch
assert abs(crossover - 1.6e6) < 1 and MOE_SZ < crossover < DENSE_SZ

for g, b in [(1.0, 1.0), (3.0, 1.0), (0.2, 5.0)]:     # overlap gather with broadcast
    assert 1.0 < (g + b) / max(g, b) <= 2.0           # bounded by 2x, always
assert abs((1.0 + 1.0) / max(1.0, 1.0) - 2.0) < 1e-12          # best case: balanced
assert (3.0 + 1.0) / max(3.0, 1.0) < 1.34                       # skewed: little gain
assert 2.0 < speedups["Qwen3 30B-A3B"]                # so overlap alone cannot explain 10x

def ring_bw(n_rings, per_link, n_links):
    return min(n_rings, n_links) * per_link
assert ring_bw(8, 50e9, 8) == 400e9                   # the quoted ceiling
assert ring_bw(16, 50e9, 8) == 400e9                  # extra rings do not add links

def step_speedup(refit_share, refit_gain):            # Amdahl, per RL step
    return 1.0 / ((1 - refit_share) + refit_share / refit_gain)
assert abs(step_speedup(0.15, 10.0) - 1.1561) < 1e-4
assert abs(step_speedup(0.40, 10.0) - 1.5625) < 1e-4
assert abs(step_speedup(0.03, 10.0) - 1.0277) < 1e-4
assert step_speedup(0.15, 100.0) / step_speedup(0.15, 10.0) < 1.02   # past 10x, nothing
assert step_speedup(0.03, np.inf) < 1.032             # a 3% refit share caps the whole idea
print(f"MoE coalescing {naive/packed:.1f}x vs dense {d_naive/d_packed:.3f}x | "
      f"crossover {crossover/1e6:.2f} MB | step gain at 3/15/40% share: "
      f"{[round(step_speedup(s, 10.0), 3) for s in (0.03, 0.15, 0.40)]}")

Four conclusions worth carrying into your own refit path:

Profile the launch count before the payload size. In the modelled MoE case, 96% of naive broadcast time is collective-launch overhead, not bytes. Coalescing buys about 24x there and about 2% on a dense checkpoint with few large tensors. The crossover is where one tensor's bytes take as long as one launch, around 1.6 MB at the modelled bandwidth and overhead. Below that size you are launch-bound and coalescing is the fix; above it you are link-bound and delta encoding is.
Overlap is capped at 2x and only reaches it when the phases are balanced. Gather at 3 s against broadcast at 1 s yields at most 1.33x. The published 10x therefore has to come from the composition of all three optimizations, not from overlap alone.
Extra NCCL rings help only up to the number of inter-node links. The 400 GB/s figure is 8 links at 50 GB/s; a ninth ring adds nothing.
Ask what fraction of the step refit actually is. A 10x refit speedup is worth 1.16x per step at a 15% refit share, 1.56x at 40%, and only 1.03x at 3%. Even an infinitely fast refit caps out at 1.031x when refit is 3% of the step. Measure the share first; it decides whether any of this work is worth doing, and it is the same question that decides whether delta sync is worth doing.

The two levers compose. Delta encoding shrinks the payload, which helps when you are link-bound; coalescing and ring topology attack launch overhead and link utilization, which helps when you are not. Neither is a substitute for knowing which regime you are in.

Quantization and layout¶

Three questions replace a blanket precision rule:

What is compared? PULSE's evidence compares consecutive BF16 projections. slime compares canonical checkpoint bytes and is dtype-blind.
Does quantization preserve sparsity? A changed shared scale can rewrite a whole block, so the measured byte density may be much higher than the underlying weight-update density.
Can the receiver apply the format? vLLM's sparse NCCL path expects its runtime parameter names and layouts; slime patches a canonical local checkpoint and then invokes the normal loader.

Quantized checkpoint bytes can therefore be correct for slime while still compressing poorly. A sparse runtime-tensor API can be efficient while rejecting packed checkpoint layouts. Benchmark the exact quantizer, checkpoint format, tensor-parallel layout, and loader rather than inferring support from the dtype name.

The same compute-visibility idea can filter DiLoCo pseudo-gradients, but PULSELoCo uses FP32 error feedback and reports quality matching rather than bit-identical equivalence to dense DiLoCo.

Failure modes¶

Base-version mismatch. A valid delta applied to the wrong snapshot produces a different model. Reject it before mutation.
Ambiguous retry. XOR is non-idempotent. A partial success followed by replay can revert bytes; use overwrite or a transactional apply journal.
Corrupted positions. Hash indices and values, then verify the reconstructed target hash.
No recovery anchor. New or repaired rollout hosts cannot join a delta chain without its base. Publish and test full snapshots at a bounded cadence.
Unsupported topology. vLLM sparse NCCL is currently TP=1 and PP=1; verl delta_sharded requires the documented SGLang, BF16, FSDP sharding, and disaggregated mode.
Dense deltas. Quantizer-scale changes or a larger learning rate can remove the bandwidth advantage. Switch to full sync when measured encoded bytes cross the configured threshold.
Policy staleness. Faster transfer only bounds one component of rollout age. Track version age at sample creation, completion, and learner consumption.
Prototype treated as release. verl's path is newer than v0.8.0, slime's path is newer than v0.3.0, and TRL's delta PRs remain unmerged as of 2026-07-20.

References¶

Miahi and Belilovsky, "Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL" (PULSE, revised 2026-05-19): https://arxiv.org/abs/2602.03839
vLLM PR #40096, initial sparse flat NCCL weight-update implementation: https://github.com/vllm-project/vllm/pull/40096
vLLM stable NCCL weight-transfer documentation: https://docs.vllm.ai/en/stable/training/weight_transfer/nccl/
NeMo-RL, Optimizing Non-Colocated Refit for Asynchronous RL (discussion 3315, 2026-07-22): https://github.com/NVIDIA-NeMo/RL/discussions/3315
verl PR #6974, sharded delta weight synchronization: https://github.com/verl-project/verl/pull/6974
slime delta-weight-sync documentation: https://github.com/THUDM/slime/blob/main/docs/en/advanced/delta-weight-sync.md
Cursor Research, "Composer 2 Technical Report": https://cursor.com/resources/Composer2.pdf
Fireworks, "Frontier RL Is Cheaper Than You Think": https://fireworks.ai/blog/frontier-rl-is-cheaper-than-you-think
TRL PR #5417, draft delta-weight-sync prototype: https://github.com/huggingface/trl/pull/5417
Hugging Face, "Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL": https://huggingface.co/blog/delta-weight-sync

NeMo-RL discussion 3315 (Kwon, Hu, Iwazaki, Kong and Guo, 2026-07-22): topology-aware NCCL broadcast (PR #1264), coalesced collectives and batched updates (PR #1313), and CUDA-stream overlap of gather with broadcast (PR #1379). Reported refit times: Qwen3 30B-A3B 15s to 1.5s, Qwen3 235B-A22B 23s to 5s, DSV3 70s to 14s. Listed future work includes delta-weight transfers for cross-datacenter scenarios and NCCL-based P2P resharding. ↩