Zero RL at trillion scale: numerics, context parallelism, and staging¶
Scope: the systems problems that appear when reinforcement learning with verifiable rewards is run directly on a pretrained base model at trillion-parameter scale and long context, and the fixes that make it stable. This page is about the infrastructure, not the algorithm: numerical precision between the training and inference engines, context-parallel communication, and why the run has to be staged rather than continuous. The RL objectives themselves are in GRPO and its variants; the predictability question is in RL scaling laws; the actor and learner topology is in async RL systems.
Source: Tang et al., "Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning" (arXiv 2607.12395, submitted 2026-07-14, revised 2026-07-16). Reported results are for
Ring-2.5-1T-Zeroon 320 H200 GPUs, with Megatron as the training engine and SGLang as the inference engine. Nothing here was reproduced: no trillion-parameter run was performed. The Python block is this page's own model of the numerical and communication mechanisms the paper describes, executed and asserted on a CPU host.
flowchart LR
BASE["Pretrained base model<br/>(no reasoning behaviour)"] --> RL1["Stage 1 RL: elicitation<br/>clipped IS + KL + token-level loss"]
RL1 --> SD["Self-distillation reset<br/>compress traces, close train/infer gap"]
SD --> RL2["Stage 2 RL: sustained<br/>sample-level loss normalization"]
RL2 --> RL3["Stage 3 RL: tier-based<br/>adaptive reasoning depth"]
RL3 --> OUT["Ring-2.5-1T-Zero"]What it is¶
"Zero RL" means running reinforcement learning with verifiable rewards straight from a pretrained base checkpoint, with no supervised reasoning data and no human annotation. It is attractive because it needs no reasoning corpus, and it has been demonstrated repeatedly at small scale. What Ring-Zero adds is the observation that at 1T parameters the algorithm is not the hard part. Two systems problems dominate, and a third forces the run to be staged.
The training engine and the inference engine disagree. Rollouts come from SGLang; gradients come from Megatron. Different kernels and different floating-point paths produce slightly divergent logits for the same tokens. In supervised training this is irrelevant. In RL it is not, because the importance ratio divides two probabilities that both came through a softmax, and exponentiation turns a small logit difference into a large ratio difference. The paper's fix has two parts:
- Targeted FP32. Keep the model body in BF16, but compute the attention softmax and the LM head in FP32. These are the two places where exponentiation happens, so they are the two places where precision buys stability. The paper reports this eliminates sudden loss spikes and closes the numerical gap between Megatron and SGLang.
- Put the training engine's own logits in the numerator. The importance ratio is formed as the current training-engine policy over the rollout policy from the inference engine, so the ratio measures the true divergence between the two engines rather than hiding it.
Long context makes ring attention the bottleneck. Standard ring attention passes KV chunks sequentially around a circle, so each device waits for a full round of communication before it can finish. The dependency depth grows with the parallelism degree. Ring-Zero replaces it per layer type, exploiting its hybrid architecture:
- MLA layers get all-to-all context parallelism. One collective reshuffles data along the head dimension, after which each device computes full attention for a subset of heads over the whole sequence. Because MLA compresses keys and values into a low-rank latent, the volume moved is far smaller than full KV tensors.
- Lightning Attention layers get a single AllGather. These layers depend on a small fixed-size KV state matrix, so broadcasting all local states at once lets every device finish immediately.
The paper states both replacements are mathematically equivalent to ring attention and produce identical gradients. That claim is what makes them adoptable: this is a latency optimization, not an approximation.
The run has to be staged. The first RL stage deliberately uses an unnormalized token-level loss so that longer correct responses accumulate stronger gradients, which is how reasoning gets elicited from a base model that rarely produces it. The side effect is runaway verbosity, and because engine numerical discrepancies accumulate along a trajectory, extreme length inflates the training-inference gap until optimization collapses. So a self-distillation phase resets the model, compressing traces before RL resumes with sample-level loss normalization. A third stage adds tier-based training for adaptive reasoning depth.
Why these choices matter¶
Each fix targets a failure that only appears at scale, which is why smaller replications do not surface them.
Gradient coverage. Zero RL from a base model has to amplify tokens whose probability is very low, because those are the reasoning tokens the base policy almost never emits. Standard PPO clipping zeroes the gradient entirely for tokens outside the trust region, which discards exactly the population the run depends on. Ring-Zero instead applies a stop-gradient to the clipped ratio while letting gradients flow through the log-probability term, so every generated token contributes. It also drops the lower clip bound entirely, keeping only an upper bound to prevent oversized updates.
Precision where it pays. Running the whole model in FP32 would be unaffordable at 1T parameters. Running it all in BF16 destabilizes the ratio. Two FP32 islands is the compromise, and the executed model below shows why those two specifically.
Communication depth, not just volume. The all-to-all and AllGather substitutions do not merely move fewer bytes; they collapse a chain of CP-1 sequential steps into one. At CP=32 that is a 31x reduction in dependency depth, which is a latency property no amount of extra bandwidth fixes.
When this applies to you (and when it does not)¶
This page is relevant when:
- You are running RL post-training with separate training and inference engines. That is nearly every modern RL stack, and the numerical-gap problem is generic to the pattern, not specific to Megatron and SGLang.
- Your rollouts are long. The gap compounds along the trajectory, so the longer the chain of thought, the more the instability bites.
- You use context parallelism. The ring-attention dependency chain is the default and it is the thing to replace.
- You are training from a base checkpoint rather than an instruct checkpoint. The clipping and loss-normalization choices are specific to eliciting behaviour that is not there yet.
It does not apply when:
- You run colocated training and inference in one engine. No engine gap exists to close, though precision still matters.
- Your sequences are short. Ring attention's dependency chain is not the bottleneck at small CP degrees, and the accumulated ratio drift is small.
- You are fine-tuning an already-reasoning model. The elicitation-specific machinery (unnormalized token-level loss, no lower clip bound) is counterproductive once the behaviour exists; it just produces verbosity.
Architecture: where precision and communication are spent¶
flowchart TB
subgraph TR["Training engine (Megatron)"]
BODY["Model body: BF16"]
SM["Attention softmax: FP32"]
HEAD["LM head: FP32"]
end
subgraph INF["Inference engine (SGLang)"]
ROLL["Rollout generation"]
end
ROLL -->|"rollout policy log-probs (denominator)"| RATIO["Importance ratio"]
BODY --> SM --> HEAD -->|"current policy log-probs (numerator)"| RATIO
RATIO -->|"clip upper bound only, stop-gradient"| LOSS["Policy gradient + KL to frozen reference"]How to validate the mechanisms¶
These are the two claims worth checking before adopting either fix. Run: python3 zerorl.py.
import numpy as np
rng = np.random.default_rng(0)
def to_bf16(x: np.ndarray) -> np.ndarray:
"""Round float32 to bfloat16 precision (8 exp bits, 7 mantissa bits)."""
u = np.asarray(x, dtype=np.float32).view(np.uint32)
# round-to-nearest-even on the low 16 bits, then truncate
rounded = (u + 0x7FFF + ((u >> 16) & 1)) & 0xFFFF0000
return rounded.view(np.float32)
def softmax(z: np.ndarray) -> np.ndarray:
z = z - z.max(axis=-1, keepdims=True)
e = np.exp(z)
return e / e.sum(axis=-1, keepdims=True)
# --- 1. Why softmax and the LM head need FP32 in an RL loop ---------------
# The importance ratio divides two probabilities. Both come through exp().
# A tiny logit error is amplified multiplicatively, not additively.
V = 32_000
logits = rng.normal(0, 4.0, size=V).astype(np.float32)
# Two engines produce slightly different logits for the same tokens.
gap = rng.normal(0, 2e-3, size=V).astype(np.float32) # kernel/precision divergence
p_fp32 = softmax(logits)
q_fp32 = softmax(logits + gap)
ratio_fp32 = p_fp32 / q_fp32
p_bf16 = softmax(to_bf16(logits).astype(np.float32))
q_bf16 = softmax(to_bf16(logits + gap).astype(np.float32))
ratio_bf16 = p_bf16 / q_bf16
err_fp32 = np.abs(ratio_fp32 - 1.0)
err_bf16 = np.abs(ratio_bf16 - 1.0)
print(f"[1] engine logit gap: std {gap.std():.2e}, max {np.abs(gap).max():.2e}")
print(f"[1] importance ratio deviation from 1.0")
print(f"[1] softmax in FP32 : mean {err_fp32.mean():.3e} p99 {np.percentile(err_fp32, 99):.3e} "
f"max {err_fp32.max():.3e}")
print(f"[1] softmax in BF16 : mean {err_bf16.mean():.3e} p99 {np.percentile(err_bf16, 99):.3e} "
f"max {err_bf16.max():.3e}")
print(f"[1] BF16 amplifies the max ratio error by {err_bf16.max() / err_fp32.max():.1f}x")
assert err_bf16.max() > err_fp32.max(), "BF16 softmax must widen the train/infer ratio gap"
# The policy gradient multiplies the advantage by this ratio, so a ratio that
# should be ~1.0 arriving as 1.03 is a 3% mis-weighted gradient on that token.
# --- 2. The gap compounds with sequence length ---------------------------
# A trajectory's weight is the product of per-token ratios. Independent
# per-token error of size e over L tokens grows as sqrt(L) in log space.
print()
for L in (1_000, 8_000, 64_000):
log_dev_fp32 = np.abs(np.log(ratio_fp32[:L])).mean() * np.sqrt(L)
log_dev_bf16 = np.abs(np.log(ratio_bf16[:L])).mean() * np.sqrt(L)
print(f"[2] L={L:>6,} tokens: accumulated |log ratio| drift "
f"FP32 {log_dev_fp32:8.3f} BF16 {log_dev_bf16:8.3f}")
assert np.abs(np.log(ratio_bf16[:64_000])).mean() > np.abs(np.log(ratio_fp32[:64_000])).mean()
# This is why the paper reports the gap inflating as CoT length grows, and why
# it inserts a self-distillation phase to shorten traces before continuing RL.
# --- 3. Clipped importance sampling: who still gets a gradient ------------
# PPO-clip zeroes the gradient for tokens outside the trust region. The
# stop-gradient formulation keeps a gradient on every token.
print()
ratios = np.exp(rng.normal(0, 0.35, size=200_000))
EPS_HIGH = 1.2
def ppo_clip_active(r: np.ndarray, lo: float, hi: float) -> float:
"""Fraction of tokens whose gradient survives standard PPO clipping."""
return float(np.mean((r >= lo) & (r <= hi)))
def sg_active(r: np.ndarray) -> float:
"""Stop-gradient form: the ratio is a scalar weight, log-prob carries the grad."""
return 1.0
print(f"[3] PPO-clip [0.8, 1.2] keeps a gradient on {100 * ppo_clip_active(ratios, 0.8, 1.2):.1f}% of tokens")
print(f"[3] upper-only clip (no lower bound) keeps {100 * ppo_clip_active(ratios, 0.0, EPS_HIGH):.1f}% inside range")
print(f"[3] stop-gradient formulation keeps a gradient on {100 * sg_active(ratios):.1f}% of tokens")
assert ppo_clip_active(ratios, 0.8, 1.2) < ppo_clip_active(ratios, 0.0, EPS_HIGH) < sg_active(ratios)
# Removing the lower bound matters most at the start of zero RL, where the
# reasoning tokens the run needs to amplify are exactly the low-ratio ones.
low_ratio_mass = float(np.mean(ratios < 0.8))
print(f"[3] {100 * low_ratio_mass:.1f}% of tokens sit below a 0.8 lower bound and would be "
f"discarded; these are the low-probability tokens zero RL must amplify")
assert low_ratio_mass > 0.15
# --- 4. Context parallelism: ring vs all-to-all vs allgather --------------
# Ring attention passes KV chunks around a circle: CP-1 sequential steps on
# the critical path. A single collective has one step regardless of CP.
print()
SEQ, HEADS, D_HEAD, BYTES = 64_000, 128, 128, 2
def ring_steps(cp: int) -> int:
return cp - 1
def collective_steps(cp: int) -> int:
return 1
def full_kv_bytes(seq: int) -> float:
return 2 * seq * HEADS * D_HEAD * BYTES / 2**30 # K and V, GiB
def mla_latent_bytes(seq: int, latent: int = 512) -> float:
return seq * latent * BYTES / 2**30 # one compressed latent
for cp in (2, 8, 32):
print(f"[4] CP={cp:2d}: ring attention {ring_steps(cp):2d} sequential comm steps, "
f"single collective {collective_steps(cp)} step "
f"({ring_steps(cp) / collective_steps(cp):.0f}x the dependency depth)")
assert ring_steps(32) == 31 and collective_steps(32) == 1
full = full_kv_bytes(SEQ)
latent = mla_latent_bytes(SEQ)
print(f"[4] payload per rank at {SEQ:,} tokens: full K+V {full:6.2f} GiB, "
f"MLA latent {latent:6.3f} GiB ({full / latent:.0f}x reduction)")
assert latent < full / 10, "MLA's low-rank latent is what makes all-to-all CP affordable"
# Lightning-attention layers carry a fixed-size state, so AllGather cost is
# independent of sequence length. That is the whole reason it can replace ring.
STATE_BYTES = HEADS * D_HEAD * D_HEAD * BYTES / 2**30
payloads = []
for seq in (8_000, 64_000, 1_000_000):
payloads.append(full_kv_bytes(seq))
print(f"[4] seq={seq:>9,}: full-KV ring payload {full_kv_bytes(seq):8.2f} GiB, "
f"linear-attention state {STATE_BYTES:.3f} GiB (constant)")
# The KV payload grows linearly in sequence length; the linear-attention state
# does not depend on it at all. That gap is what makes AllGather viable.
assert np.allclose(np.diff(payloads) > 0, True), "full-KV payload must grow with sequence"
assert np.isclose(payloads[2] / payloads[0], 1_000_000 / 8_000), "growth is exactly linear"
assert full_kv_bytes(1_000_000) > 1_000 * STATE_BYTES
print("\nAll assertions passed.")
Executed output:
[1] engine logit gap: std 2.01e-03, max 9.46e-03
[1] importance ratio deviation from 1.0
[1] softmax in FP32 : mean 1.599e-03 p99 5.149e-03 max 9.420e-03
[1] softmax in BF16 : mean 1.765e-03 p99 3.058e-02 max 6.470e-02
[1] BF16 amplifies the max ratio error by 6.9x
[2] L= 1,000 tokens: accumulated |log ratio| drift FP32 0.052 BF16 0.057
[2] L= 8,000 tokens: accumulated |log ratio| drift FP32 0.146 BF16 0.161
[2] L=64,000 tokens: accumulated |log ratio| drift FP32 0.405 BF16 0.447
[3] PPO-clip [0.8, 1.2] keeps a gradient on 43.6% of tokens
[3] upper-only clip (no lower bound) keeps 69.8% inside range
[3] stop-gradient formulation keeps a gradient on 100.0% of tokens
[3] 26.2% of tokens sit below a 0.8 lower bound and would be discarded; these are the low-probability tokens zero RL must amplify
[4] CP= 2: ring attention 1 sequential comm steps, single collective 1 step (1x the dependency depth)
[4] CP= 8: ring attention 7 sequential comm steps, single collective 1 step (7x the dependency depth)
[4] CP=32: ring attention 31 sequential comm steps, single collective 1 step (31x the dependency depth)
[4] payload per rank at 64,000 tokens: full K+V 3.91 GiB, MLA latent 0.061 GiB (64x reduction)
[4] seq= 8,000: full-KV ring payload 0.49 GiB, linear-attention state 0.004 GiB (constant)
[4] seq= 64,000: full-KV ring payload 3.91 GiB, linear-attention state 0.004 GiB (constant)
[4] seq=1,000,000: full-KV ring payload 61.04 GiB, linear-attention state 0.004 GiB (constant)
All assertions passed.
Read section 1 carefully, because the honest result is more specific than "BF16 is bad." The mean ratio error barely moves, from 1.60e-03 to 1.77e-03. The tail is what explodes: p99 rises about 6x and the maximum about 6.9x. RL training does not fail on average behaviour; it fails on the outlier token whose ratio arrives at 1.065 instead of 1.009 and mis-weights a gradient by 6%. That is the loss spike the paper describes eliminating, and it is why a mean-error argument would have concluded, wrongly, that BF16 is fine.
Section 2 is deliberately modest. Accumulated drift grows with the square root of trajectory length, and BF16 is roughly 10% worse than FP32 at every length. The point is not that BF16 is catastrophic per token; it is that the effect compounds along the trajectory, so verbosity and instability are the same problem. That is the mechanism connecting the unnormalized token-level loss in stage 1 to the need for a self-distillation reset before stage 2.
Section 3 quantifies the clipping argument. Under the modelled ratio distribution, standard symmetric PPO clipping keeps a gradient on 43.6% of tokens, and 26.2% of tokens fall below the lower bound alone. In zero RL those low-ratio tokens are precisely the low-probability reasoning tokens the run exists to amplify, so discarding them is not a neutral regularization choice.
How to run it¶
- Instrument the training-inference gap as a first-class metric. Log the distribution of the importance ratio, not its mean. Track p99 and max. A widening tail is the leading indicator of the collapse this work is designed to prevent.
- Apply FP32 surgically. Attention softmax and LM head. Adding more FP32 costs memory and throughput without addressing the mechanism; adding less leaves the tail open.
- Form the ratio from the training engine's own logits. If the numerator comes from cached inference logits, the ratio stops measuring engine divergence and the metric above becomes blind to the failure it exists to catch.
- Choose the CP strategy per layer type. A hybrid model does not want one CP strategy. Compressed-latent attention wants all-to-all; fixed-state linear attention wants AllGather. Verify gradient equivalence against a ring-attention reference before trusting either.
- Stage the run and expect a reset. Plan for a self-distillation phase between RL stages rather than treating it as a recovery action. The paper's own configuration for that phase is 3 epochs at 64k sequence length with learning rate 7e-5, against RL stages at a constant 2e-6 with weight decay 0, batch size 512 with minibatch 32 and later 256 with minibatch 32.
- Do not tune the learning rate first. The paper evaluates 1e-6, 2e-6, and 3e-6 and reports minimal impact across that range, concluding training is robust to learning-rate variation at this scale. Spend the effort on the numerics and the staging instead.
How to maintain it¶
- Re-check the gap after every engine upgrade. A new kernel in either engine changes the logit divergence, and the fix is calibrated to the divergence.
- Re-validate CP equivalence after attention changes. The all-to-all and AllGather substitutions are equivalent for the specific layer types they target. A new attention variant needs its own proof.
- Watch response length as a stability signal, not just a cost signal. Length inertia is reported as a real phenomenon in zero RL, and growing length is what inflates the accumulated gap.
Failure modes¶
- Loss spikes with no algorithmic cause. Usually the ratio tail, not the objective. Check p99 of the importance ratio before touching hyperparameters.
- Silent gradient starvation from symmetric clipping. The run trains, the loss looks fine, and the reasoning behaviour never emerges, because the tokens that would have produced it were clipped away.
- Ring-attention latency mistaken for bandwidth shortage. Adding interconnect bandwidth does not shorten a CP-1 step dependency chain.
- Runaway verbosity read as improving reasoning. The unnormalized token-level loss rewards length directly. Longer traces in stage 1 are the objective working as designed, not evidence of better thinking, and they are what destabilizes stage 2.
- Numerator from the wrong engine. Makes the training-inference gap invisible in exactly the metric that should expose it.
- Assuming small-scale results transfer. The paper's central claim is that the training dynamics at 1T differ from small models. Treat sub-100B replications as evidence about the algorithm, not about the systems behaviour.
Open questions and validation¶
- Nothing here was reproduced. No trillion-parameter or long-context run was executed; the Python model tests the mechanisms, not the paper's results.
- The reported emergent behaviours (self-verification, parallel reasoning, structured formatting, and what the authors call anthropomorphism and context anxiety) are qualitative observations from one training run, not measured phenomena with error bars.
- The claim that the all-to-all and AllGather CP strategies produce gradients identical to ring attention is stated by the authors and not independently verified here.
- The paper proposes its own three-dimensional chain-of-thought quality framework (comprehensibility, reproducibility, efficiency) and reports its own model favourably on it. Self-proposed metrics evaluated by their proposers warrant the usual caution.
- Whether the four-stage pipeline is necessary or merely sufficient is untested; no ablation removing the self-distillation reset at full scale is reported.
References¶
- Tang et al., "Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning", arXiv 2607.12395: https://arxiv.org/abs/2607.12395
- Liu et al., "Ring Attention with Blockwise Transformers for Near-Infinite Context", arXiv 2310.01889: https://arxiv.org/abs/2310.01889
- Megatron-LM, the training engine used: https://github.com/NVIDIA/Megatron-LM
- SGLang, the inference engine used: https://github.com/sgl-project/sglang
Related: GRPO · GRPO variants · RL scaling laws · Async RL systems · RLVR · Tensor and mixed precision · Post-training system map