Markdown

Rollout reuse under policy lag¶

Scope: the decision of how many learner updates to take from a single rollout batch in LLM reinforcement learning, the policy lag that reuse creates inside the batch, and the importance-weighting choices that determine how much lag an objective survives. This page covers the weighting ladder from token-local ratios to response-level statistics, Prefix-Normalized Policy Optimization (PNPO, arXiv 2608.01418) as the prefix-normalized point on that ladder, the numerical and normalization traps in implementing it, and how reuse changes the split between generation and training capacity. It is the reuse-side companion to async and disaggregated RL systems, which covers lag created by concurrent generation instead, and it feeds the capacity arithmetic in rollout fleet sizing.

Evidence status, verified 2026-08-05. Method and result claims are from arXiv 2608.01418v1, submitted 2026-08-02 by authors at Tencent, Harbin Institute of Technology, Jinan University, and the University of Science and Technology of China. The paper publishes no code and names no repository, so nothing here was checked against a released implementation. The training runs were not reproduced: they need 32 NVIDIA H20 GPUs, DeepSeek-R1-Distill-Qwen-1.5B, and DAPO-Math-17k. The three NumPy blocks below were executed and their assertions pass; each states which inputs are the paper's and which are modelling choices. The verl and TRL snippets are unexecuted reference templates, with key names checked against the current volcengine/verl actor.yaml and policy-loss registry and the TRL GRPOConfig documentation on 2026-08-05.

flowchart TB
  GEN["Rollout generation<br/>256 prompts x 8 responses"] --> BATCH["Fixed batch<br/>behaviour logprobs and advantages frozen"]
  BATCH --> UPD["K policy-update epochs<br/>4 optimizer updates each"]
  UPD --> LAG["Learner departs from behaviour policy<br/>lag grows with every update"]
  LAG --> W{"Which weight corrects it?"}
  W --> TOK["Token-local ratio<br/>GRPO, DAPO"]
  W --> EX["Exact cumulative prefix ratio<br/>correct, unbounded scale"]
  W --> PN["Prefix geometric mean<br/>PNPO"]
  W --> SEQ["Response mean, broadcast<br/>GSPO"]
  TOK --> NEXT["Next rollout batch"]
  EX --> NEXT
  PN --> NEXT
  SEQ --> NEXT
  NEXT --> GEN

What it is¶

Rollout reuse is the decision to run more than one learner update over a rollout batch before generating the next one. It is one integer in a config: ppo_epochs in verl, num_iterations in TRL. Setting it above 1 amortizes autoregressive generation over more optimizer steps, and generation is usually the expensive half of an RL step.

The cost is policy lag inside the batch. The sampled responses, the behaviour log probabilities in the ratio denominator, and the group-relative advantages are all fixed when the batch is collected. Every subsequent update moves the learner further from the policy that produced that data. The mismatch already exists within a single pass, because later minibatches are processed after earlier updates have landed, and it compounds when the same trajectories are revisited.¹

What corrects the mismatch is an importance weight, and the choice of weight is a choice of granularity. At token position t the state is the whole prefix, so an exact correction has to account for both the current action and the probability of having reached that prefix. Because the autoregressive transition deterministically appends a token, that probability factorizes and the exact state-action change-of-measure weight is the cumulative product of per-token ratios along the prefix. This gives four positions on a ladder:²

Weight at position `t`	Support	Used by
`rho_t`	current token only	GRPO, DAPO, PPO
`C_t = prod(rho_1..rho_t)`	exact causal prefix	CTPO, exact per-decision importance sampling
`C_t ^ (1/t)`	causal prefix, log scale compressed	PNPO
`(prod(rho_1..rho_L)) ^ (1/L)`	whole response, broadcast to every position	GSPO

The token-local ratio corrects the action distribution at a fixed prefix but omits the prefix-visitation factor entirely. The exact cumulative ratio restores it, at the cost of a log weight that is a random walk in prefix length. GSPO controls scale by computing one length-normalized statistic per response and sharing it across all positions, which makes an early-position weight depend on tokens sampled later. PNPO takes the geometric mean along the prefix: the same information as the exact ratio, divided by the number of decisions in it.²

The consequences of that division are worth stating plainly, because they are what the method is. At t = 1 the prefix mean is exactly the token-local ratio. At t = L it is exactly GSPO's sequence ratio. In between, it depends on position and on the observed prefix only, never on the future suffix. PNPO is an interpolation between two objectives the KB already covers, indexed by position, not a new family.³

It is also explicitly biased. For t > 1 the prefix geometric mean is not a target-to-behaviour density ratio, so it does not preserve the change of measure that makes the exact form correct. The paper is direct about this: the normalization trades exactness for scale control, and the complete objective, which adds a position-dependent acceptance gate, response-level aggregation, and a group-relative advantage standing in for the exact target-policy advantage, is a deliberately biased proximal surrogate.⁴

Why use it¶

The reason to reuse rollouts is capacity, and the paper's own timings let you price it. It reports 283 seconds per step at one epoch and 510 seconds at four, on the same 32 H20 GPUs with the same batch. Two measurements and one linear model recover the split between generation and optimizer work, and everything else follows from it.

# reuse_econ.py -- what reusing a rollout batch buys, and what the paper's own
# timings imply. Inputs are the PNPO configuration: 2,400 optimizer updates,
# 256 prompts / 64-group minibatches (4 updates per epoch), 2,048 responses per
# batch, and measured step times of 283 s (1 epoch) and 510 s (4 epochs) on 32 H20s.
import numpy as np

UPDATES_PER_EPOCH = 4          # 256 prompts / 64 prompt groups per minibatch
BUDGET = 2400                  # optimizer updates held fixed across regimes
RESP_PER_BATCH = 2048          # 256 prompts x 8 responses


def split_step_cost(t_one: float, t_four: float) -> tuple[float, float]:
    """Back-solve generation cost G and per-epoch optimizer cost T from two
    measured step times, using step(K) = G + K*T."""
    train = (t_four - t_one) / 3.0
    return t_one - train, train


def regime(epochs: int, gen: float, train: float) -> dict:
    steps = BUDGET / (epochs * UPDATES_PER_EPOCH)
    step_s = gen + epochs * train
    return {
        "epochs": epochs,
        "steps": steps,
        "step_s": step_s,
        "hours": steps * step_s / 3600.0,
        "responses": steps * RESP_PER_BATCH,
        "gen_share": gen / step_s,
    }


GEN, TRAIN = split_step_cost(283.0, 510.0)
one, four = regime(1, GEN, TRAIN), regime(4, GEN, TRAIN)

# (1) The back-solved split reproduces both published step times exactly.
assert np.isclose(GEN + 1 * TRAIN, 283.0) and np.isclose(GEN + 4 * TRAIN, 510.0)
# (2) It also reproduces the step counts the paper reports for each regime.
assert one["steps"] == 600 and four["steps"] == 150
# (3) Reuse cuts generated responses by exactly the epoch count, not approximately.
assert one["responses"] / four["responses"] == 4.0
# (4) Rollout generation dominates a single-epoch step and stops dominating at four.
assert one["gen_share"] > 0.70 and four["gen_share"] < 0.45
# (5) The whole four-epoch run is shorter than the 22.4 h first-passage time the
#     paper attributes to four-epoch GSPO, so that crossing is outside the run.
assert four["hours"] < 22.4 < one["hours"]
assert 22.4 * 3600.0 / four["step_s"] > four["steps"]

# (6) Speedup is bounded: dropping generation entirely still leaves the optimizer.
speedup = lambda k: k * (GEN + TRAIN) / (GEN + k * TRAIN)
ceiling = (GEN + TRAIN) / TRAIN
ks = np.array([1, 2, 4, 8, 16, 64])
assert np.all(np.diff(speedup(ks)) > 0) and np.all(speedup(ks) < ceiling)

print(f"generation {GEN:.1f} s/batch, optimizer {TRAIN:.1f} s/epoch")
print(f"1 epoch : {one['steps']:.0f} steps x {one['step_s']:.0f} s = {one['hours']:.2f} h,"
      f" {one['responses']:,.0f} responses, generation {one['gen_share']:.1%} of step")
print(f"4 epochs: {four['steps']:.0f} steps x {four['step_s']:.0f} s = {four['hours']:.2f} h,"
      f" {four['responses']:,.0f} responses, generation {four['gen_share']:.1%} of step")
print(f"matched-budget speedup {one['hours'] / four['hours']:.2f}x, ceiling {ceiling:.2f}x")
print("speedup by epochs:", {int(k): round(float(speedup(k)), 2) for k in ks})
print(f"GSPO-4 first passage 22.4 h implies step {22.4 * 3600 / four['step_s']:.0f}"
      f" of a {four['steps']:.0f}-step run")

# (7) The reuse tax is method-dependent: peak macro Avg@32, one epoch vs four.
peak = {"GRPO": (48.25, 47.05), "GSPO": (49.42, 47.24), "PNPO": (50.05, 50.24)}
tax = {m: round(v[1] - v[0], 2) for m, v in peak.items()}
assert tax["GSPO"] < -2.0 and tax["GRPO"] < -1.0 and tax["PNPO"] > 0.0
print("peak change from 1 to 4 epochs (points):", tax)

# (8) On a 30-problem benchmark, the four-epoch separations are about one problem.
gaps = {"AIME24": 38.85 - 34.90, "AIME25": 30.94 - 27.92}
problems = {k: round(v / 100 * 30, 2) for k, v in gaps.items()}
assert all(p < 1.3 for p in problems.values())
print("four-epoch PNPO minus GSPO, in problems out of 30:", problems)

Executed output:

generation 207.3 s/batch, optimizer 75.7 s/epoch
1 epoch : 600 steps x 283 s = 47.17 h, 1,228,800 responses, generation 73.3% of step
4 epochs: 150 steps x 510 s = 21.25 h, 307,200 responses, generation 40.7% of step
matched-budget speedup 2.22x, ceiling 3.74x
speedup by epochs: {1: 1.0, 2: 1.58, 4: 2.22, 8: 2.79, 16: 3.19, 64: 3.59}
GSPO-4 first passage 22.4 h implies step 158 of a 150-step run
peak change from 1 to 4 epochs (points): {'GRPO': -1.2, 'GSPO': -2.18, 'PNPO': 0.19}
four-epoch PNPO minus GSPO, in problems out of 30: {'AIME24': 1.19, 'AIME25': 0.91}

Four numbers from that run carry the argument. Generation is 73.3% of a one-epoch step in this configuration, which is why reuse is worth considering at all. At a matched budget of 2,400 optimizer updates, four epochs finish in 21.25 hours against 47.17, a 2.22x saving, using exactly one quarter as many generated responses. And the saving is capped: even infinite reuse only reaches 3.74x here, because the optimizer passes remain. Most of the available gain arrives by four epochs, and the step from 16 to 64 buys 0.4x.

The paper's own accuracy comparison is what makes reuse tolerable rather than merely cheap: at four epochs PNPO's peak macro Avg@32 is 50.24 against 50.05 at one epoch, while GSPO gives back 2.18 points and GRPO 1.20.⁶ The honest reading is not that PNPO is more accurate. It is that reuse is roughly free for PNPO and expensive for the other two, and the 3.00-point headline gap at four epochs is mostly GSPO degrading rather than PNPO improving.

When to use it (and when not)¶

Measure the split before choosing an epoch count. The whole case for reuse rests on the generation share. Run one step at one epoch and one at two, subtract, and you have G and T for your own model, sequence length, and fleet. If generation is under half your step, reuse is a small lever.
Reuse when responses are long and the policy is small relative to the sampler load. A 15,360-token response limit is what makes generation dominate here. Short-response RL, tool-calling loops with short completions, and very large policies shift the balance toward the optimizer.
Do not push epochs past the point where the reuse tax exceeds the wall-clock saving. The tax is measurable: run the same update budget at one and at K epochs and compare final held-out score, not peak. If the gap is larger than the speedup is worth, stop.
Do not assume an objective transfers. The evidence here is one 1.5B model, three mathematics benchmarks, one seed, and one non-agentic task shape. Whether prefix normalization helps a mixture-of-experts policy, a multi-turn agentic rollout, or a 100B-class model is untested.
Reuse and asynchrony are different lag sources and stack. This page's lag comes from repeated updates on a fresh batch. Asynchronous generation adds version staleness on top, bounded by the admission rule in the RL orchestrator control loop. Measure them separately before combining them, since the paper explicitly scopes replay and asynchronous staleness out.⁸
Prefer the cheaper levers first when generation dominates. Prompt deduplication and shared-prefix attention in rollout redundancy cut generation cost without changing the objective or introducing any lag at all.

Architecture¶

Reuse changes where the time goes, and that changes how a disaggregated cluster should be proportioned. At one epoch this configuration spends 207.3 seconds generating and 75.7 seconds training per step, a ratio near 2.7 to 1. At four epochs the same generation feeds 302.7 seconds of optimizer work, a ratio near 0.7 to 1. In a rate-matched split where a rollout fleet feeds a trainer pool, that is a fourfold reduction in the rollout capacity each trainer needs, which is the same arithmetic the rollout fleet sizing runbook applies to matched instance counts.

The weight itself attaches at one point in the GRPO loop: the per-token coefficient multiplying the score function. It does not change the reward source, the group-relative advantage, or the rollout mechanism, so it composes with reward design and the fixes in GRPO variants unchanged. What it does change is the aggregation and the clipping, since PNPO uses response-level averaging and a hard acceptance gate rather than a PPO-style clip.

How to use it¶

The gate is the part of PNPO that looks arbitrary and is not. The acceptance interval at position t is scaled by sqrt(L/t), so it is widest at the first token and reduces to the base interval at the last one.⁵ The reason is that the prefix mean of t roughly independent log-ratios has a spread proportional to 1/sqrt(t), so a fixed-width interval would reject almost everything early and almost nothing late. The sqrt(L/t) factor cancels exactly that, holding the gate at a constant width in standard deviations.

# prefix_weights.py -- the four weighting granularities and why PNPO's acceptance
# gate carries a sqrt(L/t) factor. Log-ratios are modelled as iid noise, which is a
# first-order stand-in for a real drifted policy, not a measurement.
import numpy as np

L, SIGMA, N = 15360, 0.05, 400          # max response length, per-token log-ratio scale
EPS_PNPO = (7e-4, 9.5e-4)               # PNPO base acceptance tolerances
EPS_GSPO = (3e-4, 4e-4)                 # GSPO sequence-ratio clip tolerances
EPS_GRPO = (0.2, 0.28)                  # GRPO/DAPO token-ratio clip tolerances
t = np.arange(1, L + 1)

rng = np.random.default_rng(0)
log_rho = rng.normal(0.0, SIGMA, (N, L))
cum = np.cumsum(log_rho, axis=1)

log_local = log_rho                     # (a) token-local ratio
log_exact = cum                         # (b) exact cumulative prefix ratio
log_pnpo = cum / t                      # (d) PNPO prefix geometric mean
log_gspo = np.repeat(cum[:, -1:] / L, L, axis=1)   # (c) response mean, broadcast

# (1) PNPO reduces to the token-local ratio at the first position and to GSPO's
#     sequence statistic at the last one; it is an interpolation, not a new family.
assert np.allclose(log_pnpo[:, 0], log_local[:, 0])
assert np.allclose(log_pnpo[:, -1], log_gspo[:, -1])

# (2) The exact ratio is a random walk (spread grows as sqrt(t)); the prefix mean
#     is its running average (spread shrinks as 1/sqrt(t)). Same information,
#     opposite scale behaviour.
for pos in (100, 1000, L):
    assert np.isclose(log_exact[:, pos - 1].std(), SIGMA * np.sqrt(pos), rtol=0.12)
    assert np.isclose(log_pnpo[:, pos - 1].std(), SIGMA / np.sqrt(pos), rtol=0.12)
assert log_exact[:, -1].std() / log_pnpo[:, -1].std() > 1e4

# (3) Because the prefix mean's spread shrinks as 1/sqrt(t), a flat tolerance
#     would reject nearly every early position. h(t,L)=sqrt(L/t) cancels exactly
#     that factor, holding the gate at a constant width in standard deviations.
w_pnpo = np.exp(log_pnpo)
def accept(w, lo, hi, h):
    return ((w >= 1 - lo * h) & (w <= 1 + hi * h)).mean(axis=0)

sched = accept(w_pnpo, *EPS_PNPO, h=np.sqrt(L / t))
flat = accept(w_pnpo, *EPS_PNPO, h=1.0)
assert sched.min() > 0.90 and sched.max() - sched.min() < 0.08     # position-independent
assert flat[0] < 0.02 and flat[-1] > 0.90                          # collapses when early
assert (flat.max() - flat.min()) > 10 * (sched.max() - sched.min())
sds = EPS_PNPO[0] * np.sqrt(L / t) / (SIGMA / np.sqrt(t))
assert np.allclose(sds, sds[0])                                    # the t terms cancel

# (4) At the terminal position PNPO's gate and GSPO's clip act on the identical
#     statistic, so the reported runs differ in tolerance as well as in weighting.
ratio = tuple(round(p / g, 2) for p, g in zip(EPS_PNPO, EPS_GSPO))
assert ratio == (2.33, 2.38)

# (5) At the first position the gate is a token-ratio band under half the width of
#     GRPO's, so the schedule spans both established clip scales.
first = tuple(e * np.sqrt(L) for e in EPS_PNPO)
assert all(0.4 < f / g < 0.45 for f, g in zip(first, EPS_GRPO))

print(f"log-weight std at t=1 / t={L}: exact {log_exact[:, 0].std():.4f} -> "
      f"{log_exact[:, -1].std():.4f}, PNPO {log_pnpo[:, 0].std():.4f} -> "
      f"{log_pnpo[:, -1].std():.6f}")
print(f"scheduled gate acceptance: min {sched.min():.3f}, max {sched.max():.3f}"
      f" ({sds[0]:.2f} sd half-width at every position)")
print(f"flat gate acceptance: t=1 {flat[0]:.4f}, t=L {flat[-1]:.3f}")
print(f"gate half-widths at t=1: -{first[0]:.4f} / +{first[1]:.4f}"
      f" (GRPO clip {EPS_GRPO[0]} / {EPS_GRPO[1]})")
print(f"terminal tolerance vs GSPO: {ratio[0]}x lower, {ratio[1]}x upper")

Executed output:

log-weight std at t=1 / t=15360: exact 0.0523 -> 6.4185, PNPO 0.0523 -> 0.000418
scheduled gate acceptance: min 0.920, max 0.973 (1.74 sd half-width at every position)
flat gate acceptance: t=1 0.0125, t=L 0.940
gate half-widths at t=1: -0.0868 / +0.1177 (GRPO clip 0.2 / 0.28)
terminal tolerance vs GSPO: 2.33x lower, 2.38x upper

Two results there are worth keeping. First, the schedule works: acceptance stays between 92.0% and 97.3% across all 15,360 positions, where a flat tolerance on the same statistic would accept 1.25% at the first position and 94% at the last. Second, at the first token the gate is a band of about -8.7% and +11.8% on the token-local ratio, roughly 0.43 times GRPO's (0.2, 0.28) clip, and at the last token it is a band on the same statistic GSPO clips. The schedule spans both established scales rather than inventing a third.

That second result also exposes a confound in the reported comparison. At the terminal position PNPO's gate and GSPO's clip operate on the identical quantity, but the runs use (7e-4, 9.5e-4) and (3e-4, 4e-4), so PNPO's tolerance is about 2.3x wider. Since DAPO's clip-higher established that widening the upper bound alone changes training behaviour, part of the four-epoch gap could be tolerance rather than weighting granularity. The paper does not ablate the gate and says so.⁸

How to develop with it¶

An implementation has two ways to go wrong that a passing loss curve will not reveal. The first is exponentiating the cumulative log-ratio. The second is renormalizing after gating.

# pnpo_coeff.py -- the detached PNPO coefficient, in log space, with the two details
# that decide whether an implementation is correct: the exact cumulative ratio is not
# representable at reasoning lengths, and gated tokens must not be renormalised away.
import numpy as np


def pnpo_coeff(logp_new, logp_old, adv, eps_lo=7e-4, eps_hi=9.5e-4, renorm=False):
    """Per-token detached coefficient for one response of length L: gate times
    prefix-geometric-mean weight times advantage, divided by the response length."""
    L = logp_new.shape[0]
    t = np.arange(1, L + 1)
    w = np.exp(np.cumsum(logp_new - logp_old) / t)      # never exponentiate the sum
    gate = (w >= 1 - eps_lo * np.sqrt(L / t)) & (w <= 1 + eps_hi * np.sqrt(L / t))
    denom = max(gate.sum(), 1) if renorm else L         # renorm=True is the common bug
    return gate * w * adv / denom


rng = np.random.default_rng(7)

# (1) Against a direct cumulative product, on a short response where that product is
#     still representable, the log-space weight agrees to floating-point noise.
L_s = 64
d = rng.normal(0.0, 0.02, L_s)
naive = np.cumprod(np.exp(d)) ** (1.0 / np.arange(1, L_s + 1))
assert np.allclose(np.exp(np.cumsum(d) / np.arange(1, L_s + 1)), naive, rtol=1e-12)

# (2) At reasoning length, a small systematic drift puts the exact cumulative ratio
#     outside floating-point range, while the prefix mean stays at the drift itself.
L, mu = 15360, 0.05                                     # mean per-token log-ratio
d = mu + rng.normal(0.0, 0.05, L)
with np.errstate(over="ignore"):
    c32 = np.cumprod(np.exp(d).astype(np.float32))
    c64 = np.cumprod(np.exp(d))
log_w = np.cumsum(d) / np.arange(1, L + 1)
died = lambda c: int(np.argmax(~np.isfinite(c)))
assert 0 < died(c32) < died(c64) < L                    # float32 first, then float64
assert np.all(np.isfinite(np.exp(log_w)))
assert abs(np.exp(log_w[-1]) - np.exp(mu)) < 0.01

# (3) Gated positions contribute exactly zero and the survivors keep the 1/L
#     denominator, so rejection shrinks the response's contribution instead of
#     redistributing it. Renormalising by the accepted count erases that shrinkage.
logp_old = rng.normal(-1.5, 0.4, L)
logp_new = logp_old + rng.normal(0.0, 0.05, L)
coef = pnpo_coeff(logp_new, logp_old, adv=1.0)
kept = coef != 0.0
assert coef[~kept].sum() == 0.0
assert coef.sum() < 0.5                                 # this draw keeps about a third
assert abs(pnpo_coeff(logp_new, logp_old, 1.0, renorm=True).sum() - 1.0) < 0.05
assert np.isclose(coef.sum() / pnpo_coeff(logp_new, logp_old, 1.0, renorm=True).sum(),
                  kept.mean(), rtol=1e-9)

# (4) A zero advantage zeroes the response, and the coefficient carries the sign of
#     the advantage rather than of the weight, which is always positive.
assert np.all(pnpo_coeff(logp_new, logp_old, adv=0.0) == 0.0)
assert np.all(pnpo_coeff(logp_new, logp_old, adv=-1.0)[kept] < 0.0)

print(f"cumulative ratio overflows at token {died(c32)} in float32 and "
      f"{died(c64)} in float64 of {L}; prefix mean stays at {np.exp(log_w[-1]):.4f}")
print(f"gate keeps {kept.mean():.1%} of this response; coefficient sum {coef.sum():.4f}"
      f" against {pnpo_coeff(logp_new, logp_old, 1.0, renorm=True).sum():.4f} if"
      f" renormalised by the accepted count")

Executed output:

cumulative ratio overflows at token 1843 in float32 and 14286 in float64 of 15360; prefix mean stays at 1.0510
gate keeps 36.0% of this response; coefficient sum 0.3594 against 0.9985 if renormalised by the accepted count

The overflow result is the practical argument for normalizing at all, and it is independent of every variance argument in the paper. With a mean per-token log-ratio of 0.05, the exact cumulative ratio leaves float32 range at token 1,843 and float64 range at token 14,286 of a 15,360-token response, while the prefix mean sits at 1.05 throughout. Any implementation of the exact form needs its own log-space treatment; the normalized form gets it for free.

The renormalization result is the subtler trap. The paper specifies that the 1/L denominator survives gating: rejected tokens contribute zero and the retained ones are not rescaled by their accepted count.⁵ That means gating shrinks a response's total contribution. In the draw above, 36.0% of positions survive and the response contributes 0.3594 where a renormalized version would contribute 0.9985. Renormalizing looks like a harmless cleanup and silently converts a rejection mechanism into a reweighting one.

In verl the integration seam is the policy-loss registry, which already carries gspo, geo_mean, and cispo entries alongside vanilla. A prefix-normalized loss registers the same way and reuses the same config fields:

# Reference template, unexecuted. Pinned against volcengine/verl main on 2026-08-05,
# where verl/trainer/ppo/core_algos.py exposes @register_policy_loss and actor.yaml
# carries ppo_epochs, clip_ratio_low/high, and policy_loss.loss_mode.
import torch
from verl.trainer.ppo.core_algos import register_policy_loss, agg_loss

@register_policy_loss("pnpo")
def compute_policy_loss_pnpo(old_log_prob, log_prob, advantages, response_mask,
                             loss_agg_mode="seq-mean-token-mean", config=None,
                             rollout_is_weights=None):
    delta = (log_prob - old_log_prob) * response_mask
    t = torch.cumsum(response_mask, dim=-1).clamp(min=1)          # position within response
    log_w = torch.cumsum(delta, dim=-1) / t                       # prefix mean, log space
    L = response_mask.sum(dim=-1, keepdim=True).clamp(min=1)
    h = torch.sqrt(L / t)
    w = torch.exp(log_w.clamp(max=10.0)).detach()                 # coefficient is detached
    gate = ((w >= 1 - config.clip_ratio_low * h) &
            (w <= 1 + config.clip_ratio_high * h)).to(w.dtype)
    losses = -(gate * w * advantages).detach() * log_prob         # gradient only via log_prob
    if rollout_is_weights is not None:
        losses = losses * rollout_is_weights
    loss = agg_loss(loss_mat=losses, loss_mask=response_mask,
                    loss_agg_mode="seq-mean-token-mean", **config.global_batch_info)
    return loss, {"actor/pnpo_reject_frac": (1 - gate)[response_mask.bool()].mean().item()}

# verl actor config for the four-epoch regime. Verify key names on your pinned release.
actor_rollout_ref:
  actor:
    ppo_epochs: 4                  # the reuse lever; 1 is the upstream default
    ppo_mini_batch_size: 64        # prompt groups per optimizer update
    clip_ratio_low: 0.0007         # PNPO base tolerances, not GSPO's 3e-4 / 4e-4
    clip_ratio_high: 0.00095
    loss_agg_mode: seq-mean-token-mean
    policy_loss:
      loss_mode: "pnpo"

TRL exposes the reuse lever as num_iterations (documented as the mu of the GRPO paper, default 1) and the granularity choice as importance_sampling_level, which accepts "token" and "sequence". There is no prefix level upstream, so PNPO on TRL means a custom loss, not a config flag.

How to maintain it¶

Reuse changes which training-health signals matter, because a four-epoch run has one quarter as many reward observations for the same wall clock and each observation is followed by four times as much learner movement.

Track the reuse tax directly, not the objective in isolation. The comparison that matters is one epoch against K epochs at a matched optimizer-update budget, on final held-out score. That is the only comparison the paper runs that is not peak-selected, and it is the one that answers whether reuse is costing you.
Log the rejection fraction per epoch index. Within a batch, epoch 4 sits further from the behaviour policy than epoch 1, so the gate should reject more as epochs advance. A rejection fraction that is flat across epochs means the gate is not binding and the tolerance is too wide; one that approaches total rejection in later epochs means the batch has been exhausted and the extra epochs are contributing nothing but time.
Watch entropy and response length exactly as in a one-epoch run. The signals in the GRPO training-run health runbook do not change, but they are sampled less often. Evaluate on step count, not on hours.
Re-derive G and T after any change to sequence length, batch shape, or engine. The generation share is the whole basis for choosing K, and it moves when the response limit or the sampler does.
Separate the training reward curve from the evaluation curve before believing either. In the reported four-epoch runs, GRPO's reward stays respectable while its evaluation curve declines after its maximum, which is the classic divergence covered in reward design.

How to run it in production¶

Treat the epoch count as a capacity decision with an accuracy budget attached, and validate it on your own workload before it becomes a default.

Start by measuring G and T and computing the speedup curve for your fleet. If generation is under half the step, reuse is not your lever and rollout redundancy or a faster sampler will pay better. If generation dominates, pick the smallest K that captures most of the available speedup, which in the shape above is 4 out of a 3.74x ceiling; going to 16 buys another 0.97x for four times the lag.

Then run the matched-budget A/B before committing. Hold optimizer updates, prompts, and evaluation schedule fixed, vary only the epoch count, and compare final held-out score. Budget for it: this is two full runs, and the paper's own evidence is one seed per configuration, which is not enough to distinguish a 3-point gap on 30-problem benchmarks from noise. In its four-epoch results the AIME24 and AIME25 separations are 1.19 and 0.91 problems out of 30.

Do not copy the tolerances. (7e-4, 9.5e-4) was chosen for a 1.5B model at a 15,360-token limit, and the gate half-width in standard deviations depends on your per-token log-ratio scale, which changes with model size, learning rate, and epoch count. Derive it: measure the standard deviation of the per-token log-ratio in your own run, then choose the base tolerance so that the terminal-position gate sits at the acceptance rate you want.

On a disaggregated cluster, apply the capacity change deliberately rather than letting it absorb itself. Moving from one epoch to four in this configuration cuts required rollout throughput per trainer by 4x. Rebalance the fleet using the matched-instance arithmetic in rollout fleet sizing, and expect the generation share to fall from 73.3% to 40.7% of the step, which changes which side of the split is the bottleneck and therefore which side is worth optimizing next.

Finally, keep the fallback cheap. The weight is one function in the policy-loss registry and the epoch count is one integer. If the reuse tax turns out to be larger than the wall-clock saving on your workload, reverting is a config change, not a rewrite.

Failure modes¶

Exponentiating the cumulative log-ratio. The exact prefix product leaves float32 range partway through a long response and float64 range not much later. Work in log space and normalize; never materialize C_t.
Renormalizing after gating. Dividing by the accepted count instead of the response length turns rejection into reweighting. In the executed draw above that is the difference between a response contributing 0.36 and 1.00.
Reading the headline gap as an accuracy gain. At four epochs PNPO's peak is 50.24 against its own 50.05 at one epoch. The gap to GSPO comes from GSPO losing 2.18 points under reuse, not from PNPO gaining.
Trusting peak-selected tables. The reported per-benchmark bests are each selected independently over 15 evaluations, so the reported average of the three peaks is not achieved by any single checkpoint. Use the final macro score for decisions.
Reading the timing figure as an observed result. The reported 22.4-hour first passage for four-epoch GSPO corresponds to step 158 of a 150-step run at the paper's own 510 seconds per step, and the four-epoch reward panel shows GSPO's smoothed trajectory ending visibly below the 0.25 threshold. That number is a projection past the end of the run, so the "6.4 hours earlier" comparison sets an observed crossing against an extrapolated one.⁷
Treating the comparison as tolerance-matched. At the terminal position the two objectives clip the same statistic at tolerances 2.3x apart, and the gate is not ablated.
Assuming the result generalizes. One 1.5B model, one seed, three small mathematics benchmarks, no mixture-of-experts policy, no agentic rollout, no released code.
Stacking reuse on top of asynchronous staleness without re-measuring. They are additive sources of lag with different structure, and the stability envelope measured for one does not carry to the sum.

References¶

Zhang et al., Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning (arXiv 2608.01418v1, 2 Aug 2026) — PNPO, the weighting ladder, the acceptance gate, and every experimental number quoted above: https://arxiv.org/abs/2608.01418
Zheng et al., Group Sequence Policy Optimization (GSPO), the response-level statistic PNPO reduces to at the terminal position: https://arxiv.org/abs/2507.18071
Shao et al., DeepSeekMath (original GRPO), the token-local ratio and group-relative advantage: https://arxiv.org/abs/2402.03300
Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale — clip-higher and the training data used here: https://arxiv.org/abs/2503.14476
Zhang et al., Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective (CTPO) — the exact cumulative prefix ratio with position-adaptive clipping: https://arxiv.org/abs/2605.07331
Lei et al., A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization (MinPRO) — minimum over prefix ratios instead of the product: https://arxiv.org/abs/2601.22718
Lin et al., Token-Level Policy Optimization (TEPO) — evaluates a length-normalized prefix ratio as its "Sentence Prefix IS" ablation: https://arxiv.org/abs/2604.12736
Arnal et al., Efficient RL Training for LLMs with Experience Replay — the replay-based alternative to in-batch reuse: https://arxiv.org/abs/2604.08706
Precup et al., Eligibility Traces for Off-Policy Policy Evaluation — per-decision importance sampling, the classical foundation: https://www.cs.mcgill.ca/~dprecup/publications/PSS-00.pdf
verl policy-loss registry and actor config, the integration seam for a custom weighting: https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py
TRL GRPOConfig, num_iterations and importance_sampling_level: https://huggingface.co/docs/trl/main/en/grpo_trainer

arXiv 2608.01418v1, Sections 1 and 3.2, and Table 2. Rollout batch 256 prompts by 8 responses; PPO minibatch 64 prompt groups, giving 4 optimizer updates per epoch; 1,024 prompt and 15,360 response token limits; learning rate 1e-6 with 10-step warmup and weight decay 0.1; rollout temperature 1.0; 600 steps at one epoch and 150 at four, both 2,400 optimizer updates; 32 NVIDIA H20 GPUs; one run per configuration at seed 42. The sampled responses, behaviour log probabilities, and group advantages stay fixed while a batch is reused. ↩
arXiv 2608.01418v1, Sections 2.2 and 5, and Equation 3. The state ratio factorizes along the unique prefix path because the transition deterministically appends the sampled token, making C_t the exact joint state-action change-of-measure weight at position t. The related-work section places CTPO at the exact cumulative ratio, MinPRO at the prefix minimum, and TEPO's length-normalized prefix variant as an ablation rather than a primary weight. ↩↩
arXiv 2608.01418v1, Section 3.1 and Equation 7. The weight recovers the local ratio at t = 1 and the full-response geometric mean at t = L, depends only on y_{i,<=t} given the prompt, and preserves the sign of log C_t and the ordering across responses. ↩
arXiv 2608.01418v1, Sections 2.3 and 3.2, Equations 5, 6, and 11, and Appendix B. Retaining the behaviour advantage without bias requires the cumulative prefix score, not the current-token score; the reported objective keeps the current-token score with a group-relative proxy advantage, so it is a proximal surrogate with C_t as its exact reference. The gate, weight, and advantage are all stop-gradient coefficients, so gradients flow only through the current-token log probability. ↩
arXiv 2608.01418v1, Section 3.2, Equations 8, 9, and 11. The scale is h(t, L) = sqrt(L/t) with base tolerances (7e-4, 9.5e-4); bounds are evaluated independently at each position; rejection removes only that position's score term and is hard rather than a truncation to the boundary; the 1/L normalization is retained after gating and the accepted tokens are not renormalized. ↩↩
arXiv 2608.01418v1, Table 1 and Sections 4.2 and 4.3. Table 1 reports the best observed Avg@32 within each training horizon, selected independently per benchmark, with evaluation every 50 steps at one epoch and every 10 at four. At four epochs PNPO is higher at 14 of 15 evaluations and finishes 2.66 points ahead of GSPO; final macro Avg@32 is 49.66 after 150 four-epoch batches against 49.56 after 600 one-epoch batches. AIME 2024 and AIME 2025 comprise 30 problems each, which is what makes the per-benchmark separations about one problem wide. ↩
arXiv 2608.01418v1, Figure 3 and Section 4.3, cross-checked against Section 4.1 and Table 2. The caption states first-passage times computed at 283 and 510 seconds per step for the one- and four-epoch settings. A 150-step four-epoch run is therefore 21.25 hours end to end, and 22.4 hours corresponds to step 158. The one-epoch values (37.7 to 38.0 hours, steps 480 to 483 of 600) and the four-epoch PNPO value (16.0 hours, step 113 of 150) both fall inside their runs. ↩
arXiv 2608.01418v1, Section 6 and the related-work section. The authors scope the findings to a single 1.5B model, three mathematics benchmarks, and one run per configuration; state that the acceptance gate and response-level aggregation are not ablated, so the evidence applies to the complete configuration rather than to prefix normalization alone; and place replay, asynchronous collection, offline data, and training-inference mismatch outside the study. ↩↩