Markdown

REINFORCE and RLOO¶

Scope: the critic-free, baseline-only end of the policy gradient family. This page covers REINFORCE as a Monte Carlo estimator of the vanilla policy gradient, the leave-one-out (RLOO) baseline and why excluding a completion's own reward matters, REINFORCE++ global advantage normalization, the exact algebraic relationship between RLOO and GRPO, and when a group baseline beats both a learned critic (PPO) and group standardization. TRL ships this as RLOOTrainer; the KL machinery it uses is in KL regularization.

The numpy block is executed and asserted here, by exact enumeration rather than sampling. The TRL snippet is a reference template on a real API: pin the version and verify field names on the installed release.

What it is¶

REINFORCE is the vanilla policy gradient made concrete: sample completions from the current policy, score them, and weight each completion's log-probability gradient by its return. The estimator, in the completion-level (bandit) form that dominates LLM use:

grad J = (1/N) sum_i  (R_i - b) * grad_theta log pi_theta(y_i | x_i)

b is a baseline, any quantity that does not depend on the sampled action. It cannot change the expected gradient (the EGLP lemma, proven in policy gradient foundations), only its variance. The whole design space of critic-free RL is the choice of b:

Baseline	Name	Rollouts per prompt	Unbiased
None	Plain REINFORCE	1	Yes
Batch mean over all prompts	REINFORCE with a global baseline	1	Only if the sample is excluded
Mean of the other `K-1` completions for the same prompt	RLOO	K	Yes
Group mean, then divide by group std	GRPO	G	No (see below)
Global batch normalization of advantages	REINFORCE++	1 or more	Bias vanishes with batch size

RLOO (REINFORCE Leave-One-Out) samples K completions per prompt and baselines each one with the average reward of the other K-1. Two properties follow. The baseline is prompt-specific, so it removes prompt difficulty from the signal rather than leaving it in as an offset. And because a completion's own reward is excluded, the baseline stays action-independent and the estimator stays unbiased. Both are validated below.

REINFORCE is strictly on-policy in its basic form: the rollouts come from the policy being updated, so no importance ratio is needed. TRL's RLOOTrainer departs from this deliberately, adding a sequence-level clipped importance ratio so that more than one gradient step can be taken per generation batch, which is the same amortization argument PPO makes.

Why use it¶

One trained model. No critic. At roughly 16 GB per billion parameters for a model under AdamW against roughly 2 GB per billion for an inference-only copy, dropping the critic is the single largest memory saving available in RL post-training, larger than dropping the reward model.
Unbiased where GRPO is not. RLOO's leave-one-out construction is the reason it survives the EGLP argument intact. GRPO's group standardization does not, and the difference is a difficulty reweighting across the batch, quantified below.
Fewer moving parts than PPO. No GAE lambda, no value-loss coefficient, no critic warmup, no critic-policy divergence to diagnose. The original RLOO paper's argument is that most of what PPO defends against is not a practical concern in RLHF.
Cheaper than GRPO at small K. GRPO needs a group large enough for the mean and standard deviation to be stable. RLOO needs K >= 2 and degrades gracefully; plain REINFORCE runs at K = 1, which GRPO cannot do at all.

When to use it (and when not)¶

Use RLOO for RLHF with a reward model, where rewards are continuous and a group of 2 to 8 is affordable. This is the setting the original paper measured, and where it beat PPO and DPO.
Use plain REINFORCE when the rollout budget only allows one completion per prompt, or as the reference implementation to check a more complex optimizer against.
Use REINFORCE++ when prompt-level groups are small or unavailable and the batch is large; global advantage normalization is the stated fix for the bias in prompt-local normalization.
Prefer GRPO for large-scale verifiable-reward reasoning runs. It is not better mathematically, but it is where the tooling, the variant literature, and the production recipes live.
Prefer PPO when process rewards or genuinely token-level credit assignment matter. A group baseline is a single scalar per completion; a critic gives a value at every token.
Do not use any of these without a rollout budget. DPO is the offline alternative.

Architecture¶

flowchart LR
  P["Prompt"] --> G["Sample K completions"]
  G --> RW["Reward model or verifier"]
  RW --> B{"Baseline choice"}
  B -->|"none"| V["REINFORCE"]
  B -->|"mean of other K-1"| L["RLOO (unbiased)"]
  B -->|"group mean / group std"| S["GRPO (difficulty reweighted)"]
  B -->|"global batch normalization"| PP["REINFORCE++"]
  V --> U["Policy update: log-prob gradient times advantage"]
  L --> U
  S --> U
  PP --> U
  REF["Frozen reference"] -.->|"KL penalty into the reward"| RW

Note where the KL enters: REINFORCE and RLOO conventionally fold the KL penalty into the reward, computed once and held fixed, rather than into the loss. GRPO puts it in the loss and lets gradients flow through it. KL regularization covers why that distinction changes what is optimized.

How to use it¶

Every claim about which baseline is safe is provable by enumerating all possible groups, with no sampling noise. The block below does that.

# rloo_baselines.py -- what a leave-one-out baseline buys, by EXACT enumeration over every
# possible group of K completions. No sampling, so each claim is proven, not estimated.
# numpy only.
import itertools
import numpy as np

theta = np.array([0.4, -0.2, 1.1, 0.0])       # 4-action stand-in for "which completion"
R = np.array([1.0, 0.0, 0.0, 1.0])            # verifiable (binary) reward per completion
K = 3                                         # completions sampled per prompt

def pi(th):
    e = np.exp(th - th.max())
    return e / e.sum()

def score(a, th):                             # grad log pi(a), softmax policy
    g = -pi(th)
    g[a] += 1.0
    return g

p = pi(theta)
g_true = sum(p[a] * R[a] * score(a, theta) for a in range(4))   # exact policy gradient

def expected_estimator(adv_fn):
    """E over all K-tuples of the group estimator (1/K) sum_i A_i * grad log pi(a_i)."""
    total = np.zeros(4)
    for grp in itertools.product(range(4), repeat=K):
        w = float(np.prod([p[a] for a in grp]))
        adv = adv_fn(np.array([R[a] for a in grp]))
        total += w * sum(adv[i] * score(grp[i], theta) for i in range(K)) / K
    return total

loo      = lambda r: r - (r.sum() - r) / (K - 1)          # RLOO: baseline excludes self
incl     = lambda r: r - r.mean()                          # group mean INCLUDING self
grpo_std = lambda r: (r - r.mean()) / (r.std() + 1e-8)     # GRPO: also divide by group std

g_loo, g_incl, g_std = map(expected_estimator, (loo, incl, grpo_std))

# (1) RLOO IS UNBIASED. Its baseline is independent of the sample it baselines, so the EGLP
#     lemma applies and the expectation is the exact policy gradient, to machine precision.
assert np.allclose(g_loo, g_true, atol=1e-12), (g_loo, g_true)

# (2) THE INCLUDE-SELF MEAN IS BIASED, and the bias is exactly a shrinkage by (K-1)/K.
#     Direction survives, magnitude does not: it quietly rescales the learning rate.
assert not np.allclose(g_incl, g_true, atol=1e-9)
assert np.allclose(g_incl, (K - 1) / K * g_true, atol=1e-12)

# (3) EQUIVALENCE: RLOO's advantage is exactly K/(K-1) times the mean-only (Dr. GRPO)
#     advantage on EVERY group. The two are one method with different constants.
rng = np.random.default_rng(7)
for _ in range(500):
    r = rng.random(K)
    assert np.allclose(loo(r), K / (K - 1) * incl(r), atol=1e-12)

# (4) The std division is NOT a bias within one prompt: with binary rewards it is exactly
#     collinear with the true gradient, a pure rescale whose constant depends only on K.
c = float(g_std @ g_true / (g_true @ g_true))
assert np.linalg.norm(g_std - c * g_true) < 1e-12
assert 1.41 < c < 1.42                                     # K=3 -> sqrt(2)

# (5) ADVERSARIAL: the bias appears ACROSS prompts, because that constant is a function of
#     difficulty. At K=8 the std term amplifies a near-uniform group (1 of 8 correct) by
#     3.02x and a balanced group (4 of 8) by only 2.00x, so easy and hard prompts dominate
#     the batch gradient. That difficulty reweighting is what Dr. GRPO removes.
def amplification(K, m):                                   # |adv| ratio, std-norm vs mean-only
    r = np.array([1.0] * m + [0.0] * (K - m))
    return float(np.abs((r - r.mean()) / r.std())[0] / np.abs(r - r.mean())[0])
amp = {m: round(amplification(8, m), 2) for m in (1, 2, 4, 6, 7)}
assert amp[4] < amp[1] and amp[4] < amp[7]
assert not np.isclose(amp[1], amp[4])

# (6) A PROMPT-SPECIFIC baseline beats a global one when prompts differ in difficulty: a
#     global batch mean leaves the difficulty offset inside every advantage.
easy, hard = rng.normal(0.9, 0.05, 512), rng.normal(0.1, 0.05, 512)
batch = np.concatenate([easy, hard])
global_adv = batch - batch.mean()
prompt_adv = np.concatenate([easy - easy.mean(), hard - hard.mean()])
assert prompt_adv.var() < global_adv.var() / 10

# (7) BOUNDARY: a group where every completion scores the same. RLOO gives exactly zero
#     advantage and stays finite; the std form is finite only because of its epsilon.
flat = np.ones(K)
assert np.allclose(loo(flat), 0.0) and np.all(np.isfinite(loo(flat)))
assert np.all(np.isfinite(grpo_std(flat)))

print("exact gradient        :", np.round(g_true, 4))
print("RLOO estimator        :", np.round(g_loo, 4))
print("include-self estimator:", np.round(g_incl, 4), "| ratio to exact:",
      round(float(g_incl[0] / g_true[0]), 4))
print("std-norm rescale constant (binary, K=3):", round(c, 4))
print("K=8 std amplification by #correct:", amp)
print("advantage variance, global vs per-prompt:",
      round(float(global_adv.var()), 4), "/", round(float(prompt_adv.var()), 4))

Executed output:

exact gradient        : [ 0.143  -0.0512 -0.1877  0.0959]
RLOO estimator        : [ 0.143  -0.0512 -0.1877  0.0959]
include-self estimator: [ 0.0953 -0.0341 -0.1252  0.0639] | ratio to exact: 0.6667
std-norm rescale constant (binary, K=3): 1.4142
K=8 std amplification by #correct: {1: 3.02, 2: 2.31, 4: 2.0, 6: 2.31, 7: 3.02}
advantage variance, global vs per-prompt: 0.1625 / 0.0024

Four results are worth carrying away. RLOO reproduces the exact gradient to machine precision. The include-self mean shrinks it by exactly (K-1)/K, here 0.6667, which is a learning-rate change disguised as a baseline. RLOO's advantage is exactly K/(K-1) times the mean-only advantage, which is the precise sense in which RLOO and Dr. GRPO are the same method. And check (4) is the one that corrects a common overstatement: within a single prompt with binary rewards, GRPO's standard-deviation division is not a bias at all, it is an exact rescale by a constant that depends only on K. The bias is a cross-prompt effect, and check (5) measures it: at K = 8, a group with one correct completion has its advantages amplified 3.02 times against the mean-only form, while a balanced group is amplified 2.00 times, so the batch gradient tilts toward prompts that are nearly solved or nearly hopeless.

How to develop with it¶

TRL ships RLOO as a first-class trainer with the same reward-function interface as GRPOTrainer.

# train_rloo.py. Reference template on TRL's documented RLOO API (docs: main, checked
# 2026-08-25). Pin the release and verify RLOOConfig field names on the installed version.
from datasets import load_dataset
from trl import RLOOTrainer, RLOOConfig
from trl.rewards import accuracy_reward

cfg = RLOOConfig(
    num_generations=4,          # K completions per prompt; the leave-one-out group
    beta=0.05,                  # KL coefficient; the penalty goes into the REWARD
    num_iterations=1,           # 1 keeps the run strictly on-policy (ratio == 1)
    epsilon=0.2,                # clip band, only bites when num_iterations > 1
    normalize_advantages=False, # leave the leave-one-out advantage unstandardized
    use_vllm=True,              # rollouts dominate wall-clock; back them with an engine
)

trainer = RLOOTrainer(
    model="Qwen/Qwen2-0.5B-Instruct",
    reward_funcs=accuracy_reward,
    train_dataset=load_dataset("trl-lib/DeepMath-103K", split="train"),
    args=cfg,
)
trainer.train()

Two details in that config decide whether the run is the algorithm on the page. num_iterations=1 keeps generation and optimization on the same parameters, so the importance ratio is identically one and the clip never activates; raising it amortizes generation across several gradient steps at the cost of going off-policy inside the batch, which is the tradeoff analysed in rollout reuse under policy lag. And normalize_advantages is the switch between the RLOO advantage validated above and a GRPO-style standardized one; turning it on reintroduces exactly the difficulty reweighting that check (5) measures.

The importance ratio TRL applies is sequence-level, pi_theta(o_i | q) / pi_old(o_i | q), not per-token as in GRPO. That places TRL's RLOO on the same side of the token-versus-sequence question as GSPO, which matters when comparing runs across libraries.

How to maintain it¶

Watch the fraction of zero-advantage groups. With binary rewards, a group that is all-correct or all-wrong gives every member an advantage of exactly zero, so the prompt contributes nothing. This is the same frac_reward_zero_std pathology GRPO has, and DAPO's dynamic sampling is the same fix.
Watch advantage mean per batch. RLOO advantages should hover at zero within each prompt. A persistent offset means the baseline has picked up a dependence on the sample, which by the EGLP lemma means the estimator is no longer unbiased.
Watch the KL to the reference separately from the reward. Because the penalty is folded into the reward, a rising raw reward and a rising KL can cancel into a flat training curve that hides both.
Re-derive the effective learning rate when changing K. Check (2) and (3) show that baseline choice rescales the gradient by a K-dependent constant. A K sweep is therefore also an unintentional learning-rate sweep unless the advantage normalization is held fixed.

How to run it in production¶

Size the rollout fleet first. With K completions per prompt, generation cost scales linearly in K while the training step does not. RLOO at K = 4 costs four times the generation of plain REINFORCE for a variance reduction that is usually worth it, but the cluster split between generation and training changes accordingly (rollout fleet sizing, async and disaggregated RL).
Keep num_iterations = 1 until generation is the proven bottleneck. Strictly on-policy REINFORCE has no importance ratio to get wrong, which removes an entire class of silent bugs.
Expect the sampler and trainer to disagree on log-probs anyway. Even at num_iterations = 1, a vLLM or SGLang rollout engine and an FSDP trainer produce different token probabilities for the same weights. That mismatch needs truncated importance sampling regardless of which baseline is in use.
Choose the group size against the reward's noise, not by convention. Continuous reward-model scores stabilize a leave-one-out baseline at small K; binary verifier rewards need a larger K before a group carries any signal at all, since a group of 2 can only produce advantages in a three-valued set.

Failure modes¶

A baseline that includes its own sample. Shrinks the gradient by (K-1)/K and is invisible in the loss curve. Exclude the sample, or accept a documented, deliberate rescale.
Degenerate groups. All-correct or all-wrong groups produce zero gradient; a curriculum with too many of them wastes most of the generation budget.
K = 1 with RLOO. The leave-one-out baseline is undefined with a single completion. Guard for it; do not let a division by K - 1 silently produce an infinity.
Silent off-policy drift. Raising num_iterations without the clipped ratio, or with a mis-specified one, turns a stable on-policy run into an unstable off-policy one.
Comparing across libraries without checking the ratio granularity. TRL's RLOO clips a sequence ratio; most GRPO implementations clip a token ratio. Runs that differ only in this can differ materially in stability.
Assuming a critic is always better. For outcome-only rewards, a critic learns to predict a single scalar per completion, which a group baseline estimates directly for far less compute.

References¶

Ahmadian et al., Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs (RLOO): https://arxiv.org/abs/2402.14740
Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning (REINFORCE, 1992): https://link.springer.com/article/10.1007/BF00992696
Hu et al., REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization: https://arxiv.org/abs/2501.03262
TRL RLOO Trainer documentation: https://huggingface.co/docs/trl/rloo_trainer
Liu et al., Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO, the std-term argument): https://arxiv.org/abs/2503.20783
OpenAI Spinning Up, Intro to Policy Optimization: https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html
Cameron R. Wolfe, REINFORCE: Easy Online RL for LLMs: https://cameronrwolfe.substack.com/p/reinforce
Cameron R. Wolfe, Reinforcement Learning for LLMs: The Complete Guide: https://cameronrwolfe.substack.com/p/llm-rl