Markdown

Cookbook: RL-train a reasoning model with GRPO, end to end¶

Scope: assembling one complete GRPO training step for a reasoning model from validated pieces, group a batch of sampled rollouts by prompt, score them with a verifiable reward, compute the group-relative advantage, score each rollout's sequence-level log-probability, combine the ratio, clip, and KL terms into one loss, and prove on a small, gradient-checked toy policy that the resulting update actually pushes probability toward the correct answer. This is the assembly and execution companion to GRPO (the algorithm), GRPO variants (why and when to add each stabilizer), and the GRPO training-run health runbook (how to tell if a real run is working); a stage in fine-tuning and post-training.

Every numpy/stdlib block below is executed and asserted in this page (Python 3.9+, numpy 2.4+); several are checked adversarially against each other, including one genuine discrepancy this page found and resolved (below). PyTorch/TRL snippets are reference templates on real APIs, not executed here. The pipeline and worked numbers follow Sebastian Raschka's Build a Reasoning Model (From Scratch) (Manning, 2026, Ch. 6); the code is an independent implementation.

A discrepancy this page found, and how it's handled. rl-grpo.md's validated group_relative_advantage uses numpy's ndarray.std() default, population std (ddof=0). The book's own reported advantage figure (±0.8659 for rewards [1, 1, 0, 0]) only reproduces under PyTorch's tensor.std() default, sample std (ddof=1, Bessel-corrected), which gives ±1.0 under ddof=0 and ±0.866 under ddof=1 for that exact group, a genuine ~13% difference at group size 4 that shrinks as the group grows (checked below). Both are legitimate, self-consistent conventions; they are simply not the same number. This page uses ddof=1 throughout, because it is reproducing the book's PyTorch-derived figures and because PyTorch-based trainers (TRL included) inherit ddof=1 unless the implementation explicitly overrides it. Verify which convention your own library uses before comparing advantage magnitudes across implementations or papers.

What it is¶

RL-training a reasoning model with GRPO is five computations wired into a loop: sample a group of G completions per prompt, score each with a verifiable reward, compute each completion's advantage relative to its group, score each completion's sequence-level log-probability under the policy, and combine the advantage, the log-probability ratio, and a KL penalty into one loss to backpropagate. Everything else (the optimizer, the training loop shell, checkpointing) is a standard PyTorch training loop; GRPO only changes how the loss is computed.

The book's own worked example threads through this whole page: one prompt ("What is x?", ground truth 83), four sampled rollouts, of which two reach \boxed{83} and two do not.

Why use it¶

It is the default RLVR algorithm behind reasoning models, critic-free, verifiable-reward-driven, and already covered algorithmically in GRPO and RLVR; this page is the part those pages leave as "wire it into TRL" made concrete and executable.
Every piece is checkable in isolation before it touches a real model. A sign error in the advantage, the loss, or the KL term trains a real model in the wrong direction for real GPU-hours; the adversarial checks below (finite-difference gradient checking, Monte-Carlo unbiasedness, a reduction identity against the already-validated clipped surrogate) catch exactly this class of bug on a toy problem in milliseconds.
The toy proof is a real regression test. Because the whole pipeline is small enough to gradient-check and run in full, it doubles as a unit test for a from-scratch GRPO implementation: if the toy proof fails, the real implementation has the same bug.

When to use it (and when not)¶

Use this when implementing or debugging a GRPO loss from scratch, or when you need to understand exactly what a library (TRL, verl) computes internally well enough to diagnose it.
Use a library directly (TRL's GRPOTrainer, verl) for production training; do not hand-roll the training loop itself. This page's value is understanding and verifying the pieces, not replacing a maintained trainer.
Start with SFT (or reasoning distillation) before RL; GRPO adapts an already-instruction-following policy, it does not cold-start one.
Skip GRPO if the task has no verifiable reward; see reward design for the alternatives.

Architecture¶

flowchart LR
  PROMPT["Prompt batch"] --> ROLL["Sample G rollouts per prompt<br/>(temperature/top-p)"]
  ROLL --> GROUP["Group by prompt"]
  GROUP --> RWD["Verifiable reward<br/>(math verifier)"]
  RWD --> ADV["Group-relative advantage<br/>(mean-center, std-scale)"]
  ROLL --> OLDLP["Sequence logprob<br/>under OLD (behavior) policy"]
  ROLL --> NEWLP["Sequence logprob<br/>under CURRENT policy"]
  ROLL --> REFLP["Sequence logprob<br/>under FROZEN reference"]
  OLDLP --> RATIO["ratio = exp(new - old)"]
  NEWLP --> RATIO
  NEWLP --> DELTA["delta = ref - new"]
  REFLP --> DELTA
  ADV --> LOSS["GRPO loss:<br/>-clipped_surrogate(ratio, advantage)<br/>+ kl_coef * kl_k3(delta)"]
  RATIO --> LOSS
  DELTA --> LOSS
  LOSS --> BACKWARD["backward + optimizer.step()"]
  BACKWARD -->|"weight sync"| ROLL

Three distinct policies flow through this: the current policy being updated, the old/behavior policy that generated the rollout (identical to current when num_iterations=1, i.e. fully on-policy), and the frozen reference used only for the KL anchor. Conflating old and reference is a common source of bugs; they play different roles and are almost never numerically identical.

How to use it¶

1. Group rollouts by prompt¶

# 01_grouping.py -- runnable (stdlib only).
def group_rollouts(prompt_ids, completions, group_size):
    """prompt_ids, completions: flat and aligned (completion i belongs to
    prompt_ids[i]). Returns prompt_id -> list of completions, each group_size long."""
    assert len(prompt_ids) == len(completions), "prompt_ids and completions must align"
    assert len(prompt_ids) % group_size == 0, "batch size must be a multiple of group_size"
    groups = {}
    for pid, comp in zip(prompt_ids, completions):
        groups.setdefault(pid, []).append(comp)
    for pid, comps in groups.items():
        if len(comps) != group_size:
            raise ValueError(f"prompt {pid} has {len(comps)} rollouts, expected {group_size}")
    return groups


prompt_ids = ["p0", "p1", "p0", "p1", "p0", "p1", "p0", "p1"]
completions = ["a0", "b0", "a1", "b1", "a2", "b2", "a3", "b3"]
groups = group_rollouts(prompt_ids, completions, group_size=4)
assert groups["p0"] == ["a0", "a1", "a2", "a3"] and groups["p1"] == ["b0", "b1", "b2", "b3"]

# --- ADVERSARIAL: a batch size that isn't a multiple of group_size must raise
try:
    group_rollouts(["p0"] * 3, ["x"] * 3, group_size=4)
    raise SystemExit("expected AssertionError for misaligned batch size")
except AssertionError:
    pass

# --- ADVERSARIAL: an uneven split (a dropped rollout) must raise, not silently
# produce a short group that would corrupt the advantage denominator ---------
try:
    group_rollouts(["p0", "p0", "p1", "p1"], ["a", "a", "a", "a"], group_size=1)
    raise SystemExit("expected ValueError: p0 has 2 rollouts but group_size=1")
except ValueError:
    pass

print("01_grouping: all asserts passed")

2. Score rollouts with the verifiable reward¶

The reward requires the answer to be both correct and in the required \boxed{} format (no fallback extraction), which is what keeps it resistant to "correct value, wrong format" reward hacking. This is a condensed version for a plain-integer ground truth; for LaTeX/fraction equivalence use the full canonicalize() + grade() from the verifier cookbook.

# 02_reward.py -- runnable (stdlib only).
def extract_boxed_strict(text):
    marker = text.rfind(r"\boxed")
    if marker == -1:
        return None
    i = marker + len(r"\boxed")
    while i < len(text) and text[i].isspace():
        i += 1
    if i >= len(text) or text[i] != "{":
        return None
    i += 1
    depth, start = 1, i
    while i < len(text) and depth > 0:
        if text[i] == "{":
            depth += 1
        elif text[i] == "}":
            depth -= 1
        i += 1
    return text[start:i - 1] if depth == 0 else None


def reward_rlvr(answer_text, ground_truth):
    extracted = extract_boxed_strict(answer_text)
    if extracted is None:
        return 0.0
    return 1.0 if extracted.strip() == ground_truth else 0.0


# the book's own four-rollout worked example, ground truth "83"
rollouts = [
    r"\boxed{83}",                         # correct, boxed
    r"The correct answer is \boxed{83}",   # correct, boxed
    r"The final answer is 83",             # correct VALUE, but not boxed -> 0.0
    r"We get \boxed{38}",                  # boxed, but wrong value -> 0.0
]
rewards = [reward_rlvr(r, "83") for r in rollouts]
assert rewards == [1.0, 1.0, 0.0, 0.0], rewards

# --- ADVERSARIAL: an unboxed correct answer is not rewarded (the format anchor)
assert reward_rlvr("83", "83") == 0.0
assert reward_rlvr(r"\boxed{}", "83") == 0.0                     # empty box
assert reward_rlvr(r"\boxed{unterminated", "83") == 0.0          # unbalanced braces, no crash

print("02_reward: all asserts passed, rewards:", rewards)

3. Group-relative advantage (with the ddof that matches PyTorch)¶

# 03_advantage.py -- runnable (numpy only).
import numpy as np


def group_relative_advantage(rewards, group_size, eps=1e-4, ddof=1):
    """(rewards - group_mean) / (group_std + eps). ddof=1 matches PyTorch's
    tensor.std() default; ddof=0 is numpy's ndarray.std() default."""
    r = np.asarray(rewards, dtype=np.float64).reshape(-1, group_size)
    mean = r.mean(axis=1, keepdims=True)
    std = r.std(axis=1, ddof=ddof, keepdims=True)
    return (r - mean) / (std + eps)


rewards = [1.0, 1.0, 0.0, 0.0]

# --- reproduces the book's reported figure (0.8659) via ddof=1 -------------
adv_ddof1 = group_relative_advantage(rewards, group_size=4, eps=0.0, ddof=1)
assert np.allclose(np.abs(adv_ddof1), 0.8660, atol=1e-3), adv_ddof1

# --- CONSISTENCY CHECK vs rl-grpo.md: ddof=0 gives a genuinely different number
adv_ddof0 = group_relative_advantage(rewards, group_size=4, eps=0.0, ddof=0)
assert np.allclose(np.abs(adv_ddof0), 1.0, atol=1e-9), adv_ddof0
assert not np.allclose(adv_ddof0, adv_ddof1)
ratio = np.abs(adv_ddof1[0, 0]) / np.abs(adv_ddof0[0, 0])
assert 0.85 < ratio < 0.87, ratio    # ~13.4% smaller magnitude under ddof=1 at G=4

# --- the discrepancy SHRINKS as group size grows (matters less at scale) ----
big_rewards = ([1.0] * 32) + ([0.0] * 32)
adv1_big = group_relative_advantage(big_rewards, group_size=64, eps=0.0, ddof=1)
adv0_big = group_relative_advantage(big_rewards, group_size=64, eps=0.0, ddof=0)
ratio_big = np.abs(adv1_big[0, 0]) / np.abs(adv0_big[0, 0])
assert ratio_big > ratio, (ratio_big, ratio)

# --- ADVERSARIAL: zero-variance group -> advantage 0, never NaN/Inf --------
adv_flat = group_relative_advantage([1.0, 1.0, 1.0, 1.0], group_size=4, eps=1e-4, ddof=1)
assert np.all(np.isfinite(adv_flat)) and np.allclose(adv_flat, 0.0), adv_flat

print("03_advantage: all asserts passed")
print(f"  ddof=1 (this page): {adv_ddof1.ravel()}  ddof=0 (rl-grpo.md): {adv_ddof0.ravel()}")

4. Sequence-level log-probability (sum, not mean)¶

GRPO scores a rollout at the sequence level: sum the answer tokens' log-probabilities, not average them. This differs on purpose from the mean-based confidence scorer in inference-time scaling, which length-normalizes to compare answers of different lengths fairly; GRPO's advantage is already per-rollout, so summing (not averaging) is what makes the loss advantage * sequence_logprob reduce to the correct policy-gradient estimator. One side effect worth knowing: because the sum scales with length, two equally-confident answers get different scores, the shorter one less negative, so the objective mildly favors brevity unless a longer answer earns a higher reward.

# 04_seq_logprob.py -- runnable (numpy only).
import numpy as np


def log_softmax(logits):
    z = logits - logits.max(axis=-1, keepdims=True)
    return z - np.log(np.exp(z).sum(axis=-1, keepdims=True))


def sequence_logprob(logits, token_ids, prompt_len):
    """logits: (T-1, V), logits[t] predicts token_ids[t+1] (next-token shift).
    Returns the SUM of the answer tokens' log-probabilities."""
    logprobs = log_softmax(logits)
    target_ids = token_ids[1:]
    start = max(prompt_len - 1, 0)
    idx = np.arange(len(target_ids))
    selected = logprobs[idx, target_ids]
    return float(np.sum(selected[start:]))


rng = np.random.default_rng(0)
V, prompt_len = 10, 3
token_ids = np.array([1, 2, 3, 4, 5, 6])   # 3 prompt + 3 answer tokens
target_ids = token_ids[1:]
start = max(prompt_len - 1, 0)

confident_logits = rng.normal(size=(len(token_ids) - 1, V))
for t in range(start, len(token_ids) - 1):
    confident_logits[t] = -5.0
    confident_logits[t, target_ids[t]] = 10.0
confident_score = sequence_logprob(confident_logits, token_ids, prompt_len)

unconfident_logits = confident_logits.copy()
unconfident_logits[start:] = rng.normal(size=(len(token_ids) - 1 - start, V))
unconfident_score = sequence_logprob(unconfident_logits, token_ids, prompt_len)
assert confident_score > unconfident_score and confident_score > -0.01

# --- masking invariant: prompt-position logits must not affect the score ---
perturbed_prompt = confident_logits.copy()
perturbed_prompt[:start] = rng.normal(size=(start, V)) * 50.0
assert abs(sequence_logprob(perturbed_prompt, token_ids, prompt_len) - confident_score) < 1e-9

# --- ON-POLICY IDENTITY: same tokens scored under the same logits twice ->
# the importance ratio exp(new-old) is EXACTLY 1.0 -----------------------
old_logp = sequence_logprob(confident_logits, token_ids, prompt_len)
new_logp = sequence_logprob(confident_logits, token_ids, prompt_len)
assert np.exp(new_logp - old_logp) == 1.0

# --- LENGTH SCALING: identical per-token confidence, different length ------
per_token_logp, short_len, long_len = -0.1, 3, 12
short_sum, long_sum = per_token_logp * short_len, per_token_logp * long_len
assert short_sum > long_sum                                  # shorter -> less negative
assert abs(short_sum / short_len - long_sum / long_len) < 1e-9   # but MEANS are identical

print("04_seq_logprob: all asserts passed")

5. The ratio, the clipped surrogate, and the KL penalty¶

clipped_surrogate here is identical to rl-grpo.md's validated function, repeated so this page runs standalone. The KL term is new: kl_k3 is Schulman's unbiased, always-non-negative single-sample estimator of KL(policy || reference), the estimator GRPO/TRL use in practice, in place of the naive logp_policy - logp_reference difference (which is also unbiased but can go negative and has much higher variance, both demonstrated below).

# 05_loss.py -- runnable (numpy only).
import numpy as np


def clipped_surrogate(ratio, advantage, eps=0.2):
    ratio = np.asarray(ratio, dtype=np.float64)
    advantage = np.asarray(advantage, dtype=np.float64)
    unclipped = ratio * advantage
    clipped = np.clip(ratio, 1 - eps, 1 + eps) * advantage
    return np.minimum(unclipped, clipped)


def kl_k3(delta):
    """Unbiased, non-negative KL(policy || reference) estimator from one sample
    drawn from the policy. delta = logp_ref - logp_policy. k3 = exp(d) - d - 1."""
    delta = np.asarray(delta, dtype=np.float64)
    return np.exp(delta) - delta - 1.0


def grpo_loss(ratios, advantages, deltas, clip_eps=0.2, kl_coef=0.0):
    objective = clipped_surrogate(ratios, advantages, clip_eps).mean() - kl_coef * kl_k3(deltas).mean()
    return -objective


# === REDUCTION IDENTITY: at ratio==1 (fully on-policy), the clipped surrogate
# collapses EXACTLY to the advantage -- this is what makes the book's simple
# Ch.6 loss `-(advantage * logprob).mean()` the ratio=1 special case of the
# more general clipped objective already validated in rl-grpo.md. ===========
advantages = np.array([0.8660, 0.8660, -0.8660, -0.8660])
ratio_one = np.ones_like(advantages)
assert np.allclose(clipped_surrogate(ratio_one, advantages, 0.2), advantages)

# === kl_k3: zero at delta=0, never negative (convex, minimum 0 at delta=0) ==
assert kl_k3(0.0) == 0.0
assert np.all(kl_k3(np.linspace(-3.0, 3.0, 200)) >= -1e-12)

# === UNBIASEDNESS (adversarial): Monte-Carlo E_p[k3] vs analytic KL(p||q) ===
rng = np.random.default_rng(0)
p = np.array([0.10, 0.20, 0.30, 0.25, 0.15])
q = np.array([0.15, 0.15, 0.25, 0.30, 0.15])
true_kl = float(np.sum(p * np.log(p / q)))
samples = rng.choice(len(p), size=400_000, p=p)
delta = np.log(q[samples]) - np.log(p[samples])
mc_k3 = kl_k3(delta).mean()
assert abs(mc_k3 - true_kl) < 0.01, (mc_k3, true_kl)

# === k3 is NEVER negative; naive k1 = -delta CAN be, and has higher variance
k1 = -delta
assert np.mean(k1 < 0) > 0.05                    # k1 does go negative in practice
assert np.all(kl_k3(delta) >= 0.0)                # k3 never does
assert np.var(kl_k3(delta)) < np.var(k1)          # and has lower variance

# === grpo_loss: sanity checks ================================================
deltas_zero = np.zeros_like(advantages)
assert np.isclose(grpo_loss(ratio_one, advantages, deltas_zero, kl_coef=0.0),
                   grpo_loss(ratio_one, advantages, deltas_zero, kl_coef=5.0))   # no drift -> no penalty
assert np.isclose(grpo_loss(ratio_one, advantages, deltas_zero, kl_coef=0.0), -advantages.mean())
deltas_nonzero = np.array([0.5, -0.3, 0.4, -0.2])
assert (grpo_loss(ratio_one, advantages, deltas_nonzero, kl_coef=1.0)
        > grpo_loss(ratio_one, advantages, deltas_nonzero, kl_coef=0.0))         # penalty raises the loss

print("05_loss: all asserts passed")
print(f"  MC E[k3]={mc_k3:.5f} vs analytic KL={true_kl:.5f}; var(k3)={np.var(kl_k3(delta)):.5f} vs var(k1)={np.var(k1):.5f}")

6. Does one GRPO update actually work? A gradient-checked toy proof¶

This is the capstone: a tiny categorical policy standing in for the LLM (4 possible answers to the book's own 1 prompt), with an analytic policy gradient checked against finite differences before being trusted, then used to prove the update moves probability mass toward the correct answers and that a KL coefficient controls how far the policy drifts from a frozen reference.

# 06_toy_update.py -- runnable (numpy only).
import numpy as np


def softmax(logits):
    z = logits - logits.max()
    e = np.exp(z)
    return e / e.sum()


def policy_gradient_objective(logits, actions, advantages):
    p = softmax(logits)
    return float(np.mean(advantages * np.log(p[actions])))


def policy_gradient_analytic(logits, actions, advantages):
    """d logp(a)/d logit_j = 1{j=a} - p(j) (the softmax score function)."""
    p = softmax(logits)
    grad = np.zeros_like(logits)
    for a_i, A_i in zip(actions, advantages):
        indicator = np.zeros_like(logits)
        indicator[a_i] = 1.0
        grad += A_i * (indicator - p)
    return grad / len(actions)


def categorical_kl(p, q):
    return float(np.sum(p * (np.log(p) - np.log(q))))


def full_objective(logits, actions, advantages, ref_logits, kl_coef):
    return (policy_gradient_objective(logits, actions, advantages)
            - kl_coef * categorical_kl(softmax(logits), softmax(ref_logits)))


def finite_diff_gradient(f, x, h=1e-5):
    grad = np.zeros_like(x)
    for i in range(len(x)):
        xp, xm = x.copy(), x.copy()
        xp[i] += h
        xm[i] -= h
        grad[i] = (f(xp) - f(xm)) / (2 * h)
    return grad


# === setup: matches the book's own 4-rollout worked example =================
K = 4
init_logits = np.zeros(K)                 # uniform policy
actions = np.array([0, 1, 2, 3])          # one rollout per action
rewards = np.array([1.0, 1.0, 0.0, 0.0])
advantages = (rewards - rewards.mean()) / rewards.std(ddof=1)
assert np.allclose(np.abs(advantages), 0.8660, atol=1e-3)

# === ADVERSARIAL: verify the analytic gradient against finite differences
# BEFORE trusting it for any update -- this is the check that catches sign errors
obj_pg_only = lambda logits: policy_gradient_objective(logits, actions, advantages)
analytic_grad = policy_gradient_analytic(init_logits, actions, advantages)
numeric_grad = finite_diff_gradient(obj_pg_only, init_logits)
assert np.allclose(analytic_grad, numeric_grad, atol=1e-6), (analytic_grad, numeric_grad)
# closed-form special case for this setup (advantages are zero-mean): dJ/d logit_j = A_j / N
assert np.allclose(analytic_grad, advantages / K, atol=1e-9)

# === one policy-gradient step (kl_coef=0): probability shifts toward reward =
p_before = softmax(init_logits)
p_after_no_kl = softmax(init_logits + 1.0 * analytic_grad)
assert p_after_no_kl[0] > p_before[0] and p_after_no_kl[1] > p_before[1]     # correct actions gain
assert p_after_no_kl[2] < p_before[2] and p_after_no_kl[3] < p_before[3]     # wrong actions lose

# === does the KL coefficient control drift from a FIXED, non-uniform reference?
# (KL(p||p)=0 is a minimum, so its gradient is exactly zero right AT the
# reference; start from a policy that has already moved away from it, the
# realistic case of a frozen SFT checkpoint vs a policy several steps in.) ===
ref_logits = np.array([0.20, -0.10, 0.05, -0.15])
drifted_logits = ref_logits + np.array([0.30, 0.20, -0.25, -0.25])
drift_before = float(np.abs(drifted_logits - ref_logits).sum())

small_lr, n_steps = 0.03, 20


def run_toy_training(start_logits, kl_coef):
    logits = start_logits.copy()
    for _ in range(n_steps):
        obj = lambda l: full_objective(l, actions, advantages, ref_logits, kl_coef)
        logits = logits + small_lr * finite_diff_gradient(obj, logits)
    return logits


drift_low = float(np.abs(run_toy_training(drifted_logits, kl_coef=0.1) - ref_logits).sum())
drift_high = float(np.abs(run_toy_training(drifted_logits, kl_coef=10.0) - ref_logits).sum())
assert drift_low > drift_before      # weak KL: policy keeps chasing reward, drifts further
assert drift_high < drift_before     # strong KL: policy is pulled back toward the reference
assert drift_high < drift_low

print("06_toy_update: all asserts passed")
print(f"  p_before={np.round(p_before, 3)}  p_after(no KL)={np.round(p_after_no_kl, 3)}")
print(f"  drift_before={drift_before:.4f}  low-KL={drift_low:.4f}  high-KL={drift_high:.4f}")

7. The training loop skeleton¶

Wire the pieces above into a standard PyTorch loop; only stage 4 (computing the loss) is specific to GRPO. The numpy grpo_loss from step 5 validates the maths, it is not what runs here. Calling numpy functions on tensors that require gradients breaks backpropagation silently (numpy has no autograd graph); a real implementation reimplements the identical formula with torch.exp, torch.clip, and torch.minimum in place of their numpy equivalents, a mechanical, one-for-one swap since every operation used has a direct torch counterpart.

# reference template (PyTorch, not executed here); the maths it calls is validated above.
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)
for step in range(steps):
    optimizer.zero_grad()
    prompt = math_data[step % len(math_data)]

    rollouts = sample_group(model, tokenizer, prompt, group_size=G, temperature=0.8, top_p=0.9)
    rewards = [reward_rlvr(r.text, prompt.ground_truth) for r in rollouts]
    advantages = group_relative_advantage(rewards, group_size=G)

    new_logps = [sequence_logprob(model(r.token_ids), r.token_ids, r.prompt_len) for r in rollouts]
    old_logps = [r.behavior_logprob for r in rollouts]          # captured at sampling time
    ref_logps = [sequence_logprob(ref_model(r.token_ids), r.token_ids, r.prompt_len) for r in rollouts]
    ratios = torch.exp(torch.stack(new_logps) - torch.stack(old_logps))
    deltas = torch.stack(ref_logps) - torch.stack(new_logps)

    loss = grpo_loss(ratios, torch.tensor(advantages).flatten(), deltas, clip_eps=0.2, kl_coef=0.01)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    # weight sync -> rollout engine happens here in a production stack (below)

The book's own loop generates rollouts with the same in-process model it trains, no separate inference engine, fine for a 0.6B model on one GPU, but not how production GRPO runs: rl-grpo.md's architecture uses a dedicated vLLM/SGLang rollout engine with an explicit weight sync step back to it every training step (colocated) or every few steps (disaggregated); see async and disaggregated RL systems and the memory-efficient GRPO recipe for that production shape.

8. Evaluate a checkpoint¶

Score a saved checkpoint's outputs against ground truth with the same grading rule as the reward (boxed and correct); reuse the math verifier for the general case.

# 08_eval.py -- runnable (stdlib only; extract_boxed_strict repeated from step 2
# so this block runs standalone).
def extract_boxed_strict(text):
    marker = text.rfind(r"\boxed")
    if marker == -1:
        return None
    i = marker + len(r"\boxed")
    while i < len(text) and text[i].isspace():
        i += 1
    if i >= len(text) or text[i] != "{":
        return None
    i += 1
    depth, start = 1, i
    while i < len(text) and depth > 0:
        if text[i] == "{":
            depth += 1
        elif text[i] == "}":
            depth -= 1
        i += 1
    return text[start:i - 1] if depth == 0 else None


def checkpoint_accuracy(predictions, ground_truths):
    assert len(predictions) == len(ground_truths) and len(predictions) > 0
    correct = sum(
        1 for p, g in zip(predictions, ground_truths)
        if extract_boxed_strict(p) is not None and extract_boxed_strict(p).strip() == g
    )
    return correct / len(predictions)


base_predictions = ["I think it's 5", "the answer is 12", r"\boxed{7}", "not sure"]
grpo_predictions = [r"\boxed{5}", r"\boxed{12}", r"\boxed{7}", r"\boxed{9}"]
ground_truths = ["5", "12", "7", "9"]
base_acc = checkpoint_accuracy(base_predictions, ground_truths)
grpo_acc = checkpoint_accuracy(grpo_predictions, ground_truths)
assert base_acc == 0.25 and grpo_acc == 1.0 and grpo_acc > base_acc

# --- ADVERSARIAL: an empty prediction set must not divide by zero ----------
try:
    checkpoint_accuracy([], [])
    raise SystemExit("expected AssertionError on empty input")
except AssertionError:
    pass

print(f"08_eval: all asserts passed (synthetic mechanics check: base={base_acc:.0%} grpo={grpo_acc:.0%})")

For scale, on the book's own real run (Qwen3-0.6B, MATH-500, cited as a real reported result, not reproduced here): the base model scores 15.2%, the official Qwen3 reasoning model scores 48.2%, and GRPO from the base model, after only 50 training steps (512 max tokens, 8 rollouts), reaches 47.4%, a large jump from a small number of steps on a tiny model.¹

How to develop with it¶

Group size G trades variance for cost. A larger group gives a lower-variance advantage estimate at proportionally higher rollout cost; watch the fraction of zero-variance groups (all rollouts right or all wrong) using the health runbook.
Tune rollout temperature/top-p for diversity, not just correctness; see inference-time scaling for the mechanics and trade-offs of both knobs.
Start kl_coef at 0 and raise it only if entropy or the reference-KL metric misbehaves, following the guidance in GRPO variants; the toy proof above shows the mechanism, not a universal coefficient.
Reuse num_iterations > 1 only once clipping is verified. Reusing a rollout batch for multiple gradient steps makes ratio != 1 for the second and later steps; confirm the clipped surrogate is wired in (step 5 above) before doing this, or the update is uncontrolled.

How to run it in production¶

Every computation in this cookbook (grouping, reward, advantage, sequence logprob, the loss, and evaluation) is correct at any scale; what changes in production is entirely systems, not algorithm:

Separate the rollout engine from the trainer. Use vLLM/SGLang for rollouts and sync weights to it every step (colocated) or every few steps (disaggregated); see GRPO and async RL systems for the fabric and topology detail.
Batch the sequence-logprob computation across the whole group (and across groups) instead of the one-rollout-at-a-time loop shown above; the maths is identical, only the tensor shapes grow a leading batch dimension.
Quantize if memory-constrained: the memory-efficient GRPO recipe applies QLoRA and FP8 rollouts to exactly this loop without changing the loss.

How to maintain it¶

Watch the same four signals the health runbook reads, reward, advantage std (zero-variance groups), entropy, and KL, and re-run this page's adversarial checks (the reduction identity, the finite-difference gradient check, the KL unbiasedness check) after any change to the loss implementation itself; they are cheap, fast, and catch exactly the class of sign-and-indexing bug that silently trains a broken policy.

Failure modes¶

ddof mismatch silently changes the effective learning rate. Using population std (ddof=0) where your library uses sample std (ddof=1, or vice versa) rescales every advantage by a constant factor that shrinks with group size; not wrong, but worth knowing before comparing runs across implementations (the discrepancy this page found, above).
Reference confused with old/behavior policy. The KL anchor (frozen reference) and the importance-sampling denominator (the policy that generated the rollout) are different models with different purposes; using one where the other belongs silently breaks either the trust region or the regularization.
Mean instead of sum for sequence logprob. Averaging token logprobs length-normalizes for comparing answers (right for the confidence scorer in inference-time scaling); GRPO's loss needs the sum, using the mean rescales the effective learning rate by each rollout's length.
Sign errors in the loss. The loss is -(objective); getting this backward makes the optimizer maximize the wrong thing while every intermediate metric still looks plausible. The reduction identity (step 5) and the toy proof (step 6) are exactly the tests that catch this before it reaches a real model.
Advantage not detached. The advantage must be treated as a constant during the loss's backward pass; letting gradients flow through it (via the reward or the group statistics) corrupts the policy-gradient estimator.
KL gradient vanishing at the reference. KL(p||p)=0 is a minimum, so its gradient is exactly zero when the current policy equals the reference; a KL penalty does nothing on the very first step after a reset and only resists further drift, not the step that created it (found and fixed in the toy proof's own test design, step 6).

References¶

DeepSeekMath (GRPO): https://arxiv.org/abs/2402.03300
DeepSeek-R1 (GRPO at scale, RLVR): https://arxiv.org/abs/2501.12948
Schulman, Approximating KL Divergence (the k1/k3 estimators): http://joschu.net/blog/kl-approx.html
TRL GRPO Trainer docs: https://huggingface.co/docs/trl/grpo_trainer
Raschka, Build a Reasoning Model (From Scratch), Manning Publications, 2026 (Ch. 6: the GRPO pipeline, worked rollout example, and Table 6.1 MATH-500 results this page cites and extends).

Raschka, Build a Reasoning Model (From Scratch), Ch. 6, Table 6.1: on MATH-500 with a Qwen3-0.6B base, base model 15.2%, official Qwen3 reasoning model 48.2%, and GRPO from the base model (original algorithm, no KL term) after 50 training steps at 512 max tokens and 8 rollouts reaching 47.4%. Cited as a real, reported result illustrating GRPO's effect at small scale; independently confirmed against the book's public companion notebook (rasbt/reasoning-from-scratch, ch06/01_main-chapter-code/ch06_main.ipynb) and its bonus ablation table, neither of which reports a 256-token or 4-rollout configuration for this chapter. ↩