Markdown

Runbook: GRPO training-run health check¶

Scope: diagnose an unhealthy GRPO / RLVR training run from its own metrics, reward, group-relative advantage (mean and standard deviation), policy entropy, KL-to-reference, and clip fraction, before the run wastes further GPU-hours on a collapsed or hacked policy.

Run this when a GRPO/RLVR run's reward curve looks wrong (flat, spiky, or suspiciously perfect), when entropy has dropped or spiked unexpectedly, or as a routine check partway through any new run before trusting its checkpoint. Severity: training-quality regression, not an outage; the job keeps running and burning GPU-hours while producing a bad policy. This is the algorithmic-health companion to GRPO variants and training tricks's "How to maintain it" guidance and to the generic training OOM and MFU regression runbooks, which cover crashes and throughput, not training-signal health.

The thresholds below are a documented starting point derived from the metric definitions themselves (advantage mean must be exactly zero by construction; entropy bands scale with log(vocab_size)), not a universal constant; tune them against your own model's healthy-run history before wiring them into an alert. The Python block is executed and asserted in this page (numpy only); the PromQL/W&B snippets are reference templates, adapt the metric names to your own logging setup (observability).

GRPO's advantage is computed by mean-centring (and usually std-normalizing) rewards within each group of G rollouts for one prompt, so several of its failure signatures are visible directly in that statistic, without needing a held-out eval. This runbook reads reward, advantage, entropy, and KL together, because any one of them in isolation is ambiguous: a reward mean near 1.0 alone could mean the task is solved or that the verifier is broken, and the other metrics disambiguate which.

Trigger¶

Reward curve is flat, oscillating without trend, or has reached a value near its maximum unexpectedly early.
Entropy has dropped sharply within a few hundred steps, or is rising instead of gradually settling.
KL-to-reference is climbing without bound, or the trainer's clip fraction is unusually high.
Training reward is rising while a held-out evaluation score is flat or falling (the reward-hacking signature; see reward design).
Routine mid-run check on any new GRPO/RLVR campaign before committing further compute to it.

Pre-checks¶

Confirm the metrics are actually logged per-step, reward, per-sample advantage (or its mean/std), entropy, and KL; a run missing any of these cannot be diagnosed from training signal alone and needs instrumentation added first (observability).
Confirm the verifier itself is correct before reading anything else. GRPO cannot distinguish "the policy is failing" from "the reward function is wrong"; spot-check the verifier or reward model on a handful of known-correct and known-incorrect completions.
Have a held-out evaluation set on hand. Training-metric health and true capability are different questions; several diagnoses below require cross-referencing a held-out score that the training loop does not directly optimize.

Flow¶

flowchart TB
  A["Reward / entropy / KL look wrong"] --> B{"Advantage std ~ 0?"}
  B -->|"yes"| C["Zero-variance groups:<br/>prompts too easy or too hard"]
  B -->|"no"| D{"Entropy < ~0.5 and falling fast?"}
  D -->|"yes"| E["Entropy collapse:<br/>add clip-higher / KL term"]
  D -->|"no"| F{"KL climbing without bound?"}
  F -->|"yes"| G["Policy drifting from reference:<br/>lower LR or raise KL coefficient"]
  F -->|"no"| H{"Reward mean near max, but<br/>held-out eval flat or falling?"}
  H -->|"yes"| I["Reward hacking:<br/>audit the reward and eval set"]
  H -->|"no, reward mean near max<br/>AND held-out eval also high"| J["Task solved:<br/>stop or raise difficulty"]
  C --> K["Apply fix, resume, re-check"]
  E --> K
  G --> K
  I --> K
  J --> K

Procedure¶

Recompute the advantage mean and standard deviation independently of the trainer's own logs. The mean must be (numerically) exactly zero by construction, mean-centring within a group cannot produce anything else; if the logged mean is not near zero, the bug is in the trainer's advantage computation itself, not in training dynamics.

# grpo_health_diagnose.py -- runnable (numpy only).
import numpy as np


def group_advantage_stats(rewards, group_size, eps=1e-4):
    """Recompute the same advantage the trainer computes, for independent
    verification: mean-centre and std-normalize within each group of G rollouts."""
    r = np.asarray(rewards, dtype=float).reshape(-1, group_size)
    adv = (r - r.mean(axis=1, keepdims=True)) / (r.std(axis=1, keepdims=True) + eps)
    return float(adv.mean()), float(adv.std())


def diagnose(reward_mean, advantage_std, entropy, entropy_trend, kl, vocab_size,
             zero_var_eps=1e-3, entropy_collapse=0.5, entropy_random_frac=0.9, kl_alarm=0.10):
    """Findings this runbook's procedure flags from one step's (or one rolling
    window's) health metrics."""
    findings = set()
    if advantage_std < zero_var_eps:
        findings.add("zero_variance_groups")          # no gradient flows this step
    if entropy < entropy_collapse:
        findings.add("entropy_collapsed")              # policy has gone near-deterministic
    if entropy > entropy_random_frac * np.log(vocab_size):
        findings.add("entropy_near_random")             # policy hasn't learned anything yet
    if entropy_trend > 0.02:
        findings.add("entropy_rising")                  # entropy should settle, not climb
    if kl > kl_alarm:
        findings.add("kl_blowup")                       # policy drifting far from reference
    if reward_mean > 0.98:
        findings.add("reward_saturated")                # task solved: stop or escalate difficulty
    return findings


# --- sanity check: the trainer's own advantage mean must be ~0 -------------
adv_mean, adv_std = group_advantage_stats([1.0, 1.0, 0.0, 0.0], group_size=4)
assert abs(adv_mean) < 1e-9        # nonzero here => bug in the trainer's own computation
assert 0.9 < adv_std < 1.1

# --- a healthy step produces no findings -----------------------------------
assert diagnose(reward_mean=0.55, advantage_std=0.95, entropy=1.4, entropy_trend=-0.01,
                 kl=0.02, vocab_size=151_000) == set()

# --- zero-variance groups: every rollout in the sampled groups agreed ------
zv = diagnose(reward_mean=0.99, advantage_std=0.0, entropy=1.2, entropy_trend=-0.01,
              kl=0.02, vocab_size=151_000)
assert {"zero_variance_groups", "reward_saturated"} <= zv

# --- entropy collapse: policy has gone deterministic too early -------------
assert diagnose(reward_mean=0.40, advantage_std=0.90, entropy=0.15, entropy_trend=-0.05,
                 kl=0.03, vocab_size=151_000) == {"entropy_collapsed"}

# --- KL blowup: policy racing away from the reference ----------------------
assert diagnose(reward_mean=0.60, advantage_std=0.85, entropy=1.3, entropy_trend=0.01,
                 kl=0.35, vocab_size=151_000) == {"kl_blowup"}

# --- ADVERSARIAL: multiple simultaneous findings are ALL reported ----------
compound = diagnose(reward_mean=0.99, advantage_std=0.0, entropy=0.1, entropy_trend=-0.2,
                     kl=0.40, vocab_size=151_000)
assert compound == {"zero_variance_groups", "reward_saturated", "entropy_collapsed", "kl_blowup"}

print("grpo_health_diagnose: all asserts passed")

Check advantage standard deviation for zero-variance groups. A std near 0 means every rollout in the sampled groups scored the same (all correct or all wrong), so the gradient for that step is zero, GRPO's known degenerate case. If this is widespread rather than occasional, the prompt set is miscalibrated for the current policy (too easy once reward saturates, or too hard from the start); mix in harder or easier prompts, or enable dynamic sampling to drop zero-variance groups (GRPO variants).
Check entropy level and trend. As a starting point (re-derive from your own healthy-run history, not a universal constant): entropy below roughly 0.5 with a fast negative trend signals collapse toward a near-deterministic policy; entropy approaching log(vocab_size) means the policy is still close to random and has not learned yet; entropy should decline gradually as training proceeds, a sudden cliff is the warning sign, not the absolute level at any one step. Mitigate collapse with clip-higher or a KL term (GRPO variants).
Check KL-to-reference for unbounded drift. A KL that climbs without settling means the policy is moving arbitrarily far from the reference model, which risks degenerate outputs the reward function happens to score well; lower the learning rate or raise the KL coefficient.
Check the trainer's clip fraction. A high and rising fraction of tokens being clipped means the policy is proposing large updates the surrogate objective is actively restraining; treat it as an early-warning companion to the entropy trend, not an independent alarm.
Cross-reference training reward against a held-out evaluation. This is the one diagnosis the training metrics alone cannot make: if training reward is rising or has saturated while held-out evaluation is flat or falling, that is reward hacking, not progress. Audit the reward function and the eval set itself before touching the trainer (reward design, evaluation integrity).

# REFERENCE TEMPLATE: adapt metric names to your own logging backend (W&B, Prometheus).
# PromQL sketch for a training-metrics dashboard panel; not runnable as shown.
avg_over_time(grpo_advantage_std[15m])
avg_over_time(grpo_policy_entropy[15m])
rate(grpo_kl_to_reference_sum[15m]) / rate(grpo_kl_to_reference_count[15m])

Verification¶

Advantage standard deviation is materially above 0 across most groups (the exact healthy band is model- and task-specific; compare against this run's own early-training baseline).
Entropy has stopped falling sharply and is declining gradually, or has stabilized in a band consistent with this model's own healthy-run history.
KL-to-reference is bounded and roughly steady, not monotonically climbing.
Held-out evaluation moves with training reward, not away from it.
Record the specific finding (zero-variance groups, entropy collapse, KL blowup, reward hacking, or task-solved) and the fix applied, in the run's own tracking record (experiment tracking), so the next run's baseline includes this one's lesson.

Rollback¶

If the diagnosis is reward hacking or a broken verifier, stop the run: continuing trains a worse policy every additional step. Roll back to the last checkpoint recorded before the training/held-out reward divergence began, fix the reward or verifier, and resume from that checkpoint, not from the diverged one (checkpoint recovery). If the diagnosis is entropy collapse or KL blowup, the current checkpoint is usually salvageable: resume with the adjusted hyperparameter (clip-higher, KL coefficient, learning rate) rather than restarting from scratch.

Checkpoint recovery and resume: resume cleanly after a rollback.
RL checkpoint/resume validation: certify a resumed run is genuinely the same experiment.
Training out-of-memory: a crash, not a health regression; different runbook.
Operational runbooks index.

References¶

Shao et al., DeepSeekMath (introduces GRPO): https://arxiv.org/abs/2402.03300
DeepSeek-R1 (RLVR with GRPO at scale): https://arxiv.org/abs/2501.12948
Yu et al., DAPO (clip-higher, dynamic sampling, zero-variance group filtering): https://arxiv.org/abs/2503.14476
Cui et al., The Entropy Mechanism of RL for Reasoning LLMs: https://arxiv.org/abs/2505.22617
Raschka, Build a Reasoning Model (From Scratch), Manning Publications, 2026 (Ch. 7, advantage and entropy tracking).