Markdown

On-policy distillation¶

Scope: post-training methods where a student generates its own on-policy rollouts and receives token-level guidance on those visited states. The guidance may come from a frozen teacher (GKD), the teacher's change relative to its base checkpoint (OPD²), or a skill-conditioned copy of the current agent combined with agentic RL (SEED, SDAR). These methods combine the dense signal of distillation with the state distribution induced by the student.

The framework code below is written against real APIs (TRL). Treat every trl.experimental.gkd snippet as a reference template: pin the version, verify the import path on your install, and validate before production use. The NumPy blocks labelled "core math, runnable" are self-contained and executable. The OPD² paper's official repository contained only a README at audited commit 76bc984bca982240e3ff46372366f5a0ab58fa03, so this page does not invent a framework API for it.¹⁰ SEED's loss was checked against its released implementation at commit 2cf2fadca3c5aba28da68e8e1405182ba8d90e6c; its PyTorch test suite was not run on this CPU-only documentation host.¹³

What it is¶

Standard knowledge distillation is off-policy: the student is trained to match a teacher on the teacher's own outputs (or a fixed dataset). That creates a train/inference distribution mismatch: at inference the student conditions on its own prefixes, drifts into states the teacher never demonstrated, and errors compound over the sequence (exposure bias).

On-policy distillation fixes this by sampling sequences from the student and having the teacher score those student-generated tokens, so supervision lands exactly on the states the student actually visits. The canonical method is GKD (Generalized Knowledge Distillation, Agarwal et al., ICLR 2024), which "trains the student on its self-generated output sequences by leveraging feedback from the teacher".¹ A lambda parameter interpolates from off-policy (lambda=0, teacher data) to fully on-policy (lambda=1, student rollouts), and the divergence is a generalized Jensen-Shannon that spans forward KL through reverse KL.¹ MiniLLM independently showed reverse KL (mode-seeking) beats forward KL for generative LMs and derived the on-policy objective.² Thinking Machines Lab's 2025 write-up popularized the practical framing: "sample trajectories from the student model and use a high-performing teacher to grade each token" via the teacher's log-probabilities, a reverse-KL per-token reward.³ The on-policy principle traces to DAgger in imitation learning: to avoid compounding error you must train under "the distribution of observations [the policy] induces".⁴

Terminology (read this). The umbrella term is on-policy distillation (OPD). "On-policy self-distillation (OPSD)" is a narrower, recent (2026) label for the self-teacher case, where the model teaches itself (a larger sibling in the same family, an earlier/EMA checkpoint, or a privileged-context copy) rather than a separate teacher.⁷ This page covers the general method and the self-distillation variant; OPSD is not an industry-standard name for the general technique.

OPD² and SEED are different interventions. OPD² needs a post-trained teacher and that teacher's base checkpoint; it uses the centered teacher-to-base log-probability change as the update magnitude and keeps only directions that agree with conventional OPD.⁹ SEED is a self-evolving agentic-RL method: the current policy analyzes its trajectories into natural-language hindsight skills, scores the same sampled actions with and without those skills, and adds a gated token-imitation loss to the RL objective.¹¹ SEED drops the analyzer and skills at inference.

This is the lambda=1 (fully on-policy) end of the spectrum. lambda=0 degenerates to training on a fixed corpus of teacher-generated text, ordinary off-policy SFT on synthetic data; that recipe (the DeepSeek-R1-Distill lineage: generate reasoning traces from a teacher, filter, tokenize, mask, SFT) is common enough and different enough operationally, no teacher inference at training time, to warrant its own page: see reasoning distillation via SFT on teacher traces.

Why use it¶

Dense supervision. Distillation supplies a target at each of N token positions; a full distribution carries O(NV) probability values for vocabulary size V, while episodic RL may supply one scalar per sequence.³ Whether that yields better sample efficiency depends on the teacher and task.
On-policy, so no exposure bias. Unlike off-policy KD, it supervises the student's own trajectories, correcting the mistakes the student actually makes and removing the compounding train/inference gap.¹²
Cheap versus RL. Thinking Machines report reaching teacher performance roughly 7 to 10x faster than RL, on the order of 50 to 100x less compute, and a 9 to 30x cost reduction versus off-policy distillation.³ Qwen3's strong-to-weak distillation "significantly outperforms reinforcement learning in performance and training efficiency", requiring only 1/10 of the GPU hours of its multi-stage RL pipeline.⁵
Strong results. On AIME'24, Thinking Machines report on-policy distillation at 74.4% versus 67.6% for RL and 55.0% for off-policy distillation from the same start.³
Teacher-change selectivity. OPD² attempts to suppress generic teacher preferences by measuring how post-training changed the teacher relative to its base model. In the authors' single-run experiments it leads standard OPD and ExOPD on every reported model, reasoning-mode, and domain aggregate, but it does not always beat the unmodified student.⁹
Agentic hindsight without inference-time scaffolding. SEED converts successful and failed on-policy trajectories into transient skills during training, then optimizes the ordinary policy jointly with GRPO. On the reported Qwen2.5-3B experiments, SEED scores 91.8 versus GRPO's 75.0 on ALFWorld, 45.7 versus 36.4 on Search QA, and 78.9% versus 63.3% WebShop success.¹¹

When to use it (and when not)¶

Use on-policy distillation when a stronger teacher exists and you want to transfer its capability cheaply: compressing a large model into a small one (strong-to-weak), or recovering behaviour after domain fine-tuning (continual learning, personalization).³
Prefer RLVR when a verifiable reward supplies information unavailable from a suitable teacher. Teacher supervision is not a strict evaluation ceiling, but it cannot directly label behavior outside the teacher signal.
Prefer SFT to cold-start format cheaply on demonstrations when you do not need the teacher's full distribution.
Combine them. A common 2026 recipe is SFT → on-policy distillation (cheap capability transfer) → RLVR (push past the teacher on verifiable tasks).
Use OPD² when the post-trained teacher and its exact base checkpoint are both available, share a tokenizer with the student, and the teacher's post-training change is the signal to transfer. Do not use it with an opaque teacher API or an unknown base model.
Use SEED for long-horizon, sparse-reward agent tasks when the policy can analyze trajectories into useful hindsight and the training cluster can afford analysis plus two token-scoring passes. It is not a replacement for a trustworthy environment reward.
For incompatible tokenizers, switch method rather than abandoning distillation. Plain GKD needs a shared vocabulary and now hard-errors without one, but cross-tokenizer distillation (TRL's GOLDTrainer, extending the Universal Logit Distillation loss) aligns the two token streams by visible text and merges the teacher's multi-token probabilities through the chain rule. See knowledge distillation method selection.
Avoid when the teacher cannot meaningfully score the student's out-of-distribution tokens.

Architecture¶

The loop has four moving parts: the student (being trained, holds the generation path), the teacher (frozen, resident or served, scores tokens), the per-token divergence (reverse KL by default), and the on-policy fraction lmbda that decides how much of each batch is student-generated. The teacher runs inference in the loop but takes no gradients.

flowchart LR
  P["Prompt"] --> GEN["Student samples rollout (on-policy)"]
  GEN --> TOK["Student tokens y_t"]
  TOK --> TEACH["Teacher scores each token<br/>(teacher logprobs, frozen)"]
  TEACH --> LOSS["Per-token reverse-KL loss<br/>KL(student || teacher)"]
  LOSS -->|"gradient (student only)"| GEN
  OFF["Off-policy KD: train on the TEACHER's outputs"] -.->|"distribution mismatch / exposure bias"| GEN
  subgraph SERVE["Systems shape"]
    TEACHSRV["Teacher (sharded / served pool)"] --- TEACH
    STUD["Student on FSDP"] --- GEN
  end

OPD² and SEED signal paths¶

GKD, OPD², and SEED share student-generated trajectories but differ in what scores a sampled token.

flowchart TB
  R["Student rollout tokens"] --> GKD["GKD: frozen teacher distribution"]
  R --> D2["OPD²: post-trained teacher and teacher base"]
  R --> ANA["SEED: current-policy trajectory analysis"]
  GKD --> KLL["Forward, reverse, or generalized-JSD loss"]
  D2 --> DELTA["Centered teacher/base delta<br/>sign-gated by conventional OPD"]
  ANA --> SKILL["Natural-language hindsight skill"]
  SKILL --> PAIR["Rescore same actions<br/>ordinary vs skill context"]
  PAIR --> GATE["Sigmoid-gated sampled-token NLL"]
  GATE --> JOINT["GRPO plus 0.01 auxiliary loss"]
  JOINT --> SERVE["Ordinary policy only at inference"]

OPD². For student p_s, post-trained teacher p_T, teacher base p_B, and a sampled token, conventional OPD reward is r_opd = log p_T - log p_s and the delta reward is r_delta = log p_T - log p_B. Center each reward under the student's next-token distribution. OPD² keeps the centered delta only when a_delta * a_opd > 0; otherwise its token advantage is zero.⁹ The paper approximates each centering expectation over the student's top 1,024 tokens but does not say whether the truncated probabilities are renormalized. That choice can change a centered sign and therefore the gate.

The teacher-to-base ratio includes every post-training change, including reasoning, instruction following, format, style, and safety behavior. Calling it a pure reasoning signal is an interpretation, not an identified causal quantity. The gate also retains conventional OPD's direction: delta determines the magnitude only when its sign agrees with centered OPD.

SEED. Stage 1 uses GLM-5.2 to analyze 1,440 completed trajectories, from 180 tasks with eight rollouts each, into natural-language skills, then performs three SFT epochs so the policy can act as an analyzer.¹¹ Stage 2 snapshots the current policy, samples eight trajectories per task, and produces one hindsight skill from each completed trajectory. The same sampled action token is rescored under ordinary history h and skill-augmented history H(h, s):

delta_t = stopgrad(log p(a_t | H) - log p(a_t | h))
g_t = sigmoid(beta_opd * delta_t), with beta_opd = 5
L_opd = mean(g_t * [stopgrad(log p(a_t | H)) - log p(a_t | h)]) over model-generated action tokens
L_seed = L_grpo + lambda_opd * L_opd, with lambda_opd = 0.01 and a reference-KL term

Up to the detached additive teacher term, the auxiliary gradient is gate-weighted negative log-likelihood on sampled tokens. It is not a full-distribution KL. Since 0 < g_t < 1, a negative skill gap attenuates reinforcement but does not directly penalize the sampled token. The method depends on exploration to sample a better action and on the hindsight analyzer to assign a larger gate to valuable actions. The paper's theory requires positive covariance between that gate and true action value; it does not prove monotonic return improvement.¹¹

SDAR. SDAR (Self-Distilled Agentic Reinforcement Learning) keeps RL as the primary optimization backbone and treats on-policy self-distillation as a gated auxiliary objective, which is the same architectural position SEED takes.¹² Its teacher branch is the current policy augmented with privileged context, specifically retrieved skills. The token loss is l_t = g_t * (log p_theta+(y_t | s_t+) - log p_theta(y_t | s_t)), where theta+ denotes the privileged branch and the gate is detached with a stop-gradient so gradients flow only through the student log-probability.

The contribution over SEED is that SDAR specifies three gate constructions rather than one:

Gap gating, g_t = sigmoid(beta * delta_t), weighting tokens the privileged teacher endorses and attenuating those it rejects. This is the same functional form as SEED's gate.
Entropy gating, g_t = sigmoid(beta * h_t), targeting positions where the student is most uncertain, ignoring the teacher entirely.
Soft-OR gating, g_t = sigmoid(beta * [1 - (1 - h_t)(1 - delta_t)]), firing when either signal is high.

The motivation for asymmetric treatment is specific to skill-conditioned guidance: a negative teacher gap may mean the action was genuinely wrong, or it may mean skill retrieval or utilization failed. Attenuating rather than penalizing avoids training on the second case. The first author of SDAR is also an author of SEED, and the two papers share the gate shape; read them as one line of work rather than independent confirmations.

Core math (runnable): the three SDAR gates¶

This block computes all three gate variants on the same detached signals and checks the properties that distinguish them. Run: python3 sdar_gates.py.

import numpy as np


def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-np.clip(x, -60, 60)))


BETA = 5.0

# Detached per-token signals over one batch of sampled action tokens.
# delta: privileged-teacher minus student log-prob on the SAMPLED token.
# h:     student entropy at that position, normalized to [0, 1].
delta = np.array([1.20, 0.40, 0.05, 0.00, -0.05, -0.40, -1.20])
h = np.array([0.10, 0.90, 0.10, 0.50, 0.90, 0.10, 0.90])

gap = sigmoid(BETA * delta)
ent = sigmoid(BETA * h)
soft_or = sigmoid(BETA * (1.0 - (1.0 - h) * (1.0 - delta)))

print("  delta     h |   gap    entropy  soft-OR")
for d, hh, g, e, s in zip(delta, h, gap, ent, soft_or):
    print(f"  {d:+5.2f}  {hh:4.2f} |  {g:5.3f}   {e:5.3f}    {s:5.3f}")

# 1. Gap gating is asymmetric but never reverses the sign of the update.
assert np.all((gap > 0.0) & (gap < 1.0)), "a sigmoid gate stays strictly inside (0, 1)"
assert gap[0] > 0.9 and gap[-1] < 0.1, "endorsed tokens amplified, rejected tokens attenuated"
assert np.isclose(gap[3], 0.5), "a zero gap is the neutral point"
print(f"\n[1] zero-gap neutral point = {gap[3]:.3f}; "
      f"strongest rejection still passes {gap[-1]:.3f} of the signal, not 0")
# Attenuation is soft: a rejected token is down-weighted, never penalized.
# That is the whole design claim, and it is also its limitation.
assert gap[-1] > 0.0, "negative-gap tokens are attenuated, not punished"

# 2. Gap and entropy gating disagree, which is why the choice matters.
disagree = np.sum((gap > 0.5) != (ent > 0.5))
print(f"[2] gap and entropy gating disagree on {disagree} of {len(delta)} tokens")
assert disagree > 0, "the two gates are not interchangeable"
# Entropy gating ignores the teacher entirely: it fires on student uncertainty.
assert np.isclose(ent[1], ent[4]), "entropy gate depends only on h, not on delta"
print(f"[2] tokens 1 and 4 have opposite delta ({delta[1]:+.2f} vs {delta[4]:+.2f}) "
      f"but identical entropy gates ({ent[1]:.3f})")

# 3. Soft-OR fires when EITHER signal is high, so it is the least selective.
print(f"[3] mean gate value: gap {gap.mean():.3f}, entropy {ent.mean():.3f}, "
      f"soft-OR {soft_or.mean():.3f}")
assert soft_or.mean() > gap.mean(), "soft-OR passes more signal than gap gating alone"
# Adversarial: soft-OR can amplify a token the teacher REJECTED, purely because
# the student was uncertain there. Gap gating cannot.
rejected_but_uncertain = (delta < 0) & (h > 0.8)
idx = int(np.argmax(rejected_but_uncertain))
print(f"[3] token {idx}: delta {delta[idx]:+.2f} (teacher rejects) but h {h[idx]:.2f} "
      f"-> gap {gap[idx]:.3f} attenuates, soft-OR {soft_or[idx]:.3f} amplifies")
assert soft_or[idx] > 0.5 > gap[idx], "soft-OR can amplify a teacher-rejected token"

# 4. Beta sets how sharply the gate discriminates.
print()
for b in (1.0, 5.0, 20.0):
    g = sigmoid(b * delta)
    print(f"[4] beta={b:4.1f}: gate range [{g.min():.3f}, {g.max():.3f}], "
          f"spread {g.max() - g.min():.3f}")
assert sigmoid(20.0 * delta).max() - sigmoid(20.0 * delta).min() > \
       sigmoid(1.0 * delta).max() - sigmoid(1.0 * delta).min()
print("[4] beta -> 0 makes every gate 0.5 (uniform auxiliary loss); "
      "beta -> inf makes it a hard sign filter")
assert np.allclose(sigmoid(0.0 * delta), 0.5)

print("\nAll assertions passed.")

Executed output:

  delta     h |   gap    entropy  soft-OR
  +1.20  0.10 |  0.998   0.622    0.997
  +0.40  0.90 |  0.881   0.989    0.991
  +0.05  0.10 |  0.562   0.622    0.674
  +0.00  0.50 |  0.500   0.924    0.924
  -0.05  0.90 |  0.438   0.989    0.989
  -0.40  0.10 |  0.119   0.622    0.214
  -1.20  0.90 |  0.002   0.989    0.980

[1] zero-gap neutral point = 0.500; strongest rejection still passes 0.002 of the signal, not 0
[2] gap and entropy gating disagree on 4 of 7 tokens
[2] tokens 1 and 4 have opposite delta (+0.40 vs -0.05) but identical entropy gates (0.989)
[3] mean gate value: gap 0.500, entropy 0.823, soft-OR 0.824
[3] token 4: delta -0.05 (teacher rejects) but h 0.90 -> gap 0.438 attenuates, soft-OR 0.989 amplifies

[4] beta= 1.0: gate range [0.231, 0.769], spread 0.537
[4] beta= 5.0: gate range [0.002, 0.998], spread 0.995
[4] beta=20.0: gate range [0.000, 1.000], spread 1.000
[4] beta -> 0 makes every gate 0.5 (uniform auxiliary loss); beta -> inf makes it a hard sign filter

All assertions passed.

Two consequences for anyone choosing a gate. Soft-OR is not a safe default. It amplifies a token the privileged teacher rejected whenever the student happened to be uncertain there, which is exactly the case gap gating is designed to suppress. Token 4 in the output shows this directly: gap gating attenuates it to 0.438 while soft-OR passes 0.989. Beta is the discrimination knob, and it has degenerate limits at both ends. At beta near zero every gate is 0.5 and the auxiliary loss becomes uniform, dropping the whole selectivity argument; at large beta the gate becomes a hard sign filter and the "soft attenuation" property that motivates the design disappears.

Reported evidence and its boundary¶

Method	Reported result	Compute and reproducibility boundary
OPD²	Highest of OPD, ExOPD, and OPD² in all 21 model/mode/domain aggregates across Qwen3 1.7B, 4B, 8B and Gemma E4B	One training run per setting, no seeds or uncertainty. About 8% to 28% more wall time than OPD. Official repository contains no code or recipe.⁹¹⁰
OPD² counterexample	Gemma code average is 55.2 before distillation and 49.5 after OPD²	The method is not a universal improvement over the starting student. Teacher scores and RL/SFT comparisons are absent.⁹
SEED, Qwen2.5-3B	GRPO to SEED: ALFWorld 75.0 to 91.8; Search QA 36.4 to 45.7; WebShop success 63.3% to 78.9%	Point estimates with no seeds, error bars, runtime, memory, or GPU-hour comparison. Training uses 8 A800 80GB GPUs.¹¹
SEED sample use	ALFWorld with 60% of training instances scores 80.7, above full-instance GRPO at 75.0	This is instance efficiency, not compute efficiency: analysis can generate 4,096 tokens and skill-conditioned rescoring adds a forward pass.¹¹
SEED unseen split	ALFWorld unseen tasks improve 70.9 to 86.2 overall; Clean falls 82.4 to 79.5	This is within-ALFWorld transfer across its unseen split, not cross-domain generalization.¹¹
SDAR	Over GRPO: +9.4% ALFWorld, +7.0% Search-QA, +10.2% WebShop accuracy, across the Qwen2.5 and Qwen3 families	Reported as improvements over GRPO on the same three environment families SEED uses, so the two results are not independent evidence. The paper's own framing is that naive GRPO+OPSD is unstable and SDAR fixes it; that comparison is internal to the paper.¹²

The SEED implementation matches the sampled-token loss, masking, detached skill branch, and sigmoid gate in verl/trainer/ppo/core_algos.py at the audited commit.¹³ Exact reproduction still needs reconciliation: the paper states 150 updates and KL coefficient 0.01, while the public ALFWorld and WebShop launchers default to 160 updates and the Search launcher uses 0.001 KL. The repository does not publish the exact SFT dataset or task-ID manifest.

Core math (runnable): OPD² centering and the SEED gate¶

This NumPy block checks the method-specific invariants, including a sign-conflict token that OPD² rejects, zero updates when the teacher equals its base or the student equals the teacher, a negative SEED gap that attenuates rather than reverses the sampled-token gradient, the neutral 0.5 gate, and action-token masking. Run: python3 opd2_seed_signals.py.

# opd2_seed_signals.py - core math, runnable (numpy only).
import numpy as np

def center(reward, probs):
    return reward - np.dot(probs, reward)

def opd2_advantage(student, teacher, base):
    r_opd = np.log(teacher) - np.log(student)
    r_delta = np.log(teacher) - np.log(base)
    a_opd = center(r_opd, student)
    a_delta = center(r_delta, student)
    return np.where(a_delta * a_opd > 0.0, a_delta, 0.0), a_opd, a_delta

student = np.array([0.45, 0.35, 0.20])
teacher = np.array([0.60, 0.25, 0.15])
base = np.array([1.0, 1.0, 5.0]) / 7.0
adv, a_opd, a_delta = opd2_advantage(student, teacher, base)

# Full-vocabulary centering has zero student-weighted mean.
assert abs(np.dot(student, a_opd)) < 1e-12
assert abs(np.dot(student, a_delta)) < 1e-12
# The middle token has a sign conflict and must be suppressed.
assert np.array_equal(adv != 0.0, np.array([True, False, True]))
assert np.all(np.sign(adv[adv != 0.0]) == np.sign(a_opd[adv != 0.0]))
# Boundary cases: no teacher change, or no teacher/student gap, closes the gate.
assert np.allclose(opd2_advantage(student, teacher, teacher)[0], 0.0)
assert np.allclose(opd2_advantage(teacher, teacher, base)[0], 0.0)

plain = np.array([-1.8, -0.7, -1.2, -2.0])
skill = np.array([-0.9, -1.4, -1.0, -2.0])
mask = np.array([1.0, 1.0, 1.0, 0.0])
delta = skill - plain
gate = 1.0 / (1.0 + np.exp(-5.0 * delta))
weighted_gate = gate * mask

assert gate[0] > gate[2] > 0.5 > gate[1] > 0.0
assert gate[3] == 0.5 and weighted_gate[3] == 0.0
assert np.all((gate > 0.0) & (gate < 1.0))
print("OPD2 advantage:", np.round(adv, 4).tolist())
print("OPD2 active mask:", (adv != 0.0).tolist())
print("SEED gate:", np.round(gate, 4).tolist())
print("masked SEED gate:", np.round(weighted_gate, 4).tolist())

Executed output:

OPD2 advantage: [0.9056, 0.0, -2.0902]
OPD2 active mask: [True, False, True]
SEED gate: [0.989, 0.0293, 0.7311, 0.5]
masked SEED gate: [0.989, 0.0293, 0.7311, 0.0]

Core math (runnable): the per-token reverse-KL reward¶

The signal is a per-token reverse KL, KL(student_t || teacher_t), summed over the vocabulary at each student token. This numpy-only block computes it, and asserts the properties the loss depends on: non-negativity (Gibbs), zero exactly when the student matches the teacher (the objective's optimum), asymmetry versus forward KL (adversarial: if they were equal the whole forward/reverse debate would be vacuous), a blow-up when the teacher assigns near-zero mass where the student has mass (the tokenizer/out-of-distribution failure mode), and the dense O(N) signal count. Run: python3 reverse_kl_reward.py.

# reverse_kl_reward.py — core math, runnable (numpy only).
# The teacher grades each student token t by KL(student_t || teacher_t).
import numpy as np

def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def reverse_kl(student, teacher):        # KL(student || teacher), per token (per row)
    return np.sum(student * (np.log(student) - np.log(teacher)), axis=-1)

def forward_kl(student, teacher):        # KL(teacher || student), the off-policy direction
    return np.sum(teacher * (np.log(teacher) - np.log(student)), axis=-1)

rng = np.random.default_rng(0)
T, V = 6, 50                              # 6 student tokens, vocab 50
s = softmax(rng.normal(size=(T, V)))
t = softmax(rng.normal(size=(T, V)))
per_token = reverse_kl(s, t)             # the dense per-token teacher grade

# 1) Non-negativity (Gibbs' inequality): every per-token KL >= 0.
assert np.all(per_token >= -1e-12), per_token.min()
# 2) Edge / equivalence: zero iff distributions match. The optimum of the objective.
assert np.allclose(reverse_kl(s, s), 0.0, atol=1e-12), "KL(p||p) must be 0"
# 3) Adversarial asymmetry: reverse KL != forward KL in general (mode-seeking vs covering).
assert not np.allclose(reverse_kl(s, t), forward_kl(s, t)), "KL must be asymmetric"
# 4) Edge boundary: a teacher with mass -> 0 where the student has mass blows the reward
#    up. This is the tokenizer-mismatch / OOD "teacher can't score" failure mode.
t_bad = t.copy(); t_bad[0] = 1e-9; t_bad[0, 0] = 1.0 - 49e-9
assert reverse_kl(s[0:1], t_bad[0:1])[0] > reverse_kl(s[0:1], t[0:1])[0], "OOD -> larger penalty"
# 5) Dense-reward accounting: N tokens -> N signals; RL gives 1 scalar per sequence.
assert per_token.shape[0] == T and per_token.size == T
print("reverse-KL per-token reward: PASS", np.round(per_token[:3], 4).tolist())

How to use it¶

TRL implements on-policy distillation twice. GKDTrainer is the original and the clearer one to learn on; DistillationTrainer is the scaled implementation of the same method, and is what a real cluster run should use. Both live under trl.experimental as of TRL v1.8.0 (GKD graduated out of the top-level namespace; verify the import path on your installed version).

trl.experimental.gkd.GKDTrainer: wraps SFTTrainer, takes a teacher_model, and exposes the two defining knobs lmbda (on-policy fraction) and beta (forward to reverse KL). Teacher weights are co-resident, and the loss runs over the full vocabulary.
trl.experimental.distillation.DistillationTrainer: the same objective, rebuilt for scale. It adds an external teacher server (use_teacher_server=True, teacher_model_server_url=..., so the teacher never has to fit on the training GPUs), a generation buffer that batches prompts across gradient-accumulation steps into one vLLM call, and top-k logit truncation (loss_top_k, default 1) with a residual tail bucket. Its defaults also differ: lmbda=1.0, beta=1.0, temperature=1.0, so it is fully on-policy reverse KL out of the box, where GKDConfig defaults to 0.5 / 0.5 / 0.9. See knowledge distillation method selection for the teacher-server constraints and the cost of top-k truncation.

# on_policy_distill.py — REFERENCE TEMPLATE (needs TRL + transformers, not run here).
# TRL >=1.7: import from trl.experimental.gkd. Verify the import path on your version.
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl.experimental.gkd import GKDConfig, GKDTrainer

student = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B")   # the model being trained
teacher = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B")     # stronger; frozen, scores tokens
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")        # student + teacher must share a tokenizer

trainer = GKDTrainer(
    model=student,
    teacher_model=teacher,
    args=GKDConfig(
        lmbda=1.0,        # 1.0 = fully on-policy (student generates); 0.0 = off-policy (teacher data)
        beta=1.0,         # 0.0 = forward KL, 1.0 = reverse KL (mode-seeking, MiniLLM/TML choice); 0.5 = JSD (TRL default)
        temperature=0.9,  # sampling temperature for the student rollouts
        max_new_tokens=512,
        bf16=True,
    ),
    processing_class=tokenizer,
    train_dataset=ds,     # rows are chat "messages"; the student generates completions on-policy
)
trainer.train()

lmbda is what makes it on-policy: at 1.0 every batch is student-generated and scored by the teacher; at 0.0 it degenerates to supervised (off-policy) distillation on teacher token-probabilities. It is a per-step Bernoulli coin-flip over whole batches, not a fraction of each batch: the trainer runs if random.random() <= self.lmbda: once per training step and swaps the entire batch to student-generated tokens on success. At lmbda=0.5, half your steps are pure off-policy SFT; no batch is ever half-and-half. DistillationConfig words it correctly ("each gradient accumulation slice is randomly assigned as on- or off-policy"). beta picks the divergence: the reverse-KL end (beta=1.0) is the mode-seeking objective MiniLLM and Thinking Machines favour, though the GKD authors note the optimal beta is task-dependent (TRL defaults to 0.5, the JSD midpoint).¹²

What beta actually interpolates (core math, runnable). GKD's beta selects a generalized Jensen-Shannon divergence whose limits recover the two KLs: beta -> 0 is forward KL KL(teacher || student), beta -> 1 is reverse KL KL(student || teacher), and beta = 0.5 is the standard symmetric JSD. This numpy block builds that family with the teacher weighted by beta in the mixture (the TRL/MiniLLM convention) and asserts the endpoints match the KLs computed independently (equivalence to a slow reference), the midpoint is bounded by ln 2, identical distributions give zero at every beta, and the two ends genuinely differ (adversarial). Run: python3 gkd_beta_jsd.py.

# gkd_beta_jsd.py — core math, runnable (numpy only).
# beta interpolates forward KL (beta->0) through reverse KL (beta->1), matching GKDConfig.beta.
import numpy as np

def kl(a, b):
    return float(np.sum(a * (np.log(a) - np.log(b))))

def gjsd(student, teacher, beta):        # mixture weights the TEACHER by beta
    m = beta * teacher + (1.0 - beta) * student
    term_t = beta * kl(teacher, m) if beta > 0.0 else 0.0
    term_s = (1.0 - beta) * kl(student, m) if beta < 1.0 else 0.0
    return term_t + term_s

student = np.array([0.60, 0.25, 0.15])
teacher = np.array([0.30, 0.45, 0.25])
fwd = kl(teacher, student)               # forward KL -> beta->0 end
rev = kl(student, teacher)               # reverse KL -> beta->1 end

# 1) Equivalence to slow reference: normalized generalized JSD recovers the KL endpoints.
d0 = gjsd(student, teacher, 1e-4) / (1e-4 * (1 - 1e-4))
d1 = gjsd(student, teacher, 1 - 1e-4) / ((1 - 1e-4) * 1e-4)
assert abs(d0 - fwd) < 1e-2, (d0, fwd)   # beta->0 == forward KL
assert abs(d1 - rev) < 1e-2, (d1, rev)   # beta->1 == reverse KL
# 2) Midpoint is the symmetric JSD and is bounded by ln 2.
mid = gjsd(student, teacher, 0.5)
assert 0.0 < mid < np.log(2) + 1e-12, mid
# 3) Edge: identical distributions -> zero divergence at every beta.
for b in (0.0, 0.25, 0.5, 0.75, 1.0):
    assert abs(gjsd(student, student, b)) < 1e-12, b
# 4) Adversarial: forward != reverse, so the two endpoints must differ.
assert abs(fwd - rev) > 1e-6 and not np.isclose(d0, d1), (fwd, rev)
# 5) Boundary: gjsd is finite and non-negative across the whole sweep.
assert all(g >= -1e-12 and np.isfinite(g) for g in (gjsd(student, teacher, b) for b in np.linspace(0, 1, 11)))
print("generalized JSD (beta): PASS  fwd=%.5f rev=%.5f jsd_mid=%.5f" % (fwd, rev, mid))

Operational contracts for OPD² and SEED¶

OPD² requires three compatible next-token distributions on each student prefix:

Generate the rollout with the student.
Score every sampled prefix with the student, post-trained teacher, and the teacher's exact base model.
Center r_opd and r_delta under the student distribution, apply the strict positive sign gate, mask non-model tokens, and optimize the student only.
Record whether centering used the full vocabulary or a top-k approximation, including truncation and renormalization rules.

There is no released OPD² implementation to import as of the audited commit.¹⁰ Keep the NumPy formulation above as the executable reference, and require an independent slow full-vocabulary implementation before optimizing or truncating it.

SEED requires a two-stage launch rather than a drop-in loss switch:

Bootstrap the analyzer with completed trajectories and externally generated skills, then SFT the policy to emit the required skill schema.
Freeze the current policy for rollout and skill generation within an outer update.
Preserve sampled token IDs, masks, tool observations, and ordinary history. Add the skill only to the detached rescoring branch.
Compute GRPO on model-generated action tokens, compute the gated OPD auxiliary on the same tokens, and update the ordinary policy.
Evaluate and serve without the skill branch.

The released SEED scripts and compute_opd_loss are reference templates pinned to commit 2cf2fadca3c5aba28da68e8e1405182ba8d90e6c.¹³ They were source-audited but not executed here. Reconcile the paper/launcher differences above before treating a run as a paper reproduction.

How to develop with it¶

Iterate on the teacher, the divergence, and the on-policy fraction:

Teacher choice sets the supervision quality and scoring cost. Compare specialized and general teachers on the same student-state distribution rather than assuming size determines the result.
beta (divergence). Reverse KL (beta→1) concentrates the student on the teacher's dominant modes (crisper, less diverse); forward KL (beta→0) spreads mass to cover the teacher (more diverse, can over-generalize). The sweet spot is task-dependent.¹ Do not sweep it naively. TRL hard-codes the two endpoints but computes the interior as an un-rescaled mixture, so the loss magnitude is discontinuous at both ends: an interior beta silently shrinks the gradient by roughly a factor of beta * (1 - beta), confounding the divergence shape with the effective learning rate. That factor is symmetric and collapses at both ends, so beta=0.95 is not "almost reverse KL at 95% magnitude", it carries about 5% of it. Compare the endpoints against each other, or renormalise the interior loss by beta * (1 - beta) yourself. Derivation and executed numbers in knowledge distillation method selection.
lmbda (on-policy fraction). The GKD authors find "on-policy data (high lmbda) performs better"; drop below 1.0 only if student generation is the bottleneck.
temperature controls exploration in the student rollouts; too low starves the teacher of informative mistakes to correct.
seq_kd=True switches to sequence-level KD (supervised fine-tuning on teacher-generated sequences), an off-policy baseline to compare against.

What lmbda mixes, and why the reward is dense (core math, runnable). In GKDTrainer, lmbda is the Bernoulli probability that an entire training step uses student-generated data; it is not a within-batch fraction. The expected loss across steps is a convex combination of on-policy and off-policy losses. Separately, the reason on-policy distillation is roughly 50 to 100x cheaper than RL per unit of signal is that an N-token episode yields N per-token grades under distillation but a single scalar under RL. This block asserts the endpoints (lmbda 0 and 1), convexity, agreement with a Monte-Carlo Bernoulli(lmbda) reference, rejection of out-of-range lmbda (adversarial guard), and the N-token-position-versus-one-episodic-scalar count including the N=1 boundary. Run: python3 gkd_lambda_dense.py.

# gkd_lambda_dense.py — core math, runnable (numpy only).
import numpy as np

def gkd_expected_loss(loss_on_policy, loss_off_policy, lmbda):
    assert 0.0 <= lmbda <= 1.0            # lmbda is a fraction; reject anything else
    return lmbda * loss_on_policy + (1.0 - lmbda) * loss_off_policy

L_on, L_off = 0.30, 0.50
# 1) Edge endpoints: lmbda=1 -> pure on-policy, lmbda=0 -> pure off-policy (SFT-like).
assert gkd_expected_loss(L_on, L_off, 1.0) == L_on
assert gkd_expected_loss(L_on, L_off, 0.0) == L_off
# 2) Convexity: any interior lmbda lies strictly between the endpoints.
assert min(L_on, L_off) < gkd_expected_loss(L_on, L_off, 0.5) < max(L_on, L_off)
# 3) Equivalence to a slow reference: Monte-Carlo the Bernoulli(lmbda) mixture.
rng = np.random.default_rng(1)
draws = np.where(rng.random(200_000) < 0.7, L_on, L_off)
assert abs(draws.mean() - gkd_expected_loss(L_on, L_off, 0.7)) < 5e-3, draws.mean()
# 4) Adversarial guard: lmbda outside [0,1] must raise, not be used silently.
for bad in (-0.1, 1.1):
    try:
        gkd_expected_loss(L_on, L_off, bad); raise SystemExit("accepted bad lmbda")
    except AssertionError:
        pass
# 5) Dense vs sparse: N-token episode -> N signals (distill) vs 1 (RL). Includes N=1.
signal_count = lambda n, mode: n if mode == "distill" else 1
for N in (1, 8, 512):
    assert signal_count(N, "distill") == N and signal_count(N, "rl") == 1
    assert signal_count(N, "distill") >= signal_count(N, "rl")
print("lmbda mixing + dense reward: PASS  E[loss]@0.5=%.3f" % gkd_expected_loss(L_on, L_off, 0.5))

How to integrate with it¶

On-policy distillation slots between cheap format alignment and open-ended RL. The canonical 2026 pipeline is SFT (cold-start format) then on-policy distillation (transfer the teacher's capability densely) then RLVR (push past the teacher on verifiable tasks); each stage feeds the next its checkpoint. Integration checklist:

Tokenizer contract. Student and teacher must share a tokenizer, because the loss aligns per-token logits (below). Confirm identical vocab and special tokens before wiring anything; a mismatch corrupts the KL silently. Within a model family (Qwen3, Llama) this is usually satisfied by construction.
Data contract. train_dataset rows are chat messages with prompts only for the on-policy path: the student generates the completion, the teacher scores it. Do not pre-fill completions unless you are deliberately running the lmbda<1 off-policy mix.
Where it sits in the stack. GKDTrainer wraps SFTTrainer, so it inherits the same SFT/LoRA plumbing (PEFT adapters, packing, bf16); a LoRA student keeps the trainable footprint small while the frozen teacher scores. Feed its output straight into a GRPO/RLVR stage.
Serving the teacher. For large teachers, serve logprobs from a dedicated inference pool (vLLM-style) rather than co-resident weights; the trainer consumes teacher log-probabilities per token, so the integration point is a logprob endpoint, not a full generation API.
OPD² base contract. Pin the teacher and base revisions as a pair, verify identical token IDs and chat rendering across all three models, and fail before training if either revision or vocabulary differs.
SEED trajectory contract. Keep environment observations out of both GRPO and OPD action masks. Preserve the exact sampled action tokens while rendering the skill only in the detached branch.
SEED analyzer boundary. Treat trajectory text as untrusted input. Validate the skill schema, cap its length, strip executable instructions, and retain the source trajectory and outcome for audit.

How to run it in production¶

Two models resident (or one served). Budget memory for student training state (optimizer + activations under FSDP) and a teacher doing forward-only inference. If the teacher does not fit alongside, serve it on a separate pool and stream logprobs; that decouples teacher throughput from trainer step time.
Pin versions. GKD moved to trl.experimental.gkd in TRL >= 1.7; experimental namespaces move. Pin trl, transformers, and the model revisions, and re-verify the import path in CI, since the reference templates above assume a specific layout.
Throughput is teacher-bound. The teacher scores every student token in the loop, so its inference rate caps the trainer. Size the teacher pool to the student's rollout rate; an under-provisioned teacher starves the trainer (a failure mode below).
Fabric. Two co-trained/served models reuse the same NVLink/IB-with-GDR concerns as any multi-model job; keep teacher-logprob transfer off the critical path where possible (performance tuning).
Determinism and cost. Fix the student sampling temperature and seeds for reproducible rollouts. Because the reward is dense, far fewer gradient steps reach a target than RL, which is where the ~50 to 100x compute reduction and ~7 to 10x fewer steps come from.³
OPD² capacity. A training step needs student, teacher, and base scoring. Replicate or serve the frozen pair separately, batch prefixes across both scorers, and budget for the paper's measured 8% to 28% wall-time increase over standard OPD rather than treating the extra model as free.⁹
SEED capacity. Separate rollout, trajectory analysis, skill-conditioned rescoring, and actor update queues. The paper reports eight A800 80GB GPUs but no wall time, utilization, memory peak, or cost comparison, so measure all four queues before claiming efficiency.¹¹
Inference parity. Evaluate the plain policy without OPD² scorers or SEED skills. A deployment that retains privileged skill context measures a different system from the paper.

How to maintain it¶

Track the student-minus-teacher gap without treating it as a theorem. A shrinking gap can indicate saturation of the available teacher signal, but a student may exceed the teacher on a downstream metric through regularization, data selection, or architecture. Use RLVR when a verifiable reward adds a new signal.
Monitor divergence health. Log per-token reverse KL and output diversity. A collapsing diversity signals reverse-KL mode collapse (beta too aggressive); a persistent large KL on in-distribution tokens signals a broken tokenizer alignment or an over-strong/OOD teacher.
Re-validate on upgrades. Any bump of trl/transformers/teacher revision can change the GKD API or teacher behaviour; re-run the numpy core-math blocks (they are version-independent) plus a smoke train, and re-check the import path.
Keep the off-policy baseline. Retain a lmbda=0 (or seq_kd=True) run as a regression anchor so you can always prove the on-policy contribution is still positive after a change (the ablation below).
Monitor OPD² gates. Track sign-gate acceptance, centered-advantage means, top-k mass retained, and agreement between truncated and full-vocabulary centering on a canary batch. A near-zero acceptance rate is a stalled run; disagreement points to a truncation artifact.
Monitor SEED hindsight. Track skill parse failures, skill length, positive/neutral/negative delta fractions, gate quantiles, analyzer latency, invalid-action rate, and reward by skill source. Review sampled skills against their trajectories.
Keep causal ablations. For OPD² retain original student, OPD, and ungated delta baselines. For SEED retain GRPO, no-hindsight-SFT, no-evolving-OPD, and a fixed-analyzer-on-current-trajectories arm; the paper omits the last arm, so it does not isolate analyzer freshness.

How to scale it¶

Plain GKD avoids reward-model inference and sparse-reward search, but the frozen teacher must score every student token. Teacher memory and throughput therefore dominate. OPD² adds a second frozen scorer for the teacher's base model; place the teacher/base pair behind one batched service when compatible, or allocate three model replicas explicitly. Keep the trainable student on FSDP and treat log-probability transport as a first-class bandwidth budget (performance tuning).

SEED has the systems shape of agentic RL, not cheap teacher distillation. It needs tool-environment rollouts, trajectory analysis, ordinary and skill-conditioned scoring, GRPO/reference computation, and actor updates. Disaggregate those queues when analyzer or tool latency would otherwise stall the GPUs, and preserve the token/mask identity across every handoff.

At the extreme, the teacher is served entirely off-box: run it behind a logprob endpoint on an inference pool, batch the student's rollouts against it, and the trainer scales like an SFT job plus a network round-trip per batch. This is the shape that lets a small student distill from a very large teacher without ever co-locating both sets of weights.

On-policy self-distillation (the OPSD variant)¶

Self-distillation sets the teacher to a version of the same model. The explicit form, and the one least likely to break across TRL versions, is to pass the same checkpoint as both model and teacher_model; TRL's GKD docs also describe a teacher_model_name_or_path=None shortcut ("the teacher model will be the same as the model being trained"; verify on your installed version).⁶ Three practical shapes:

Strong-to-weak within a family. A large sibling teaches a small one: Qwen3 distills from Qwen3-32B/235B into the smaller models.⁵ This is the most common "self" case.
Privileged-context self-teacher. The teacher is the same model conditioned on privileged information (a verified reasoning trace, the answer) that the student does not see; the student learns to reproduce that behaviour from the question alone. This is the setting that coined "on-policy self-distillation (OPSD)".⁷
Earlier/EMA checkpoint. The model distills from a stronger past or averaged copy of itself to stabilize or recover behaviour.

A self-teacher supplies no external correctness label. Privileged context or hindsight can reorganize behavior already reachable by the policy, but the loop can also reinforce shared blind spots. Use a stronger external teacher or a trustworthy RLVR signal when the missing information is external to the current policy.

Prefix failure and its fix. Dense per-token supervision has its own pathology, distinct from reverse-KL mode collapse: when the teacher's distribution along a rollout is effectively bimodal (a privileged-context OPSD teacher diverges onto a different valid continuation than the one the student's prefix already committed to), per-token loss truncation or reweighting cannot repair it, because the gradient fragments across the two modes rather than pointing at either one. Trajectory-Refined Distillation (TRD) addresses this at the trajectory level instead of the token level: it revises the student's rollout under teacher guidance while staying within on-policy support, correcting the problematic prefix before the per-token loss is computed. TRD applies to plain on-policy distillation and to the OPSD privileged-context case.⁸

Cookbook (common use cases)¶

1. Recover behaviour after domain fine-tuning (continual learning)

# REFERENCE TEMPLATE (needs TRL, not run here).
# Fine-tuning on internal docs degraded instruction-following. Distill it back on-policy,
# using the ORIGINAL instruct checkpoint as the teacher, on the student's own outputs.
from trl.experimental.gkd import GKDConfig, GKDTrainer
trainer = GKDTrainer(
    model=domain_tuned_model,                      # lost some IF ability
    teacher_model="Qwen/Qwen3-8B",                 # original instruct behaviour = the teacher
    args=GKDConfig(lmbda=1.0, beta=1.0, bf16=True),
    processing_class=tokenizer, train_dataset=if_prompts)   # prompts only; student generates
trainer.train()

2. Self-distillation (model is its own teacher)

# REFERENCE TEMPLATE (needs TRL, not run here).
# Self-distillation: the teacher is a frozen copy of the SAME checkpoint (same id for model + teacher).
# TRL also documents GKDConfig(teacher_model_name_or_path=None) as a shortcut; verify it on your version.
from trl.experimental.gkd import GKDConfig, GKDTrainer
BASE = "Qwen/Qwen3-8B"
trainer = GKDTrainer(
    model=BASE,
    teacher_model=BASE,                     # same checkpoint -> the model is its own (frozen) teacher
    args=GKDConfig(lmbda=1.0, beta=1.0, bf16=True),
    processing_class=tokenizer, train_dataset=ds)
trainer.train()

3. Off-policy baseline to prove the on-policy win

# REFERENCE TEMPLATE (needs TRL, not run here).
# Ablation: lmbda=0.0 is supervised (off-policy) distillation. Compare eval vs lmbda=1.0
# on the SAME teacher to isolate the on-policy contribution.
GKDConfig(lmbda=0.0, beta=1.0)   # off-policy; expect exposure-bias gap vs lmbda=1.0

Failure modes¶

Assuming a strict teacher ceiling. The teacher bounds the direct labels available, not every downstream metric. Students and self-distilled models can improve through regularization or data effects; use RLVR when a verifiable reward provides information the teacher does not.
Tokenizer mismatch. Per-token logit alignment assumes a shared vocabulary. GKDTrainer now hard-errors at startup on a vocab_size mismatch (loud, not silent) and its error message names the fix: use GOLDTrainer for cross-tokenizer distillation. Equal vocab sizes with different tokenizers still corrupt the KL silently, so verify the tokenizer identity, not just the size.
Off-policy regression (lmbda too low). Dropping toward lmbda=0 reintroduces the exposure bias on-policy distillation exists to remove; keep it high unless generation cost forces otherwise.¹
Reverse-KL mode collapse. Aggressive reverse KL (beta→1) can over-concentrate the student on a few teacher modes, cutting output diversity; back off beta if generations degrade.
Prefix failure. A bimodal teacher mixture along a rollout fragments the per-token gradient; token-level loss truncation or reweighting cannot fix it. Use trajectory-level correction (TRD) instead of tuning the per-token loss further.⁸
Teacher memory/throughput. Holding a large teacher resident (or serving it) alongside the student is the dominant cost; under-provisioning the teacher starves the trainer.
OPD² proxy contamination. Teacher-to-base log-probability change mixes reasoning with format, style, instruction following, and safety post-training. Audit token categories and downstream behavior rather than naming the ratio a pure reasoning delta.
OPD² centering drift. Top-k truncation can change centered signs. The paper does not specify renormalization; treat the full-vocabulary result as the reference until the approximation matches on canaries.
OPD² artifact gap. The official repository has no code, configs, tests, checkpoints, data manifest, or license. A local implementation is a reimplementation, not a reproduction.¹⁰
SEED shared blind spots. The actor and analyzer share weights, so a wrong trajectory interpretation or prompt-injected observation can be distilled into the policy. Format validation does not establish skill correctness.
SEED gate floor. Sigmoid gates remain positive. An opposed skill view reduces a sampled token's reinforcement but cannot reverse it; poor actions must be displaced through exploration and reward.
SEED recipe drift. Paper and public launcher update counts and KL coefficients differ. Archive the resolved config and do not label a public-default run an exact paper reproduction.¹³
Mistaking every variant for RL. GKD and OPD² use on-policy rollouts but optimize teacher-derived token signals, not environment return. SEED explicitly combines its token auxiliary with GRPO and therefore inherits both distillation and agentic-RL failure modes.

References¶

On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD, ICLR 2024): https://arxiv.org/abs/2306.13649
MiniLLM: On-Policy Distillation of Large Language Models (reverse KL, ICLR 2024): https://arxiv.org/abs/2306.08543
On-Policy Distillation (Thinking Machines Lab, 2025): https://thinkingmachines.ai/blog/on-policy-distillation/
On-Policy Delta Distillation (OPD²): https://arxiv.org/abs/2607.15161
OPD² official repository: https://github.com/naver-ai/opd2
SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning: https://arxiv.org/abs/2607.14777
SDAR: Self-Distilled Agentic Reinforcement Learning: https://arxiv.org/abs/2605.15155
SDAR reference implementation: https://github.com/ZJU-REAL/SDAR
SEED project page: https://jinyangwu.github.io/seed/
SEED source repository at the audited commit: https://github.com/jinyangwu/SEED/tree/2cf2fadca3c5aba28da68e8e1405182ba8d90e6c
A Reduction of Imitation Learning to No-Regret Online Learning (DAgger, AISTATS 2011): https://arxiv.org/abs/1011.0686
Qwen3 Technical Report (strong-to-weak distillation): https://arxiv.org/abs/2505.09388
TRL Generalized Knowledge Distillation (GKD) Trainer: https://huggingface.co/docs/trl/gkd_trainer
Self-Distilled Reasoner: On-Policy Self-Distillation for LLMs (coins "OPSD"): https://arxiv.org/abs/2601.18734
A Survey of On-Policy Distillation for Large Language Models: https://arxiv.org/abs/2604.00626
Trajectory-Refined Distillation (TRD, prefix failure and trajectory-level correction): https://arxiv.org/abs/2606.08432

Agarwal et al., GKD: trains the student on its self-generated sequences with teacher feedback; lambda interpolates off-policy (0) to on-policy (1); generalized JSD spans forward KL to reverse KL; on-policy (high lambda) performs better and optimal beta is task-dependent. https://arxiv.org/abs/2306.13649 ↩↩↩↩↩↩
Gu et al., MiniLLM: replaces forward KLD with reverse KLD (mode-seeking, better for generative LMs) and derives an on-policy optimization approach, reducing exposure bias. https://arxiv.org/abs/2306.08543 ↩↩↩
Lu et al. (Thinking Machines Lab), On-Policy Distillation: sample trajectories from the student and grade each token with a teacher (reverse KL); per-token supervision across N positions; roughly 7 to 10x fewer gradient steps and 50 to 100x less compute than RL, 9 to 30x cheaper than off-policy distillation; AIME'24 74.4% (on-policy) vs 67.6% (RL) vs 55.0% (off-policy). https://thinkingmachines.ai/blog/on-policy-distillation/ ↩↩↩↩↩↩
Ross, Gordon, Bagnell, DAgger: sequential prediction must train under the distribution of states the policy itself induces to avoid compounding error; the imitation-learning root of "on-policy". https://arxiv.org/abs/1011.0686 ↩
Qwen3 Technical Report: strong-to-weak distillation for smaller models "significantly outperforms reinforcement learning in performance and training efficiency", using only ~1/10 of the GPU hours of the multi-stage RL pipeline. https://arxiv.org/abs/2505.09388 ↩↩
TRL GKD Trainer: GKDTrainer wraps SFTTrainer with a teacher_model (a model object or a checkpoint id); GKDConfig.lmbda sets the on-policy student-data fraction, beta interpolates forward KL (0.0) to reverse KL (1.0). For self-distillation, pass the same checkpoint as model and teacher_model; the docs also describe a teacher_model_name_or_path=None shortcut (verify on your installed version). https://huggingface.co/docs/trl/gkd_trainer ↩
"On-policy self-distillation (OPSD)" is a narrow 2026 label for the self-teacher case (e.g. a privileged-context copy of the same model), not a standard name for on-policy distillation in general; the umbrella term remains "on-policy distillation (OPD)". https://arxiv.org/abs/2601.18734 ↩↩
Trajectory-Refined Distillation (2026): identifies "prefix failure" in on-policy distillation, where dense per-token supervision induces a bimodal teacher mixture and fragmented gradients that token-level loss truncation or reweighting fail to address; proposes revising the student's rollout under teacher guidance at the trajectory level, within on-policy support, instead of intervening on the per-token loss. Applies to on-policy distillation and to the OPSD privileged-context self-teacher case. https://arxiv.org/abs/2606.08432 ↩↩
Heo et al., On-Policy Delta Distillation, arXiv 2607.15161 v1. OPD² centers conventional OPD and teacher-to-base log-probability rewards under the student distribution, then retains delta advantages whose sign agrees with conventional OPD. Results cover four students, seven model/mode configurations, and math, code, and science aggregates. They are single-run point estimates; Gemma code regresses from 55.2 to 49.5 versus the original model. Table 5 reports Qwen3-1.7B thinking Science as 43.4 while Table 8 reports 43.5 for the corresponding full setting. https://arxiv.org/abs/2607.15161 ↩↩↩↩↩↩
The official OPD² repository at audited commit 76bc984bca982240e3ff46372366f5a0ab58fa03 contains one README saying code and training recipes will be released later. It has no implementation, configs, tests, dependency lock, checkpoints, sampled-data manifest, or repository license. https://github.com/naver-ai/opd2/commit/76bc984bca982240e3ff46372366f5a0ab58fa03 ↩↩↩↩
Wu et al., SEED, arXiv 2607.14777 v1. Main results use Qwen2.5-3B/7B and Qwen3-1.7B on ALFWorld, seven-dataset Search QA, and WebShop. The paper reports 150 updates, eight trajectories per task, beta_opd=5, lambda_opd=0.01, reference KL 0.01, and eight A800 80GB GPUs. Results have no training seeds, repeats, confidence intervals, runtime, memory, or cost comparison. https://arxiv.org/abs/2607.14777 ↩↩↩↩↩↩↩↩
Lu et al., SDAR (Self-Distilled Agentic Reinforcement Learning), arXiv 2605.15155. Gate definitions and the detached token loss are from the paper's method section; the three gate variants are entropy gating, gap gating, and soft-OR gating, with the gate detached via a stop-gradient in all cases. Reported gains are over GRPO on ALFWorld, Search-QA, and WebShop across the Qwen2.5 and Qwen3 families. The first author is also an author of SEED and the gap gate matches SEED's functional form, so the two papers are one line of work rather than independent replications. Repository metadata checked 2026-07-30: ZJU-REAL/SDAR, Python, Apache-2.0, 316 stars, last push 2026-07-30. The implementation was not run on this host. https://arxiv.org/abs/2605.15155 ↩↩
SEED source at audited commit 2cf2fadca3c5aba28da68e8e1405182ba8d90e6c. The implementation in verl/trainer/ppo/core_algos.py matches the masked, detached, sigmoid-gated sampled-token loss. Public ALFWorld and WebShop launchers default to 160 rather than the paper's 150 updates; Search uses KL 0.001 rather than the paper table's 0.01. Dependencies are broadly unpinned, and exact SFT data and task-ID manifests are absent. Source was inspected; official PyTorch tests and GPU training were not run here. https://github.com/jinyangwu/SEED/tree/2cf2fadca3c5aba28da68e8e1405182ba8d90e6c ↩↩↩↩