Markdown

RL with rich feedback: self-distillation policy optimization¶

Scope: what to do when a verifiable environment returns more than a number. A failing test suite emits a stack trace; a judge emits a critique; a compiler emits an error with a line number. Reinforcement learning with verifiable rewards throws all of that away and keeps a scalar, creating a credit-assignment bottleneck. This page covers the alternative: treat the model conditioned on that feedback as its own teacher and distill the resulting shift back into the policy. The scalar-reward baseline is in RLVR and GRPO; distillation from an external teacher is in on-policy distillation.

Source: Hubotter et al., "Reinforcement Learning via Self-Distillation" (arXiv 2601.20802, submitted 2026-01-28, revised 2026-02-16), which introduces Self-Distillation Policy Optimization (SDPO). Reported results cover scientific reasoning, tool use, and competitive programming on LiveCodeBench v6. Nothing was reproduced here: no training run was performed. The Python block is this page's own validation of the mechanisms the paper specifies, executed and asserted on a CPU host.

flowchart TB
  ROLL["Policy generates a rollout"] --> ENV["Environment returns rich feedback<br/>(runtime error, judge critique, test output)"]
  ENV --> TEACH["Self-teacher = same model, feedback in context"]
  ROLL --> TEACH
  TEACH --> SCORE["Re-score the ORIGINAL rollout token by token<br/>(one forward pass, no sampling)"]
  SCORE --> DELTA["Per-token disagreement = dense learning signal"]
  DELTA --> UPD["Distill back into the policy (JS divergence, top-K)"]
  UPD --> ROLL

What it is¶

RLVR gives one scalar per attempt. Every token in a 4,000-token rollout receives the same credit, so the model learns that the attempt was bad without learning which decision made it bad. Many verifiable environments already explain the failure in text, and that text is discarded.

SDPO formalizes this as reinforcement learning with rich feedback. The key move is what the teacher is and what it does:

The teacher is the current policy itself, conditioned on the feedback in context. There is no external teacher model and no explicit reward model.
The teacher does not generate anything. It re-computes the log-probabilities of the original attempt under the feedback-augmented context. Including the feedback transforms the next-token distribution, so the self-teacher agrees or disagrees with the student's original choices at specific token positions.

That second point is what makes the method cheap. Sampling a fresh teacher response would double the generation cost, and generation is the expensive, sequential part of an RL step. Re-scoring an existing sequence is one parallel forward pass.

The per-token disagreement is the dense signal. Where the feedback-informed model would have written something different, the gradient is large; where it agrees, the gradient is small. This is credit assignment derived from the environment's own explanation of the failure, rather than from a scalar.

Three implementation details make it work in practice:

Top-K distillation. Computing the full KL between student and teacher would require holding both models' complete logits in memory. SDPO keeps the top-K student logits, the teacher's values at those indices, and one term capturing the tail probability. The paper suggests K=100.
Symmetric Jensen-Shannon divergence for the distillation loss, rather than a directional KL. The same choice has been reported to improve stability in on-policy distillation from external teachers.
A regularized self-teacher, implemented either as an exponential moving average of the student parameters or by interpolating the current teacher with the initial teacher.

The method also applies where feedback is only scalar, by using successful rollouts as implicit feedback for failed attempts, and it can be applied to a single question at test time to accelerate discovery on hard binary-reward tasks.

Why it helps¶

The reported gains are about sample efficiency rather than a new capability ceiling. SDPO is reported to reach an improved GRPO's final accuracy in 4x fewer generations on LiveCodeBench v6 with Qwen3-8B, and at test time to reach the same discovery probability as best-of-k sampling or multi-turn conversation with 3x fewer attempts.

The mechanism for that efficiency is worth stating plainly: generation is the scarce resource in an RL loop, and SDPO extracts more supervision per generated token without generating more. Every other component is a way of keeping that extraction affordable.

When to use it (and when not)¶

Use SDPO when:

Your environment already emits text explaining failures. Compiler errors, test output, stack traces, judge rationales, tool error messages. If you are discarding that text today, this is the argument for keeping it.
Generation is your bottleneck. The overhead is one forward pass per rollout, which the executed model below puts at well under 1% of a generation.
Credit assignment is the problem. Long rollouts with a single terminal reward are the pathological case SDPO targets.

Do not use it when:

The model cannot use the feedback in context. The whole method rests on the model being able to retrospectively identify its own mistakes when told about them. If conditioning on the feedback does not change the distribution, there is no signal, and the paper's own framing is that SDPO leverages exactly this in-context ability.
Feedback is adversarial or wrong. A misleading error message becomes a confident dense training signal.
You need the guarantees of an external teacher. The self-teacher is only as good as the current policy plus the feedback.

How to validate the mechanisms¶

The three claims worth checking before adopting SDPO are that top-K is a safe approximation, that the symmetric loss is doing something a KL would not, and that the overhead really is small. Run: python3 sdpo.py.

import numpy as np

rng = np.random.default_rng(7)
V = 128_000  # vocabulary


def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)


# Student and feedback-conditioned self-teacher over the same next-token slot.
# A real LLM next-token distribution is sharply peaked, so model it as Zipf-like
# rather than Gaussian over logits: top-100 holds about 99.4% of the mass here.
rank = np.arange(V, dtype=float)
base = -2.0 * np.log(rank + 1.0)
rng.shuffle(base)                     # token id order is arbitrary
# The teacher differs from the student on a handful of PLAUSIBLE tokens: that is
# what "the feedback changed my mind about this continuation" looks like.
head = np.argsort(base)[::-1][:200]   # tokens the student already finds likely
shift = np.zeros(V)
shift[rng.choice(head, size=40, replace=False)] = rng.normal(0, 2.5, size=40)
p_student = softmax(base)
p_teacher = softmax(base + shift)
print(f"[0] student top-1 {np.sort(p_student)[::-1][0]:.3f}, "
      f"top-100 mass {np.sort(p_student)[::-1][:100].sum():.4f} (sharply peaked, as an LLM is)")


def kl(p, q):
    m = p > 0
    return float(np.sum(p[m] * np.log(p[m] / q[m])))


def js(p, q):
    m = 0.5 * (p + q)
    return 0.5 * kl(p, m) + 0.5 * kl(q, m)


# --- 1. Top-K distillation: does K=100 capture the divergence? ------------
# Full-vocabulary KL needs both models' complete logits in memory. SDPO keeps
# the top-K student logits, the teacher's values at those same indices, and a
# single tail term for everything else.
def topk_kl(p, q, K):
    idx = np.argpartition(-p, K)[:K]
    p_head, q_head = p[idx], q[idx]
    p_tail, q_tail = 1.0 - p_head.sum(), 1.0 - q_head.sum()
    out = float(np.sum(p_head * np.log(p_head / q_head)))
    if p_tail > 1e-12 and q_tail > 1e-12:
        out += p_tail * np.log(p_tail / q_tail)   # tail lumped into one bucket
    return out


exact = kl(p_student, p_teacher)
print(f"[1] full-vocabulary KL over {V:,} tokens = {exact:.6f}")
for K in (10, 100, 1_000):
    approx = topk_kl(p_student, p_teacher, K)
    head_mass = np.sort(p_student)[::-1][:K].sum()
    print(f"[1] K={K:>5}: KL {approx:.6f}  ({100 * approx / exact:6.2f}% of exact), "
          f"student head mass {100 * head_mass:6.3f}%, "
          f"logits held {200 * K / V:.2f}% of full")
assert abs(topk_kl(p_student, p_teacher, 100) - exact) / exact < 0.10, \
    "K=100 must capture most of the divergence"
assert topk_kl(p_student, p_teacher, 1_000) >= topk_kl(p_student, p_teacher, 10), \
    "more head mass captures more divergence"
# Memory is the point: K=100 stores 200 numbers per position instead of 256,000.
print(f"[1] memory per position: {2 * 100} values at K=100 vs {2 * V:,} for full logits "
      f"({V / 100:.0f}x reduction)")

# Adversarial: top-K is NOT safe when the teacher moves mass onto tokens the
# STUDENT ranks low. Those indices never enter the head, so the change hides
# inside the single tail bucket.
sneaky = np.zeros(V)
low_rank = np.argsort(p_student)[:50]          # tokens the student finds unlikely
sneaky[low_rank] = 9.0                          # feedback says: actually, these
p_teacher2 = softmax(base + sneaky)
exact2 = kl(p_student, p_teacher2)
approx2 = topk_kl(p_student, p_teacher2, 100)
print(f"[1] adversarial case (teacher promotes student-low-rank tokens): "
      f"exact {exact2:.6f} vs K=100 {approx2:.6f} ({100 * approx2 / exact2:.2f}% captured)")
assert approx2 < exact2, "top-K on the student's head under-counts teacher-only mass"

# --- 2. Why the symmetric Jensen-Shannon divergence ----------------------
print()
fwd = kl(p_teacher, p_student)
rev = kl(p_student, p_teacher)
print(f"[2] forward KL(teacher||student) = {fwd:.6f}")
print(f"[2] reverse KL(student||teacher) = {rev:.6f}  (asymmetric: differ by "
      f"{100 * abs(fwd - rev) / max(fwd, rev):.1f}%)")
print(f"[2] Jensen-Shannon             = {js(p_student, p_teacher):.6f}")
assert not np.isclose(fwd, rev), "KL is asymmetric, which is why the choice matters"
assert np.isclose(js(p_student, p_teacher), js(p_teacher, p_student)), "JS is symmetric"
# JS is bounded by log 2 regardless of how far apart the two get. KL is not,
# so a single confidently-wrong token can dominate a KL-based loss.
far = softmax(base + 40.0 * shift)
print(f"[2] pushed far apart: KL {kl(p_student, far):9.4f} vs JS "
      f"{js(p_student, far):.6f} (JS bounded by log 2 = {np.log(2):.6f})")
assert js(p_student, far) <= np.log(2) + 1e-9, "JS cannot exceed log 2"
assert kl(p_student, far) > np.log(2), "KL is unbounded and here exceeds the JS ceiling"

# --- 3. EMA self-teacher: bounding how fast the target can move ----------
# The self-teacher is the student itself, so without regularization the target
# chases the policy and the loop can run away.
print()


def ema_trace(steps: int, decay: float, drift: float = 1.0) -> np.ndarray:
    student = np.arange(steps, dtype=float) * drift    # student drifts linearly
    teacher = np.zeros(steps)
    for t in range(1, steps):
        teacher[t] = decay * teacher[t - 1] + (1 - decay) * student[t]
    return student - teacher


for decay in (0.0, 0.9, 0.99):
    lag = ema_trace(400, decay)[-1]
    label = "no regularization (teacher == student)" if decay == 0.0 else f"EMA decay {decay}"
    print(f"[3] {label:40s} -> steady-state teacher lag {lag:7.2f} steps")
assert ema_trace(400, 0.0)[-1] == 0.0, "an unregularized self-teacher is the student"
assert ema_trace(400, 0.99)[-1] > ema_trace(400, 0.9)[-1], "higher decay lags further"
# The lag is what stops the target from moving with the policy. Zero lag means
# the loss can be driven to zero without learning anything.
print("[3] zero lag means the distillation target is the policy itself, so the loss "
      "can go to zero without the policy improving")

# --- 4. What SDPO actually costs versus GRPO ----------------------------
# The self-teacher does NOT generate. It re-scores the existing rollout under a
# feedback-augmented context: one forward pass, no sampling.
print()
ROLLOUT_TOKENS = 4_096
PREFILL_TOK_PER_S = 40_000.0     # scoring is prefill-bound and parallel
DECODE_TOK_PER_S = 90.0          # generation is sequential

gen_s = ROLLOUT_TOKENS / DECODE_TOK_PER_S
score_s = ROLLOUT_TOKENS / PREFILL_TOK_PER_S
print(f"[4] generating a {ROLLOUT_TOKENS:,}-token rollout : {gen_s:8.2f} s")
print(f"[4] re-scoring the same rollout (1 forward): {score_s:8.2f} s "
      f"({gen_s / score_s:.0f}x cheaper)")
print(f"[4] SDPO step overhead vs GRPO = {100 * score_s / gen_s:.2f}% of one generation")
assert score_s < gen_s / 100, "re-scoring must be far cheaper than sampling"
# Contrast with an approach that samples a fresh teacher response: that would
# double generation cost, which is the expensive axis.
print(f"[4] an external teacher that SAMPLES instead would add {gen_s:.2f} s, "
      f"{gen_s / score_s:.0f}x the re-scoring cost")

print("\nAll assertions passed.")

Executed output:

[0] student top-1 0.608, top-100 mass 0.9940 (sharply peaked, as an LLM is)
[1] full-vocabulary KL over 128,000 tokens = 1.479516
[1] K=   10: KL 1.409493  ( 95.27% of exact), student head mass 94.215%, logits held 0.02% of full
[1] K=  100: KL 1.468243  ( 99.24% of exact), student head mass 99.396%, logits held 0.16% of full
[1] K= 1000: KL 1.479516  (100.00% of exact), student head mass 99.940%, logits held 1.56% of full
[1] memory per position: 200 values at K=100 vs 256,000 for full logits (1280x reduction)
[1] adversarial case (teacher promotes student-low-rank tokens): exact 0.000015 vs K=100 0.000000 (0.12% captured)

[2] forward KL(teacher||student) = 3.374241
[2] reverse KL(student||teacher) = 1.479516  (asymmetric: differ by 56.2%)
[2] Jensen-Shannon             = 0.383967
[2] pushed far apart: KL  280.5201 vs JS 0.693053 (JS bounded by log 2 = 0.693147)

[3] no regularization (teacher == student)   -> steady-state teacher lag    0.00 steps
[3] EMA decay 0.9                            -> steady-state teacher lag    9.00 steps
[3] EMA decay 0.99                           -> steady-state teacher lag   97.20 steps
[3] zero lag means the distillation target is the policy itself, so the loss can go to zero without the policy improving

[4] generating a 4,096-token rollout :    45.51 s
[4] re-scoring the same rollout (1 forward):     0.10 s (444x cheaper)
[4] SDPO step overhead vs GRPO = 0.23% of one generation
[4] an external teacher that SAMPLES instead would add 45.51 s, 444x the re-scoring cost

All assertions passed.

Four things follow.

Top-K distillation is sound, for the reason that makes LLM distributions special. At K=100 the approximation captures 99.24% of the full-vocabulary KL while holding 200 values per position instead of 256,000, a 1,280x memory reduction. That works because a real next-token distribution is sharply peaked: the top 100 tokens hold 99.4% of the mass in the modelled distribution. This result is not robust to the shape of the distribution. An earlier version of this model used Gaussian logits over the vocabulary, which is far flatter than any real LLM, and under that distribution K=100 captured essentially none of the divergence. If you apply top-K distillation at high sampling temperature or to a deliberately flattened distribution, re-measure before trusting K=100.

Top-K has a specific blind spot, and it is the one that matters here. The adversarial case promotes tokens the student ranks low, which is exactly what strong corrective feedback should do: the whole point is that the model was wrong, so the right token may not be in its head. Those indices never enter the top-K set, so the change collapses into the single tail bucket and only 0.12% of the divergence is captured. Top-K measures how the teacher redistributes mass within what the student already considered. Budget for a larger K, or monitor the tail term, when feedback is expected to be strongly corrective.

The Jensen-Shannon choice is about bounded loss, not elegance. Forward and reverse KL differ by 56.2% on the same pair, so the direction is a real modelling decision. More importantly, when the two distributions are pushed far apart the KL reaches 280.5 while the JS is capped at log 2, which it approaches at 0.693053. In a self-distillation loop where the teacher is the policy plus feedback, a single confidently-wrong token can dominate a KL-based loss; JS cannot be dominated that way.

Self-teacher regularization is not optional. With no regularization the teacher lag is exactly zero, because the teacher is the student. The distillation loss can then be driven to zero without the policy improving at all. EMA introduces lag (9 steps at decay 0.9, 97 at 0.99) and that lag is what makes the target a target.

How to run it¶

Keep the raw feedback, not a summary of it. The dense signal comes from the model conditioning on the specific error. A normalized error category carries far less information than the stack trace.
Score, do not sample. If your implementation generates a teacher response instead of re-scoring the original, you have paid for an external-teacher method and kept a self-teacher's quality. The executed model puts that at roughly 444x the re-scoring cost under its assumptions.
Start with K=100 and verify against a full-vocabulary reference on a sample. Cheap to check once, and the blind spot above is real.
Regularize the self-teacher from the first step, via EMA or interpolation with the initial teacher. Do not add it after observing instability; by then the run has already collapsed toward a degenerate target.
Use a micro batch large enough to amortize the extra forward pass. The paper notes its own timing used a micro batch size of 2 and that larger sizes reduce compute time further.
Treat scalar-only environments as a fallback mode, not the main case. Using successful rollouts as implicit feedback for failed ones is reported to work, but it is a weaker signal than an actual error message.

How to maintain it¶

Re-check the top-K fraction when sampling temperature changes. Temperature flattens the distribution and directly degrades the approximation.
Monitor teacher-student agreement as a health metric. If the self-teacher stops disagreeing with the policy, either the policy has learned or the feedback has stopped being informative. Those two look identical in the loss and different in evaluation.
Re-validate when the feedback format changes. A new test harness that prints differently changes the teacher's conditioning.
Watch for feedback leakage into evaluation. The self-teacher sees the answer's consequences; the deployed policy does not.

Failure modes¶

Degenerate target from an unregularized self-teacher. Loss goes to zero, metrics do not move.
Top-K blindness to corrective feedback. The strongest corrections are the ones most likely to be missed, because they promote tokens the student ranked low.
Unbounded loss from a directional KL. One confidently-wrong token dominates the gradient.
Sampling the teacher instead of re-scoring. Doubles the expensive axis and removes the method's main advantage.
Garbage feedback becoming confident supervision. Wrong error messages train the wrong lesson densely rather than sparsely, which is worse than a scalar.
Temperature drift silently breaking the approximation. No error is raised; the captured divergence just falls.

Open questions and validation¶

Nothing here was reproduced. The 4x-fewer-generations and 3x-fewer-attempts figures are the authors' own, on their benchmarks.
The executed model validates the mechanisms under a synthetic but realistically peaked distribution. It is not a measurement of SDPO.
Whether SDPO and GRPO compose is examined in the paper's own Section 4.5 and was not evaluated here.
Whether the method degrades gracefully when feedback quality varies within a single environment is not addressed by the source.
The timing figures in section 4 of the model use assumed prefill and decode rates; substitute your own before using the overhead number.

References¶

Hubotter et al., "Reinforcement Learning via Self-Distillation", arXiv 2601.20802: https://arxiv.org/abs/2601.20802
Agarwal et al., "On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes" (source of the stability argument for a symmetric divergence), arXiv 2306.13649: https://arxiv.org/abs/2306.13649
LiveCodeBench, the competitive-programming benchmark used: https://livecodebench.github.io/