Copy for agents Markdown

RL with rich feedback: self-distillation policy optimization

Scope: what to do when a verifiable environment returns more than a number. A failing test suite emits a stack trace; a judge emits a critique; a compiler emits an error with a line number. Reinforcement learning with verifiable rewards throws all of that away and keeps a scalar, creating a credit-assignment bottleneck. This page covers the alternative: treat the model conditioned on that feedback as its own teacher and distill the resulting shift back into the policy. The scalar-reward baseline is in RLVR and GRPO; distillation from an external teacher is in on-policy distillation.

Source: Hubotter et al., "Reinforcement Learning via Self-Distillation" (arXiv 2601.20802, submitted 2026-01-28, revised 2026-02-16), which introduces Self-Distillation Policy Optimization (SDPO). Reported results cover scientific reasoning, tool use, and competitive programming on LiveCodeBench v6. Nothing was reproduced here: no training run was performed. The Python block is this page's own validation of the mechanisms the paper specifies, executed and asserted on a CPU host.

Source audit: official repository commit 7c457fc1b1f636ae794eb0362ba37d4743b06fbc was inspected. It builds separate reprompted teacher inputs, evaluates them without gradients, gathers teacher logits at the student's top-K indices, optionally appends a tail bucket, and updates a separate EMA teacher with the documented update-rate convention. The executable block below is a numpy model of those semantics, not an execution of the GPU training stack.

What it is

RLVR gives one scalar per attempt. Every token in a 4,000-token rollout receives the same credit, so the model learns that the attempt was bad without learning which decision made it bad. Many verifiable environments already explain the failure in text, and that text is discarded.

SDPO formalizes this as reinforcement learning with rich feedback. The key move is what the teacher is and what it does:

• The teacher is the current policy itself, conditioned on the feedback in context. There is no external teacher model and no explicit reward model.
• The teacher does not generate anything. It re-computes the log-probabilities of the original attempt under the feedback-augmented context. Including the feedback transforms the next-token distribution, so the self-teacher agrees or disagrees with the student's original choices at specific token positions.

That second point is what makes the method cheap in the regime it was designed for. Sampling a fresh teacher response would double the generation cost, and generation is the expensive, sequential part of an RL step. Re-scoring an existing sequence is one parallel forward pass, which the paper describes as "substantially faster than sequential generation" with a "relatively small" measured compute overhead.

The qualification is in the paper's own limitations, and it is a ratio argument rather than an absolute one. The overhead is small because generation dominates the step. Where it does not, the ratio inverts: the extra log-prob pass "may be a larger overhead for smaller models with shorter generation lengths, where generation time is comparatively small." So the cost story holds for large models emitting long traces and weakens as either of those shrinks. Measure the log-prob pass against your own generation time before assuming it is free; the reported figures also use a micro batch size of 2, and the paper notes compute time falls further with larger micro batches.

The per-token disagreement is the dense signal. Where the feedback-informed model would have written something different, the gradient is large; where it agrees, the gradient is small. This is credit assignment derived from the environment's own explanation of the failure, rather than from a scalar.

Three implementation details make it work in practice:

• Top-K distillation. Computing a full-vocabulary divergence requires holding both models' complete logits in memory. SDPO keeps the top-K student logits, the teacher's values at those indices, and one term capturing the tail probability. The reported experiments use K=100 for the scalar-feedback setting and K=20 for rich-feedback and test-time training.
• A divergence choice separate from top-K. Equation 1 is reverse KL. The reported scalar-feedback experiments use symmetric Jensen-Shannon divergence, while the rich-feedback and test-time configurations use reverse KL. Top-K is a memory approximation applied after choosing that loss.
• A regularized self-teacher, implemented either as an exponential moving average of the student parameters or by interpolating the current teacher with the initial teacher.

The method also applies where feedback is only scalar, by using successful rollouts as implicit feedback for failed attempts, and it can be applied to a single question at test time to accelerate discovery on hard binary-reward tasks.

Why use it

The reported gains are about sample efficiency rather than a new capability ceiling. SDPO is reported to reach an improved GRPO's final accuracy in 4x fewer generations on LiveCodeBench v6 with Qwen3-8B, and at test time to reach the same discovery probability as best-of-k sampling or multi-turn conversation with 3x fewer attempts.

The mechanism for that efficiency is worth stating plainly: generation is the scarce resource in an RL loop, and SDPO extracts more supervision per generated token without generating more. Every other component is a way of keeping that extraction affordable.

When to use it (and when not)

Use SDPO when:

• Your environment already emits text explaining failures. Compiler errors, test output, stack traces, judge rationales, tool error messages. If you are discarding that text today, this is the argument for keeping it.
• Generation is your bottleneck. SDPO adds feedback-conditioned log-probability computation without adding another sampled response. The paper reports relatively small step-time overhead in its measured setup; measure it on the target batch shape and hardware.
• Credit assignment is the problem. Long rollouts with a single terminal reward are the pathological case SDPO targets.

Do not use it when:

• The model cannot use the feedback in context. The whole method rests on the model being able to retrospectively identify its own mistakes when told about them. If conditioning on the feedback does not change the distribution, there is no signal, and the paper's own framing is that SDPO leverages exactly this in-context ability.
• Feedback is adversarial or wrong. A misleading error message becomes a confident dense training signal.
• You need the guarantees of an external teacher. The self-teacher is only as good as the current policy plus the feedback.

Architecture

flowchart TB
  ROLL["Student context: x<br/>sample original rollout y"] --> ENV["Environment returns feedback f<br/>(error, critique, test output)"]
  ROLL --> STUDENT["Student distribution<br/>p_theta(. | x, y_prefix)"]
  ENV --> TEACHER["Feedback-conditioned self-teacher<br/>q_theta_prime(. | x, f, y_prefix)"]
  ROLL --> TEACHER
  STUDENT --> DIV["Per-token divergence<br/>(reverse KL or JS)"]
  TEACHER --> DIV
  DIV --> TOPK["Optional top-K plus tail<br/>memory approximation"]
  TOPK --> UPDATE["Update student<br/>stop gradient through teacher"]
  UPDATE --> ROLL
  UPDATE -.-> EMA["EMA or trust-region<br/>updates teacher target"]
  EMA -.-> TEACHER

The two distributions differ in context even when theta_prime = theta. Teacher regularization controls how the feedback-conditioned target's parameters move over training; it does not create the student-teacher disagreement.

How to use it: validate the mechanisms

The claims worth checking before adopting SDPO are that feedback conditioning creates disagreement even at identical weights, that divergence choice is separate from top-K coarsening, that EMA controls parameter motion rather than creating the target, and that exposed overhead depends on measured scoring time and overlap. Run: python3 sdpo.py.

import numpy as np

rng = np.random.default_rng(7)
V = 128_000  # vocabulary


def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)


# Student and feedback-conditioned self-teacher over the same next-token slot.
# A real LLM next-token distribution is sharply peaked, so model it as Zipf-like
# rather than Gaussian over logits: top-100 holds about 99.4% of the mass here.
rank = np.arange(V, dtype=float)
base = -2.0 * np.log(rank + 1.0)
rng.shuffle(base)                     # token id order is arbitrary
# The teacher differs from the student on a handful of PLAUSIBLE tokens: that is
# what "the feedback changed my mind about this continuation" looks like.
head = np.argsort(base)[::-1][:200]   # tokens the student already finds likely
shift = np.zeros(V)
shift[rng.choice(head, size=40, replace=False)] = rng.normal(0, 2.5, size=40)
p_student = softmax(base)
p_teacher = softmax(base + shift)
print(f"[0] student top-1 {np.sort(p_student)[::-1][0]:.3f}, "
      f"top-100 mass {np.sort(p_student)[::-1][:100].sum():.4f} (sharply peaked, as an LLM is)")


def kl(p, q):
    m = p > 0
    return float(np.sum(p[m] * np.log(p[m] / q[m])))


def js(p, q):
    m = 0.5 * (p + q)
    return 0.5 * kl(p, m) + 0.5 * kl(q, m)


# --- 1. Top-K is a memory approximation, separate from divergence choice ---
# Coarsen the vocabulary into K student-head tokens plus one tail bucket, then
# evaluate the chosen divergence on that K+1-outcome distribution.
def topk_coarsen(p, q, K):
    idx = np.argpartition(-p, K)[:K]
    p_head, q_head = p[idx], q[idx]
    p_tail, q_tail = 1.0 - p_head.sum(), 1.0 - q_head.sum()
    return np.append(p_head, p_tail), np.append(q_head, q_tail)


def topk_divergence(p, q, K, divergence):
    p_coarse, q_coarse = topk_coarsen(p, q, K)
    return divergence(p_coarse, q_coarse)


exact = kl(p_student, p_teacher)
print(f"[1] full-vocabulary KL over {V:,} tokens = {exact:.6f}")
for K in (10, 100, 1_000):
    approx = topk_divergence(p_student, p_teacher, K, kl)
    head_mass = np.sort(p_student)[::-1][:K].sum()
    print(f"[1] K={K:>5}: KL {approx:.6f}  ({100 * approx / exact:6.2f}% of exact), "
          f"student head mass {100 * head_mass:6.3f}%, "
          f"logits held {200 * K / V:.2f}% of full")
kl100 = topk_divergence(p_student, p_teacher, 100, kl)
js100 = topk_divergence(p_student, p_teacher, 100, js)
assert 0 <= kl100 <= exact                         # data processing under coarsening
assert 0 <= js100 <= js(p_student, p_teacher)
assert abs(kl100 - exact) / exact < 0.10           # property of THIS synthetic pair
assert topk_divergence(p_student, p_teacher, 1_000, kl) >= \
       topk_divergence(p_student, p_teacher, 10, kl)
# Memory is the point: K=100 stores two K-token vectors plus two tail values.
held = 2 * (100 + 1)
print(f"[1] K=100 coarsened JS {js100:.6f} vs full JS {js(p_student, p_teacher):.6f}")
print(f"[1] memory per position: {held} values at K=100 vs {2 * V:,} for full logits "
      f"({2 * V / held:.0f}x reduction, excluding indices)")

# Adversarial: redistribute a FIXED amount of probability within the student's
# tail. The K+1 coarsening sees identical head tokens and identical total tail
# mass, so it cannot see the changed distribution inside that bucket.
head100 = np.argpartition(-p_student, 100)[:100]
tail_mask = np.ones(V, dtype=bool)
tail_mask[head100] = False
low_rank = np.argsort(p_student)[:50]
p_teacher2 = p_student.copy()
tail_mass = p_student[tail_mask].sum()
p_teacher2[tail_mask] *= 0.001
p_teacher2[low_rank] += 0.999 * tail_mass / len(low_rank)
assert np.isclose(p_teacher2.sum(), 1.0)
exact2 = kl(p_student, p_teacher2)
p2_coarse, q2_coarse = topk_coarsen(p_student, p_teacher2, 100)
approx2 = kl(p2_coarse, q2_coarse)
exact2_fwd, approx2_fwd = kl(p_teacher2, p_student), kl(q2_coarse, p2_coarse)
exact2_js, approx2_js = js(p_student, p_teacher2), js(p2_coarse, q2_coarse)
print(f"[1] low-rank promotion reverse KL: full {exact2:.6f}, coarsened {approx2:.6f}")
print(f"[1] low-rank promotion forward KL: full {exact2_fwd:.6f}, coarsened {approx2_fwd:.6f}")
print(f"[1] low-rank promotion JS:         full {exact2_js:.6f}, coarsened {approx2_js:.6f}")
assert 0 <= approx2 <= exact2
assert 0 <= approx2_fwd <= exact2_fwd
assert 0 <= approx2_js <= exact2_js
assert max(approx2, approx2_fwd, approx2_js) < 1e-12

# --- 2. Why the symmetric Jensen-Shannon divergence ----------------------
print()
fwd = kl(p_teacher, p_student)
rev = kl(p_student, p_teacher)
print(f"[2] forward KL(teacher||student) = {fwd:.6f}")
print(f"[2] reverse KL(student||teacher) = {rev:.6f}  (asymmetric: differ by "
      f"{100 * abs(fwd - rev) / max(fwd, rev):.1f}%)")
print(f"[2] Jensen-Shannon             = {js(p_student, p_teacher):.6f}")
assert not np.isclose(fwd, rev), "KL is asymmetric, which is why the choice matters"
assert np.isclose(js(p_student, p_teacher), js(p_teacher, p_student)), "JS is symmetric"
# JS is bounded by log 2 regardless of how far apart the two get. KL is not,
# so a single confidently-wrong token can dominate a KL-based loss.
far = softmax(base + 40.0 * shift)
print(f"[2] pushed far apart: KL {kl(p_student, far):9.4f} vs JS "
      f"{js(p_student, far):.6f} (JS bounded by log 2 = {np.log(2):.6f})")
assert js(p_student, far) <= np.log(2) + 1e-9, "JS cannot exceed log 2"
assert kl(p_student, far) > np.log(2), "KL is unbounded and here exceeds the JS ceiling"

# --- 3. Same weights do not imply the same distribution -------------------
# p_student and p_teacher above use the same base parameters; the feedback
# context creates the shift. EMA controls parameter motion, not this distinction.
print()
same_weight_js = js(p_student, p_teacher)
assert same_weight_js > 0
print(f"[3] same weights, different contexts -> JS {same_weight_js:.6f} (non-zero)")


def ema_parameter_lag(steps: int, update_rate: float, drift: float = 1.0) -> np.ndarray:
    student = np.arange(steps, dtype=float) * drift    # student drifts linearly
    teacher = np.zeros(steps)
    for t in range(1, steps):
        teacher[t] = (1 - update_rate) * teacher[t - 1] + update_rate * student[t]
    return student - teacher


for update_rate in (1.0, 0.1, 0.01):
    lag = ema_parameter_lag(400, update_rate)[-1]
    print(f"[3] EMA update rate {update_rate:4.2f} -> parameter lag {lag:7.2f} steps")
assert ema_parameter_lag(400, 1.0)[-1] == 0.0
assert ema_parameter_lag(400, 0.01)[-1] > ema_parameter_lag(400, 0.1)[-1]
print("[3] zero parameter lag still leaves feedback-conditioned disagreement")

# --- 4. Cost sensitivity without invented hardware rates ------------------
# Exposed overhead depends on the measured baseline step time, teacher scoring
# time, and how much scoring overlaps other work.
print()
def exposed_overhead(baseline_step_s, teacher_score_s, overlapped_s=0.0):
    assert baseline_step_s > 0 and teacher_score_s >= 0 and overlapped_s >= 0
    return max(0.0, teacher_score_s - overlapped_s) / baseline_step_s


baseline = 1.0                 # normalized; not a hardware measurement
teacher_score = 0.20
for overlap in (0.0, 0.10, 0.20):
    overhead = exposed_overhead(baseline, teacher_score, overlap)
    print(f"[4] normalized teacher score {teacher_score:.2f}x, overlap {overlap:.2f}x "
          f"-> exposed step overhead {100 * overhead:4.1f}%")
assert exposed_overhead(1.0, 0.20, 0.0) == 0.20
assert exposed_overhead(1.0, 0.20, 0.10) == 0.10
assert exposed_overhead(1.0, 0.20, 0.20) == 0.0
print("[4] substitute measured step, scoring, and overlap times before capacity planning")

print("\nAll assertions passed.")

Executed output:

[0] student top-1 0.608, top-100 mass 0.9940 (sharply peaked, as an LLM is)
[1] full-vocabulary KL over 128,000 tokens = 1.479516
[1] K=   10: KL 1.409493  ( 95.27% of exact), student head mass 94.215%, logits held 0.02% of full
[1] K=  100: KL 1.468243  ( 99.24% of exact), student head mass 99.396%, logits held 0.16% of full
[1] K= 1000: KL 1.479516  (100.00% of exact), student head mass 99.940%, logits held 1.56% of full
[1] K=100 coarsened JS 0.380848 vs full JS 0.383967
[1] memory per position: 202 values at K=100 vs 256,000 for full logits (1267x reduction, excluding indices)
[1] low-rank promotion reverse KL: full 0.041752, coarsened 0.000000
[1] low-rank promotion forward KL: full 0.090503, coarsened 0.000000
[1] low-rank promotion JS:         full 0.004166, coarsened 0.000000

[2] forward KL(teacher||student) = 3.374241
[2] reverse KL(student||teacher) = 1.479516  (asymmetric: differ by 56.2%)
[2] Jensen-Shannon             = 0.383967
[2] pushed far apart: KL  280.5201 vs JS 0.693053 (JS bounded by log 2 = 0.693147)

[3] same weights, different contexts -> JS 0.383967 (non-zero)
[3] EMA update rate 1.00 -> parameter lag    0.00 steps
[3] EMA update rate 0.10 -> parameter lag    9.00 steps
[3] EMA update rate 0.01 -> parameter lag   97.20 steps
[3] zero parameter lag still leaves feedback-conditioned disagreement

[4] normalized teacher score 0.20x, overlap 0.00x -> exposed step overhead 20.0%
[4] normalized teacher score 0.20x, overlap 0.10x -> exposed step overhead 10.0%
[4] normalized teacher score 0.20x, overlap 0.20x -> exposed step overhead  0.0%
[4] substitute measured step, scoring, and overlap times before capacity planning

All assertions passed.

Four things follow.

Top-K is a coarsening, not a divergence. It replaces the vocabulary with the student's K head tokens plus one tail bucket, then evaluates reverse KL, forward KL, or JS on that smaller distribution. For the first synthetic pair, K=100 retains 99.24% of full reverse KL and produces JS 0.380848 against the full 0.383967 while storing 202 probability values instead of 256,000, excluding token indices. Those figures validate this pair only. By the data-processing inequality, coarsening cannot increase these divergences; closeness to the full loss must be measured.

Top-K cannot see redistribution within its tail bucket. The adversarial test preserves every head probability and the total tail mass, but moves that tail mass among low-ranked tokens. Full reverse KL is 0.041752, forward KL is 0.090503, and JS is 0.004166; all three coarsened values are zero. The small absolute JS reflects that the tail contains only about 0.6% of student mass. This isolates approximation loss, not evidence that every low-rank correction is large. Head mass alone is insufficient: compare the coarsened and full version of the divergence actually used by the run.

The Jensen-Shannon choice is about bounded loss, not elegance. Forward and reverse KL differ by 56.2% on the same pair, so the direction is a real modelling decision. More importantly, when the two distributions are pushed far apart the KL reaches 280.5 while the JS is capped at log 2, which it approaches at 0.693053. In a self-distillation loop where the teacher is the policy plus feedback, a single confidently-wrong token can dominate a KL-based loss; JS cannot be dominated that way.

Same weights do not mean zero loss. In the block, student and self-teacher share the same base logits but receive different contexts, producing non-zero JS of 0.383967. An EMA update rate of 1.0 gives zero parameter lag and still leaves that feedback-conditioned disagreement. Lower update rates make the teacher parameters move more slowly: 0.1 produces 9 steps of lag in the linear-drift model and 0.01 produces 97.2. The paper's motivation is to keep the bootstrapping teacher from diverging rapidly from its initial policy; its unregularized teacher underperforms the regularized variants and eventually diverges in the reported ablation.

How to develop with it

• Build the student context from (x, y_prefix) and the teacher context from (x, feedback, y_prefix). Stop gradients through the teacher distribution; sharing weights does not imply sharing activations or predictions.
• Select the divergence explicitly. Reproduce the full-vocabulary loss on a sample, then compare the K-plus-tail approximation for the same divergence before choosing K.
• Implement the teacher update separately from feedback conditioning. For EMA, the paper uses theta_teacher <- (1 - alpha) * theta_teacher + alpha * theta_student; the update rate is not a JS or KL coefficient.
• Record scoring time, baseline step time, and overlap separately. The normalized block shows the algebra: exposed overhead is max(0, score_time - overlap_time) / baseline_step_time.

How to run it in production

• Keep the raw feedback, not a summary of it. The dense signal comes from the model conditioning on the specific error. A normalized error category carries far less information than the stack trace.
• Score, do not sample. Generating a new teacher response adds sequential sampling that Algorithm 1 does not require. Re-score the original rollout under the feedback-augmented context.
• Start from the paper's setting-specific K, then verify it. The source uses 100 without rich feedback and 20 for rich-feedback and test-time runs. Compare against a full-vocabulary reference on the target model rather than treating either value as universal.
• Regularize the self-teacher from the first step, via EMA or interpolation with the initial teacher. The paper's unregularized variant eventually diverges; do not wait for that failure before enabling the control.
• Use a micro batch large enough to amortize the extra forward pass. The paper notes its own timing used a micro batch size of 2 and that larger sizes reduce compute time further.
• Treat scalar-only environments as a fallback mode, not the main case. Using successful rollouts as implicit feedback for failed ones is reported to work, but it is a weaker signal than an actual error message.

Continual training: what the forgetting evidence does and does not show

On-policy algorithms are attractive partly because models trained with them tend not to lose prior capabilities, which is what makes sequential multi-task pipelines viable without retraining from scratch. The paper tests this directly, evaluating final checkpoints on three holdout benchmarks the training task did not cover: IFEval (precise format-instruction following), ArenaHard-v2 (LLM-judged real-world instruction prompts), and MMLU-Pro (broad multi-task knowledge and reasoning).

The reported result is favourable to SDPO: it "learns the new task while mitigating degradation of initial capabilities, overall achieving a better performance-forgetting tradeoff than GRPO."

The failure in that same comparison belongs to the obvious alternative. An off-policy self-distillation baseline, supervised fine-tuning on the self-teacher's successful generations, fails on both axes at once. It "significantly underperforms SDPO" on the target benchmark and leads to "worse forgetting of prior capabilities", which the authors connect to prior findings on the instability of off-policy imitation. It also costs more: because you must generate from both student and teacher, it "requires 2x the generations of SDPO for the same number of steps."

That is the practical argument for keeping the teacher a scorer rather than a generator. The tempting simplification, let the feedback-conditioned model write good answers and fine-tune on those, is more expensive per step, scores worse on the task, and damages capabilities the run was not even training.

Two further limitations the authors state, both of which decide whether SDPO applies to you at all:

• It depends on the base model's in-context learning ability. The paper says SDPO is "primarily applicable for RL-training stronger base models, while it can underperform GRPO on weaker models." The mechanism explains why: if the model cannot exploit feedback in context, the self-teacher's re-scored distribution barely differs from the student's, and there is no signal.
• It depends on feedback quality. "If the environment provides uninformative or misleading feedback, a model may not be able to learn from it through SDPO." Misleading feedback is the worse case, because it produces a confident dense gradient pointing the wrong way.

How to maintain it

• Re-check the chosen divergence's top-K error when sampling temperature changes. Temperature can flatten the distribution, but head mass alone does not bound lost within-tail structure.
• Monitor teacher-student agreement as a health metric. If the self-teacher stops disagreeing with the policy, either the policy has learned or the feedback has stopped being informative. Those two look identical in the loss and different in evaluation.
• Re-validate when the feedback format changes. A new test harness that prints differently changes the teacher's conditioning.
• Watch for feedback leakage into evaluation. The self-teacher sees the answer's consequences; the deployed policy does not.

Failure modes

• Assuming the scoring pass is free at any scale. It is small relative to generation, which is the point, but the paper flags it may be a larger overhead for smaller models with shorter generations. Measure the ratio, do not inherit it.
• Swapping the scorer for a generator. SFT on the self-teacher's successes costs 2x the generations, underperforms on the target task, and forgets prior capabilities worse.
• Applying it to a weak base model. SDPO depends on in-context learning ability and can underperform GRPO where that is lacking.
• Unregularized teacher instability. The current-parameter teacher still differs through feedback context, but the moving target can drift during bootstrapping; the paper reports eventual divergence for this variant.
• Top-K blindness inside the tail. Changes among tokens outside the student's selected head disappear when the tail is represented by one aggregate probability.
• Unbounded loss from a directional KL. One confidently-wrong token dominates the gradient.
• Sampling the teacher instead of re-scoring. Adds an unnecessary sequential generation path and removes the method's no-extra-sampling advantage.
• Garbage feedback becoming confident supervision. Wrong error messages train the wrong lesson densely rather than sparsely, which is worse than a scalar.
• Temperature drift silently breaking the approximation. No error is raised; the captured divergence just falls.

Open questions and validation

• Nothing here was reproduced. The 4x-fewer-generations and 3x-fewer-attempts figures are the authors' own, on their benchmarks.
• The executed model validates the mechanisms under a synthetic but realistically peaked distribution. It is not a measurement of SDPO.
• Whether SDPO and GRPO compose is examined in the paper's own Section 4.5 and was not evaluated here.
• Whether the method degrades gracefully when feedback quality varies within a single environment is not addressed by the source.
• The cost block is an algebraic sensitivity model, not a timing estimate. Figure 5 reports relatively small overhead for the paper's setup with micro batch size 2; deployment sizing requires locally measured baseline, scoring, and overlap times.

References

• Hubotter et al., "Reinforcement Learning via Self-Distillation", arXiv 2601.20802: https://arxiv.org/abs/2601.20802
• Official SDPO source snapshot audited for this page: https://github.com/lasgroup/SDPO/tree/7c457fc1b1f636ae794eb0362ba37d4743b06fbc
• Reprompted teacher-batch construction: https://github.com/lasgroup/SDPO/blob/7c457fc1b1f636ae794eb0362ba37d4743b06fbc/verl/trainer/ppo/ray_trainer.py#L672-L795
• Student/teacher scoring path: https://github.com/lasgroup/SDPO/blob/7c457fc1b1f636ae794eb0362ba37d4743b06fbc/verl/workers/actor/dp_actor.py#L808-L845
• EMA teacher update: https://github.com/lasgroup/SDPO/blob/7c457fc1b1f636ae794eb0362ba37d4743b06fbc/verl/workers/actor/dp_actor.py#L132-L151
• Top-K tail and divergence implementation: https://github.com/lasgroup/SDPO/blob/7c457fc1b1f636ae794eb0362ba37d4743b06fbc/verl/trainer/ppo/core_algos.py#L1085-L1166
• Agarwal et al., "On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes" (source of the stability argument for a symmetric divergence), arXiv 2306.13649: https://arxiv.org/abs/2306.13649
• LiveCodeBench, the competitive-programming benchmark used: https://livecodebench.github.io/

Related: RLVR · GRPO · On-policy distillation · RLSD (RL + self-distillation) · Reward design for RL post-training · Internalizing agent experience