Markdown

Multi-teacher on-policy distillation¶

Scope: the post-training stage that takes N independently trained RL specialists and produces one checkpoint that keeps most of each specialist's capability. This page covers the three-stage MOPD pipeline, the per-token reverse-KL objective and the two implementations of it (policy gradient and bias-corrected top-k), the teacher-prefill service that keeps the wall-clock cost near zero, the same-origin teacher constraint that is the difference between working and catastrophic, and how the same idea appears in Kimi K3 as a clipped log-ratio reward. It is the multi-teacher extension of on-policy distillation, the alternative to model merging for capability integration, and the consolidation step referenced by reasoning-effort control.

The numpy block below is executed and asserted in this page (Python 3.11, numpy 2.x). It reproduces the published normalised scores of arXiv:2606.30406 Tables 2 and 4 from their own raw accuracy columns, and verifies the mathematical claims made about the objective (non-negativity of the corrected top-k loss, exactness of the policy-gradient form, the effect of clipping) on synthetic distributions. It does not retrain anything; the accuracy numbers are the papers', not ours.

What it is¶

MOPD is a three-stage recipe:

General SFT. Fine-tune the base model on a broad corpus covering every target capability. This one checkpoint is the shared initialisation for everything downstream, both the teachers and the student.
Domain-specialised RL. For each domain d, train an expert from the Stage-1 checkpoint using whatever RL recipe is natural for that domain: verifiable-answer RL for math, executable-sandbox agent RL for software engineering, rubric-based RL for instruction following, web-environment RL for search. These runs are independent and fully parallel.
MOPD. Initialise a student from the Stage-1 checkpoint, freeze the experts as a teacher group, and train the student on a multi-domain prompt set. Each step: the student generates a rollout, the rollout is dispatched to the teacher matching that prompt's domain, that teacher prefills the trajectory to get per-token log-probabilities, and the student is updated by minimising the per-token reverse KL along the trajectory.

The defining choice is where integration happens. Mix-RL integrates in data space (pool the prompts). Param-Merge integrates in weight space (average or task-arithmetic the checkpoints). MOPD integrates in policy space: each prompt is routed to its own teacher, and the only thing that gets combined is the gradient signal arriving at a single student.

Kimi K3 uses the same construction to fold nine experts (three domains crossed with three reasoning-effort levels) into one shipped model, with the teacher selected by both domain d and requested effort e.

Why use it¶

It beats every other integration paradigm in a controlled comparison. On Qwen3-30B-A3B across math, instruction following, and SWE, MOPD scores 0.9373 on the normalised metric against 0.8818 for the best baseline (Mix-RL), 0.8574 for task-arithmetic merging, 0.8241 for off-policy finetuning, 0.7752 for cascade RL, and 0.3280 for naive weight averaging.
No exposure bias, by construction. The student is scored on trajectories it generated itself, so the training state distribution and the inference state distribution are the same. Classical distillation minimises a forward KL over a fixed corpus of teacher completions, and the student's own inference trajectories drift away from that corpus.
Dense supervision. The teacher gives a distribution at every token, not one scalar at the end of the trajectory. That is a far lower-variance signal than an outcome reward, which matters most on long agentic trajectories where a single terminal reward has to explain thousands of tokens.
Teachers develop in parallel and in isolation. Each domain team runs its own RL with its own hyperparameters. A blown-up run or a config change in one domain does not touch the others. For an organisation this is the real argument: it decouples teams that Mix-RL would otherwise couple through a shared training batch.
It sidesteps the see-saw effect. Pooling prompts from different domains into one RL run makes the gradients interfere, so gains in one domain come at the cost of another. Routing avoids the interference entirely.
The teacher cost is nearly free in wall clock. See the infrastructure section: teacher prefill overlaps with student sampling.

When to use it (and when not)¶

Use it when:

You already have, or can afford to train, more than one domain-specialised RL checkpoint from a common ancestor.
Your domains want genuinely different RL machinery (a sandbox for SWE, a verifier for math, a rubric judge for writing) and jamming them into one loop is what you are trying to avoid.
You need effort levels or personas as well as domains. The routing key is arbitrary; K3 routes on (domain, effort).

Do not use it when:

Your teachers do not share an ancestor. This is the hard constraint, and it is not a soft preference. See the same-origin section: substituting a stronger off-lineage teacher took the normalised score from 0.9373 to 0.6003 with the policy-gradient loss and to -1.1898 with top-k, which is far below simply not doing the integration at all.
You only have one domain. Then you want plain on-policy distillation, and the multi-teacher routing is dead weight.
You cannot host the teachers. Stage 3 needs every frozen expert servable for prefill concurrently. At frontier scale that is a real fleet.
You need the student to exceed its teachers. It generally will not. MOPD scored 0.9373, that is, it retained about 94% of the specialist gains, not 100%. The per-benchmark deltas against the teachers are -3.33 (AIME25), +1.66 (AIME26), -0.51 (IFBench), -1.66 (IFEval), -0.80 (SWE-bench Verified). Consolidation is a compression step with a real, if small, cost.

Architecture¶

flowchart TB
  BASE["Base model"] --> SFT["Stage 1: general SFT<br/>(shared ancestor)"]
  SFT --> T1["Stage 2: RL teacher, math<br/>verifiable-answer RL"]
  SFT --> T2["Stage 2: RL teacher, IF<br/>rubric RL"]
  SFT --> T3["Stage 2: RL teacher, SWE<br/>sandbox agent RL"]
  SFT --> STU["Stage 3: student<br/>(same init as teachers)"]
  STU -->|"1. generate rollout y ~ pi_theta"| ROUTE{"route on prompt domain<br/>(and effort level)"}
  T1 -.->|frozen| PREFILL
  T2 -.->|frozen| PREFILL
  T3 -.->|frozen| PREFILL
  ROUTE -->|"2. async prefill request"| PREFILL["Teacher prefill service<br/>(outside the RL trainer)"]
  PREFILL -->|"3. per-token logprobs / top-k"| ADV["per-token advantage<br/>A = clip(log pi_teacher - log pi_student)"]
  ADV -->|"4. policy update"| STU
  STU --> OUT["Unified checkpoint"]

The dashed teacher edges matter. The teachers are frozen and live outside the RL trainer as standalone prefill services, addressed the same way a reward model would be.

How it works: the objective, validated¶

import numpy as np

# ---------------------------------------------------------------- 1. the metric
# Table 2 of arXiv:2606.30406, Qwen3-30B-A3B. Accuracy (%) per benchmark.
ACC = {
    "Student (SFT-only)":       [45.42, 54.48, 42.69, 84.17, 35.80],
    "RL Teacher (per-domain)":  [54.79, 63.65, 78.40, 95.50, 51.20],
    "Mix-RL":                   [52.71, 63.75, 75.00, 94.58, 48.80],
    "Cascade RL":               [48.54, 61.88, 77.11, 95.80, 47.80],
    "Off-Policy Finetune":      [51.56, 63.44, 80.95, 93.35, 45.80],
    "Param-Merge (Avg.)":       [47.81, 59.58, 53.74, 88.79, 39.60],
    "Param-Merge (Task Arith.)":[49.38, 63.96, 78.23, 95.81, 48.80],
    "MOPD":                     [51.46, 65.31, 77.89, 93.84, 50.40],
}
# columns: AIME25, AIME26 | IFBench, IFEval | SWE-bench Verified
SPAN = {"math": slice(0, 2), "if": slice(2, 4), "swe": slice(4, 5)}


def normalised_score(row: list[float]) -> float:
    """Accuracy averaged within a domain, rescaled student->0 / teacher->1,
    then averaged uniformly across domains."""
    stu = np.array(ACC["Student (SFT-only)"])
    tea = np.array(ACC["RL Teacher (per-domain)"])
    m = np.array(row)
    per_domain = []
    for sl in SPAN.values():
        s, t, v = stu[sl].mean(), tea[sl].mean(), m[sl].mean()
        per_domain.append((v - s) / (t - s))
    return float(np.mean(per_domain))


PUBLISHED = {"Student (SFT-only)": 0.0000, "RL Teacher (per-domain)": 1.0000,
             "Mix-RL": 0.8818, "Cascade RL": 0.7752, "Off-Policy Finetune": 0.8241,
             "Param-Merge (Avg.)": 0.3280, "Param-Merge (Task Arith.)": 0.8574,
             "MOPD": 0.9373}
for name, row in ACC.items():
    got = normalised_score(row)
    assert abs(got - PUBLISHED[name]) < 5e-5, (name, got, PUBLISHED[name])

best_baseline = max((k for k in PUBLISHED if k not in
                     ("MOPD", "Student (SFT-only)", "RL Teacher (per-domain)")),
                    key=PUBLISHED.get)
assert best_baseline == "Mix-RL"
gap = PUBLISHED["MOPD"] - PUBLISHED[best_baseline]
assert abs(gap - 0.0555) < 1e-4, gap
assert PUBLISHED["MOPD"] < 1.0  # MOPD does NOT beat its own teachers on average

# Per-domain: MOPD trails the teacher on four of five benchmarks.
mopd = np.array(ACC["MOPD"]); tea = np.array(ACC["RL Teacher (per-domain)"])
deltas = np.round(mopd - tea, 2)
assert list(deltas) == [-3.33, 1.66, -0.51, -1.66, -0.80], list(deltas)

# Table 4: swapping the same-origin math teacher for a stronger but
# distributionally distant one (Qwen3-235B-A22B) is catastrophic, not helpful.
CROSS = {
    "same-origin, policy gradient": ([51.46, 65.31, 77.89, 93.84, 50.40],  0.9373),
    "same-origin, top-k":           ([51.77, 64.79, 75.85, 93.07, 50.20],  0.9093),
    "cross-origin, policy gradient":([45.63, 51.56, 79.25, 93.99, 50.60],  0.6003),
    "cross-origin, top-k":          ([0.94,  0.42,  72.96, 88.97, 51.20], -1.1898),
}
for name, (row, published) in CROSS.items():
    assert abs(normalised_score(row) - published) < 5e-5, (name, normalised_score(row))
assert CROSS["cross-origin, top-k"][1] < 0      # worse than doing no RL at all
assert CROSS["same-origin, top-k"][1] < CROSS["same-origin, policy gradient"][1]

# ------------------------------------------------- 2. the top-k corrected loss
def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()


def naive_topk_revkl(p_student, p_teacher, idx):
    return float(np.sum(p_student[idx] * np.log(p_student[idx] / p_teacher[idx])))


def corrected_topk_revkl(p_student, p_teacher, idx):
    s, t = p_student[idx], p_teacher[idx]
    return float(np.sum(s * np.log(s / t) - s + t))


rng = np.random.default_rng(0)
V, K = 12, 4
teacher = softmax(rng.normal(size=V) * 1.5)
idx = np.argsort(teacher)[::-1][:K]

# The corrected form is a generalized KL (the Bregman divergence of x log x):
# non-negative everywhere, and exactly zero iff student == teacher on top-k.
assert abs(corrected_topk_revkl(teacher, teacher, idx)) < 1e-12
for _ in range(2000):
    cand = softmax(rng.normal(size=V) * 1.5)
    assert corrected_topk_revkl(cand, teacher, idx) > -1e-12

# The NAIVE truncated form is not minimised at the teacher: it can be driven
# below the teacher's own value by moving student mass off the top-k set.
base = naive_topk_revkl(teacher, teacher, idx)
off = np.zeros(V); off[np.argsort(teacher)[0]] = 1.0
starved = softmax(np.log(np.clip(0.999 * off + 0.001 * teacher, 1e-12, None)))
assert naive_topk_revkl(starved, teacher, idx) < base, "naive form is gameable"
assert corrected_topk_revkl(starved, teacher, idx) > corrected_topk_revkl(teacher, teacher, idx)

# ------------------------------------- 3. the policy-gradient form is exact
V2 = 6
theta = rng.normal(size=V2)
teach2 = softmax(rng.normal(size=V2) * 1.2)


def rev_kl(th):
    p = softmax(th)
    return float(np.sum(p * np.log(p / teach2)))


def fd_grad(f, th, eps=1e-6):
    g = np.zeros_like(th)
    for j in range(len(th)):
        a = th.copy(); a[j] += eps
        b = th.copy(); b[j] -= eps
        g[j] = (f(a) - f(b)) / (2 * eps)
    return g


def pg_grad(th):
    """grad of L_PG = -E[Ahat * log pi], with Ahat = sg[log teacher - log student]."""
    p = softmax(th)
    adv = np.log(teach2) - np.log(p)              # Eq. 3, stop-gradient
    jac = np.eye(len(th)) - p[None, :]            # d log p_v / d theta_j
    return -np.sum(p[:, None] * adv[:, None] * jac, axis=0)


assert np.allclose(fd_grad(rev_kl, theta), pg_grad(theta), atol=1e-7)
# The two agree because the score-function expectation vanishes.
p = softmax(theta)
score_exp = np.sum(p[:, None] * (np.eye(V2) - p[None, :]), axis=0)
assert np.allclose(score_exp, 0.0, atol=1e-12)

# ------------------------------------------------- 4. clipping is a real bias
def mopd_advantage(logp_teacher, logp_student, a_max):
    return np.clip(logp_teacher - logp_student, -a_max, a_max)


teacher_p = np.array([0.60, 0.10, 0.29, 0.01])
student_p = np.array([0.05, 0.30, 0.29, 0.36])
assert np.isclose(teacher_p.sum(), 1.0) and np.isclose(student_p.sum(), 1.0)
raw = np.log(teacher_p) - np.log(student_p)
cl = mopd_advantage(np.log(teacher_p), np.log(student_p), a_max=2.0)
assert np.isclose(raw[2], 0.0)                      # agreement -> no push at all
assert raw[0] > 2.0 and cl[0] == 2.0                # teacher-favoured, saturated
assert raw[3] < -2.0 and cl[3] == -2.0              # student-favoured, saturated
assert np.isclose(cl[1], raw[1]) and raw[1] < 0     # mild disagreement passes through
assert np.count_nonzero(raw != cl) == 2             # exactly the two extremes clip
# Clipping shrinks the total pull toward the teacher by a measurable amount.
assert np.abs(cl).sum() < np.abs(raw).sum()
assert abs(float(np.abs(raw).sum() - np.abs(cl).sum()) - 2.068) < 1e-3

print("all MOPD assertions passed")
print("  reproduced normalised scores:",
      {k: round(normalised_score(v), 4) for k, v in ACC.items()})
print("  MOPD - best baseline (Mix-RL):", round(gap, 4))

Executed output:

all MOPD assertions passed
  reproduced normalised scores: {'Student (SFT-only)': 0.0, 'RL Teacher (per-domain)': 1.0, 'Mix-RL': 0.8818, 'Cascade RL': 0.7752, 'Off-Policy Finetune': 0.8241, 'Param-Merge (Avg.)': 0.328, 'Param-Merge (Task Arith.)': 0.8574, 'MOPD': 0.9373}
  MOPD - best baseline (Mix-RL): 0.0555

Reading the metric before you read the result¶

Block 1 reproduces all eight published normalised scores from the paper's own accuracy columns, which pins down the definition: accuracy is averaged within a domain first, then rescaled so the SFT student is 0 and the per-domain specialist is 1, then averaged uniformly across domains. That ordering matters. Averaging over the five benchmarks directly gives 0.9226 for MOPD, not 0.9373, because it would double-weight the two-benchmark domains. When you build your own version of this comparison, get the aggregation order right or the headline number is not comparable to anyone's.

The metric also has a useful property: it is unbounded in both directions. Above 1 means the student beat its own specialist; below 0 means it regressed past the SFT starting point. That is what makes the cross-origin top-k result interpretable as a catastrophe rather than a disappointment.

The two implementations are the same objective¶

The Stage-3 objective is a per-token reverse KL toward the dispatched teacher. The paper gives two ways to optimise it:

Policy gradient. Differentiating the reverse KL yields exactly a policy-gradient form whose per-token advantage is the teacher-student log-difference, A = sg[log pi_teacher(y_t) - log pi_student(y_t)], two-side clipped to [-A_max, +A_max]. Block 3 checks this is not an approximation: the finite-difference gradient of the reverse KL matches the policy-gradient expression to 1e-7. They agree because the score-function expectation E[grad log pi] is exactly zero, which the block also asserts. The practical consequence is the selling point: the advantage computation is the only change you need to make to an existing PPO or GRPO trainer.
Top-k. Rather than using only the sampled token, distil against the teacher's top-k set. The loss is not a plain truncated reverse KL; it carries an extra - pi_student(v) + pi_teacher(v) term. Block 2 shows why that term is load-bearing. The corrected form is the generalised KL divergence (the Bregman divergence of x log x), so it is non-negative over 2000 random student distributions and exactly zero only at pi_student = pi_teacher on the top-k set. The naive truncated version has neither property: the block constructs a student that pushes almost all its mass onto a token outside the top-k set and thereby scores lower than the teacher scores against itself. Optimising the naive form therefore rewards abandoning the teacher's support instead of matching it.

Which to use: the paper's own analysis says they perform comparably with same-origin teachers (0.9373 policy gradient against 0.9093 top-k), that the per-token reverse KL starts from an already low value of about 0.04, and that "with same-origin teachers, the policy-gradient form is already sufficiently stable; introducing top-k does not yield additional stability or variance reduction". Top-k's real advantage is operational: shipping k logits per token keeps the teacher payload small enough to move like a reward signal, where full-vocabulary distillation would require transmitting the entire vocabulary distribution per token.

Block 4 makes the cost of clipping concrete. On a four-token example, two of four positions saturate at A_max = 2.0 and the total magnitude of the pull toward the teacher drops by 2.068. Clipping is buying stability with bias; if you see the student converge to a visibly weaker policy than the teacher on tokens where they disagree most, A_max is the first thing to raise.

How to develop with it: the same-origin constraint¶

This is the finding to internalise, and it runs against intuition. Replacing the math teacher with Qwen3-235B-A22B, a stronger model, made the result dramatically worse:

Teacher for math	Loss	AIME25	AIME26	Normalised score
Same-origin RL teacher	policy gradient	51.46	65.31	0.9373
Same-origin RL teacher	top-k	51.77	64.79	0.9093
Qwen3-235B-A22B	policy gradient	45.63	51.56	0.6003
Qwen3-235B-A22B	top-k	0.94	0.42	-1.1898

The top-k row is not a regression, it is a collapse: 0.94 and 0.42 on AIME are effectively zero, and the normalised score below -1 means the model ended up much worse than the SFT checkpoint it started from. The mechanism is distribution mismatch. On-policy distillation scores the student's rollouts. When the teacher shares the student's lineage, those rollouts land in the teacher's high-probability region, the initial reverse KL is small (about 0.04), and the log-ratio advantage is a well-conditioned correction. When the teacher is off-lineage, the student's rollouts fall in the teacher's tail, log-ratios blow up, and the top-k form (which only looks at k tokens the teacher likes, most of which the student never samples) receives almost no usable signal at all.

Practical rules that follow:

Derive every teacher from the same SFT checkpoint that initialises the student. Not "the same base model", the same post-SFT checkpoint.
Do not swap in a third-party frontier model as a teacher, however strong it is. If you want capability from an off-lineage model, get it in with off-policy SFT on its completions first, then run MOPD with same-origin teachers.
Instrument the initial per-token reverse KL before committing to a run. A value near 0.04 is the healthy regime the paper reports. An initial KL an order of magnitude higher is the signal to stop.
Watch policy entropy. The paper reports it stable around 0.30 throughout a healthy run.

Multi-round refinement is available: the paper reports benefits from iterating the pipeline, using the consolidated student as the ancestor for a new round of specialists.

How to run it in production: the teacher-prefill service¶

The naive implementation folds teacher prefill into the RL training loop and pays for it in serial latency. MOPD instead observes that teacher prefill has the same shape as reward computation, and deploys each domain teacher as a standalone prefill service outside the trainer.

The runtime is:

The student sampler generates rollouts continuously.
As soon as one sequence finishes, the trainer fires an asynchronous prefill request to that prompt's teacher service.
The service returns per-token log-probabilities (or top-k logits).
Because teacher prefill overlaps with other sequences still sampling, wall-clock cost is dominated by sampling alone.

The paper's claim is strong: "the teacher contributed nearly no measurable wall-clock overhead." That is credible for the same reason async RL works in general, and it depends on the same conditions:

Sampling must be the bottleneck. If your rollouts are short and your teachers are large, prefill stops hiding and starts dominating. Measure the ratio before assuming it is free.
Prefill is compute, not decode. One forward pass over a completed trajectory, no autoregressive loop, so it batches well and runs at high utilisation.
The teacher fleet is a capacity line item even if it is not a latency one. N frozen experts served concurrently is N deployments. At the K3 scale of nine experts this is a fleet, and it is the main reason to consider top-k (smaller payloads) over full-vocabulary transfer.

Route on a stable, explicit key. The routing key is the prompt's domain (and, in K3, the requested effort level). Derive it from dataset metadata, not from a classifier, so a misrouted prompt is a data bug you can find rather than a silent quality regression.

How to maintain it: what to keep and what to re-run¶

Keep the teachers. The whole modularity argument dies if you discard them. A regression in one domain is then fixed by retraining one expert and re-running Stage 3, not by redoing everything.
Adding a capability is one new expert plus a Stage-3 re-run. This is the strongest operational property of the approach and the reason it scales organisationally.
Re-run Stage 3 after any teacher changes. The student is a function of the whole teacher group; you cannot patch in one teacher's contribution.
Version the ancestor. If the Stage-1 SFT checkpoint changes, every teacher is off-lineage with respect to the new student and you are in the failure regime above. Treat the SFT checkpoint hash as part of the identity of every teacher.

How it appears in Kimi K3¶

K3 states the same objective as a dense reward inside its RL framework rather than as a separate distillation loss. For domain d and sampled effort level e, the per-token OPD reward against the corresponding one of nine teachers is a clipped stop-gradient log-ratio, clip(sg[log pi_teacher(y_t | x, y_<t, e) - log pi_student(y_t | x, y_<t, e)], -R_max, R_max), which is algebraically the MOPD advantage of Eq. 3 with R_max playing the role of A_max. Two notes from that report are worth carrying over:

Framing it as a reward rather than a loss means it "seamlessly integrates into our RL framework", which in particular lets it inherit partial-rollout training for long-horizon tasks where a trajectory spans multiple iterations.
K3 also tried finer-grained top-k objectives and "observed no clear advantage in either convergence speed or final performance", independently matching the MOPD paper's own analysis.

At frontier scale the MOPD paper reports the pipeline applied to MiMo-V2-Flash (309B total, 15B active) with teachers covering math, code, instruction following, SWE, and tool use, matching or exceeding the corresponding teacher on most benchmarks with two modest regressions (IFBench -2.2, SWE-bench Verified -0.8).

Failure modes¶

Off-lineage teacher. The dominant failure. Normalised score falls to 0.60 with policy gradient and to -1.19 with top-k. Symptom: high initial reverse KL, exploding log-ratios, collapse on the affected domain.
Naive top-k truncation. Dropping the -pi_student(v) + pi_teacher(v) correction gives a loss that is minimised by moving mass off the teacher's top-k set, the opposite of the intent. Block 2 demonstrates it.
Wrong aggregation in the metric. Averaging across benchmarks instead of across domains changes the headline (0.9226 against 0.9373 here) and makes cross-paper comparison meaningless.
Clipping set too tight. A_max too small silently caps learning exactly on the tokens where teacher and student disagree most, which is where the capability lives.
Misrouted prompts. A prompt sent to the wrong teacher gets dense, confident, wrong supervision. This is worse than a missing reward, because it is not noisy, it is systematically misleading.
Teacher prefill on the critical path. Folding prefill into the trainer loop adds serial latency and gives up the main infrastructure win.
Expecting the student to beat the teachers. It scores 0.9373, not above 1. Budget for a few points of loss per domain and decide whether that is worth the single-checkpoint deployment.
Stale teachers after an SFT refresh. Silent, and it converts a working pipeline into the off-lineage failure without any config change.

References¶

MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training: https://arxiv.org/abs/2606.30406
MiMo-V2-Flash Technical Report (MOPD at 309B scale): https://arxiv.org/abs/2601.02780
Kimi K3 Technical Report (MOPD over nine domain-by-effort experts): https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf
MiniLLM (reverse-KL on-policy distillation, the policy-gradient template): https://arxiv.org/abs/2306.08543
On-policy distillation / GKD: https://arxiv.org/abs/2306.13649
TIES-Merging (weight-space integration baseline): https://arxiv.org/abs/2306.01708
Task arithmetic (weight-space integration baseline): https://arxiv.org/abs/2212.04089