Markdown

Reasoning distillation via SFT on teacher traces¶

Scope: hard distillation for reasoning models, generate reasoning traces (and final answers) from a strong teacher, optionally filter them by correctness, tokenize them with the base model's chat format, and supervise-fine-tune a smaller student on the teacher's own tokens with the loss masked to answer-only positions. This is the recipe behind DeepSeek-R1-Distill-Qwen/Llama: training data is the teacher's generated text, not its logits or a reverse-KL score. It is a stage in fine-tuning and post-training, builds directly on SFT/LoRA and chat rendering & loss masking, and is the correctness-gated consumer of the math verifier.

Not to be confused with on-policy distillation. That page's GKD/reverse-KL method explicitly names this recipe, training a student on a fixed corpus of teacher-generated text, its lambda=0, off-policy baseline: the case it exists to beat. Both are legitimate: this page's offline SFT-on-traces is cheaper, needs no teacher inference at training time, and needs only the teacher's text, not its logits; on-policy distillation needs the teacher resident (or served) throughout training but corrects the student's own exposure-bias mistakes. A common pipeline runs this page's recipe first (cheap capability transfer) and on-policy distillation or RLVR after (push past what pure imitation gives).

The dataset-curation and masking mechanics below follow the shape of Sebastian Raschka's Build a Reasoning Model (From Scratch) (Manning, 2026, Ch. 8); the code is an independent implementation, executed and asserted in this page (numpy/stdlib only, no external dependencies).

What it is¶

Hard distillation asks a strong teacher reasoning model to solve a large set of problems, keeps its generated reasoning trace and final answer as the training target, and runs ordinary supervised fine-tuning: the student's next-token loss is computed against the teacher's own tokens. If you already know the standard LLM SFT pipeline, this is exactly that, on synthetic data generated by a teacher instead of on human-written demonstrations.¹ It differs from soft distillation, which trains the student to match the teacher's full output distribution (needs teacher logits, a shared tokenizer, and access most proprietary teachers do not expose); hard distillation needs only the teacher's generated text, which is why it dominates practice.²

The dataset a hard-distillation run needs, per training example, is a (prompt, teacher_reasoning_trace, teacher_final_answer, ground_truth) tuple. Three things happen to that tuple before it becomes a training example: it is optionally filtered by correctness (does the teacher's final answer match the ground truth?), tokenized with the base model's chat template, concatenating the rendered prompt and the teacher's formatted answer, and loss-masked so the cross-entropy loss is computed only over the answer tokens, never the prompt.

Why use it¶

DeepSeek-R1's own result: distillation beat RL for small models. The R1 paper reports that distilling a smaller model directly from R1's own reasoning traces produced better results than applying RLVR/GRPO from scratch on the small model itself, and released the DeepSeek-R1-Distill family (Qwen and Llama backbones) on exactly this recipe.¹
Cheap and simple relative to RL or on-policy distillation. No rollout loop, no reward function, no teacher inference during training, generate the dataset once, offline, then run standard SFT. The teacher can even be a hosted API the student never touches at training time.
Direct supervision on the exact trace format you want. Because the target is literal teacher text, the student learns whatever structure the teacher used, including <think>...</think> reasoning-trace delimiters, verbatim, without needing a reward to shape that behaviour indirectly.

When to use it (and when not)¶

Use this recipe as the first post-training stage when a stronger reasoning teacher exists and you want to transfer its reasoning style and capability cheaply before any RL stage; it is the natural cold start for RLVR on a small base model.
Prefer on-policy distillation when the student's own exposure-bias mistakes matter more than raw transfer cost, it corrects errors the student actually makes, which this offline recipe cannot see.
Prefer RLVR directly when no teacher exists that is stronger than your own base model; distillation is bounded by the teacher, RL is not.
Watch provider terms of service. Generating training data from a proprietary hosted model (rather than an open-weight one like DeepSeek-R1) may be restricted by that provider's usage policy; check before building a dataset this way.
Do not skip correctness filtering without knowing the cost. Training on a teacher's wrong reasoning trace as if it were correct is direct bad supervision (below); at minimum, measure the teacher's accuracy on your dataset so you know how much noise you are training on.

Architecture¶

flowchart LR
  PROB["Training problems<br/>(non-overlapping with the eval set)"] --> TEACH["Teacher LLM generates<br/>reasoning trace + boxed answer"]
  TEACH --> GRADE{"Correctness filter<br/>(math verifier)"}
  GRADE -->|"correct"| KEEP["Kept: (problem, trace, answer)"]
  GRADE -->|"wrong"| DROP["Dropped (or kept as noise, a documented choice)"]
  KEEP --> TOK["Render chat prompt + format teacher answer<br/>Tokenize both, record prompt_len"]
  TOK --> LEN{"Total length <= max_len?"}
  LEN -->|"yes"| TRAIN["Training example"]
  LEN -->|"no"| DROPLEN["Dropped (long-tail cost, quantified)"]
  TRAIN --> LOSS["Cross-entropy loss,<br/>masked to answer-only tokens"]
  LOSS --> SFT["Standard SFT training loop"]

How to use it¶

1. Correctness filtering: decide how much wrong-label noise to accept¶

The teacher is not perfect; some fraction of its traces reach the wrong final answer while still looking like a plausible worked solution. Filtering by correctness (reusing the same math verifier built for RLVR and eval) removes these before they become training targets. This step is easy to skip, and doing so is not free: a dataset that is, say, 90% correct trains the student on 10% wrong-answer supervision, indistinguishable at training time from a correct trace, since the loss treats every teacher token as ground truth regardless of whether the final boxed answer was right.

# 01_correctness_filter.py -- runnable (stdlib only).
def filter_by_correctness(dataset, grade_fn):
    """Keep only entries whose teacher-generated final answer matches the
    ground truth under grade_fn (in production: the math verifier's grade())."""
    kept = [ex for ex in dataset if grade_fn(ex["teacher_answer"], ex["ground_truth"])]
    return kept, len(dataset) - len(kept)


synthetic_dataset = [
    {"teacher_answer": "83", "ground_truth": "83"},      # correct
    {"teacher_answer": "84", "ground_truth": "83"},      # WRONG teacher trace
    {"teacher_answer": "14/3", "ground_truth": "14/3"},  # correct
    {"teacher_answer": "12", "ground_truth": "13"},      # WRONG teacher trace
]
exact_match = lambda pred, gold: pred == gold   # production: cookbook-math-verifier-rlvr.md's grade()
kept, dropped = filter_by_correctness(synthetic_dataset, exact_match)
assert dropped == 2 and len(kept) == 2
assert all(ex["teacher_answer"] == ex["ground_truth"] for ex in kept)

# --- ADVERSARIAL: an empty dataset must not crash the filter ----------------
assert filter_by_correctness([], exact_match) == ([], 0)

print("01_correctness_filter: all asserts passed")

Whether to filter at all is a real, documented trade-off, not an oversight: on a 12,000-problem MATH training split, Raschka's own DeepSeek-R1-generated dataset measured 90.6% teacher accuracy against ground truth, and the book's tutorial pipeline trains the student on the full, unfiltered set anyway, accepting that ~9.4% noise rather than adding a filtering step.³ That is a defensible choice for a small tutorial run; at production scale, where the marginal cost of running the verifier is negligible next to teacher-generation and training cost, filtering out the wrong-answer fraction is the safer default, it strictly removes a documented source of bad supervision for the price of one verifier pass per example.

2. Length filtering: bound sequence length at a quantified cost¶

Reasoning traces are long and heavy-tailed; a small number of extreme outliers can be tens of thousands of tokens. Capping sequence length keeps training compute bounded, but every cap has a cost in discarded examples, quantify it rather than picking a number blind.

# 02_length_filter.py -- runnable (stdlib only).
def filter_by_length(examples, max_len):
    """Drop examples whose total token length exceeds max_len. Returns
    (kept, removed_count, avg_len_before, avg_len_after)."""
    before = [ex["length"] for ex in examples]
    kept = [ex for ex in examples if ex["length"] <= max_len]
    after = [ex["length"] for ex in kept]
    avg_before = sum(before) / len(before) if before else 0
    avg_after = sum(after) / len(after) if after else 0
    return kept, len(examples) - len(kept), avg_before, avg_after


synthetic_lengths = [{"length": n} for n in (236, 512, 1180, 2048, 3000, 5000, 42005)]
kept, removed, avg_before, avg_after = filter_by_length(synthetic_lengths, max_len=2048)
assert removed == 3                          # 3000, 5000, 42005 exceed the cap
assert all(ex["length"] <= 2048 for ex in kept)
assert avg_after < avg_before                # capping long tails always lowers the mean

# --- ADVERSARIAL: a max_len below every example's length keeps nothing ------
kept_none, removed_all, _, avg_after_none = filter_by_length(synthetic_lengths, max_len=1)
assert kept_none == [] and removed_all == len(synthetic_lengths) and avg_after_none == 0

print("02_length_filter: all asserts passed")
print(f"  example: avg {avg_before:.0f} -> {avg_after:.0f} tokens, removed {removed}/{len(synthetic_lengths)}")

On Raschka's own 12,000-example dataset (average 2,946 tokens, longest 42,005), capping at 2,048 tokens removed 5,305 examples (44%) and brought the average down to 1,180 tokens, a concrete illustration of how heavy the length tail is for reasoning traces and how much a cap actually costs.⁴

3. Answer-only loss masking: the loss must never touch the prompt¶

Tokenize the rendered prompt and the teacher's formatted answer separately, concatenate them, and record the prompt's token length. At training time, shift by one token for next-token prediction and compute the cross-entropy loss only from the position corresponding to the first answer token onward, exactly the chat-rendering masking discipline applied to a teacher-generated target instead of a human one.

# 03_answer_only_loss.py -- runnable (numpy only).
import numpy as np


def softmax(logits):
    z = logits - logits.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)


def cross_entropy(logits, targets):
    probs = softmax(logits)
    picked = probs[np.arange(len(targets)), targets]
    return float(-np.mean(np.log(np.clip(picked, 1e-12, 1.0))))


def answer_only_loss(token_ids, prompt_len, logits_full_seq):
    """token_ids: length T (prompt + teacher answer + eos).
    logits_full_seq: (T-1, V), logits[i] predicts token_ids[i+1] (next-token shift).
    Returns the cross-entropy computed ONLY over answer-token positions, or
    None if there are no answer tokens at all (caller must skip such examples)."""
    target_ids = token_ids[1:]
    answer_start = max(prompt_len - 1, 0)
    answer_logits = logits_full_seq[answer_start:]
    answer_targets = target_ids[answer_start:]
    if len(answer_targets) == 0:
        return None
    return cross_entropy(answer_logits, answer_targets)


rng = np.random.default_rng(0)
V = 12
prompt_len, answer_len = 5, 4
token_ids = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9])          # 5 prompt + 4 answer tokens
target_ids = token_ids[1:]
answer_start = max(prompt_len - 1, 0)

logits = rng.normal(size=(len(token_ids) - 1, V))
for i in range(answer_start, len(token_ids) - 1):
    logits[i] = -5.0
    logits[i, target_ids[i]] = 10.0                        # confident, correct at answer positions

confident_loss = answer_only_loss(token_ids, prompt_len, logits)

# --- masking invariant: perturbing PROMPT-position logits must NOT move the
# answer-only loss, since the slice excludes them entirely -------------------
logits_perturbed_prompt = logits.copy()
logits_perturbed_prompt[:answer_start] = rng.normal(size=(answer_start, V)) * 50.0
assert abs(confident_loss - answer_only_loss(token_ids, prompt_len, logits_perturbed_prompt)) < 1e-9

# --- sensitivity: perturbing ANSWER-position logits MUST move the loss -----
logits_worse_answer = logits.copy()
logits_worse_answer[answer_start:] = rng.normal(size=(answer_len, V))
assert answer_only_loss(token_ids, prompt_len, logits_worse_answer) > confident_loss

# --- a confident, correct answer has loss near zero -------------------------
assert confident_loss < 0.01, confident_loss

# --- ADVERSARIAL: prompt_len == 0 -> no masking, loss covers the whole sequence
assert answer_only_loss(token_ids, prompt_len=0, logits_full_seq=logits) is not None

# --- ADVERSARIAL: prompt_len == total length (empty answer) -> caller must skip
assert answer_only_loss(token_ids, prompt_len=len(token_ids), logits_full_seq=logits) is None

print("03_answer_only_loss: all asserts passed")

How to develop with it¶

Measure teacher accuracy on your own dataset before deciding whether to filter. The 90.6%-unfiltered choice above worked at tutorial scale; a lower teacher accuracy (a harder domain, a weaker or cheaper teacher) raises the cost of skipping the correctness filter and should push you toward filtering.
Watch training loss and validation loss separately. Training loss is noisier (measured on the batch currently being optimized); validation loss on a held-out slice is the cleaner signal that the student generalizes to teacher-quality reasoning rather than memorizing training examples.
Keep <think>...</think> (or your model family's reasoning-trace delimiter) if you want a separable trace at inference time; it is not required for the distillation loss itself, only for later hiding or inspecting the trace in a serving UI.

How to run it in production¶

Generate once, offline, at teacher-inference cost only. Because the dataset does not depend on the student at all, generate it in one batch pass against the teacher (a hosted API or a served checkpoint) before any student training starts; this is the main cost advantage over on-policy distillation, which needs the teacher resident throughout training.
Batch the loss computation. The single-example masking mechanics above generalize to a batch by padding to the batch's max length and additionally masking out pad positions the same way prompt positions are masked; see batching and throughput-oriented execution for the general pattern.
Reuse the SFT stack. Nothing about this recipe needs new training infrastructure: it is the SFT/LoRA pipeline pointed at a synthetic, teacher-generated dataset instead of human demonstrations.

How to maintain it¶

Re-measure teacher accuracy and the length distribution whenever the teacher, the problem set, or the prompt template changes, a new teacher checkpoint or a reworded prompt can shift both the correctness rate and how verbose traces are, silently changing how much the correctness and length filters actually remove. Keep the training/validation loss curves and the eventual downstream eval (math verifier-graded accuracy on a held-out set) in the same tracking record so a regression is traceable to a specific dataset regeneration.

Failure modes¶

Training on unfiltered wrong-answer traces without knowing the rate. The loss cannot distinguish a correct from an incorrect teacher trace; if you skip correctness filtering, at minimum measure and record the teacher's accuracy so the noise level is known, not silently absorbed.
Loss leaking onto the prompt. A masking bug that includes prompt tokens in the loss teaches the student to predict the problem statement, not the answer; the invariance check in the masking block above (perturbing prompt logits must not move the loss) is the concrete test for this.
Length cap chosen without measuring the cost. A cap set without checking the length distribution can silently discard a large, possibly non-random, fraction of examples (harder problems tend to produce longer traces); always report kept/removed counts.
Mistaking this for on-policy distillation. This recipe trains on the teacher's own generated trajectories, not the student's; it inherits standard SFT's off-policy exposure-bias limitation (on-policy distillation's entire reason to exist), not a bug in this recipe, a scope boundary.
Provider terms-of-service exposure. Generating a training corpus from a proprietary hosted model's outputs can violate that provider's usage policy; this risk does not apply when the teacher is an open-weight model like DeepSeek-R1.

References¶

DeepSeek-R1: distillation from R1 traces into smaller Qwen/Llama checkpoints outperforms applying RL directly to those checkpoints: https://arxiv.org/abs/2501.12948
Hinton, Vinyals, Dean, Distilling the Knowledge in a Neural Network (the classical hard/soft distillation framing): https://arxiv.org/abs/1503.02531
Hendrycks et al., Measuring Mathematical Problem Solving With the MATH Dataset: https://arxiv.org/abs/2103.03874
Raschka, Build a Reasoning Model (From Scratch), Manning Publications, 2026 (Ch. 8: dataset generation, correctness measurement, tokenization, and answer-only loss masking for reasoning distillation).

DeepSeek-R1, Section 4.1 "Distillation v.s. Reinforcement Learning": "DeepSeek-R1-Distill-Qwen-32B, which is distilled from DeepSeek-R1, performs significantly better than DeepSeek-R1-Zero-Qwen-32B across all benchmarks" (the latter is Qwen2.5-32B-Base trained with large-scale RL directly). Released as the DeepSeek-R1-Distill family (Qwen 1.5B/7B/14B/32B, Llama 8B/70B), SFT-only on ~800k reasoning samples generated by the R1 teacher, no additional RL on the small models. https://arxiv.org/abs/2501.12948 ↩↩
Hinton, Vinyals, Dean: the original hard (train on the teacher's chosen outputs) versus soft (match the teacher's full output distribution) distillation framing, developed for computer vision and generalized since. https://arxiv.org/abs/1503.02531 ↩
Raschka, Build a Reasoning Model (From Scratch), Ch. 8: measuring DeepSeek-R1-teacher accuracy on a 12,000-problem MATH training split via the book's own math verifier reports 90.6% (10,871/12,000); the book's tutorial pipeline trains the student on the full, unfiltered set. Cited as a real, reported data point illustrating the correctness-filtering trade-off; not independently reproduced in this KB. ↩
Raschka, Build a Reasoning Model (From Scratch), Ch. 8: on the same 12,000-example dataset, average length 2,946 tokens (longest 42,005); capping at 2,048 tokens removes 5,305 examples (44%), bringing the average down to 1,180 tokens. Cited as a real, reported data point illustrating length-filtering cost; not independently reproduced in this KB. ↩