Markdown

Continual learning with RL for LLMs¶

Scope: adding a skill to a model that already has capabilities worth keeping. This page covers why on-policy RL is empirically far more resistant to catastrophic forgetting than supervised fine-tuning, the KL-drift mechanism behind it, how to run sequential RL phases rather than one combined run, what to instrument so regression is caught during training rather than at eval, and the cases where RL still forgets. It sits between fine-tuning and post-training, which chooses the method, and KL regularization, which supplies the measurement.

The numpy block is executed and asserted here. It is a deliberately small model of the mechanism, on a 16-action categorical policy, not a measurement of any real LLM. The empirical claims about real models come from the cited sources.

What it is¶

Catastrophic forgetting is the standard failure of sequential training: a network trained on task B loses task A. The historical remedies are replay buffers, regularization toward old weights, and parameter isolation, all of which cost engineering effort.

Recent results show that on-policy RL largely does not need them. Training a model on a new task with RL preserves prior capability far better than SFT does at matched new-task performance, without replay or explicit regularization. The explanation is distributional rather than architectural: how much a fine-tuned model forgets is predicted by the KL divergence between the fine-tuned and base policies measured on the new task, and on-policy RL is implicitly biased toward the KL-minimal solutions among all the ways to solve that task. SFT has no such bias and can converge to a distribution arbitrarily far from the base model. The RL's Razor paper states the principle directly: among all ways to solve a new task, RL prefers those closest in KL to the original model.

The mechanism is visible in the gradients and is validated below. The on-policy policy gradient for an action is scaled by that action's current probability, so RL cannot move mass onto behaviour the policy never samples. The SFT cross-entropy gradient carries no such factor: a demonstration on a token the model currently assigns near-zero probability still produces a full-size update. RL is therefore structurally confined to reweighting what the policy already does.

Why use it¶

Skills become additive. If sequential RL phases mostly preserve earlier phases, capability can be built up in stages instead of re-running one enormous combined job every time a domain is added. NVIDIA's Nemotron-Cascade 2 is built this way, expanding what it calls Cascade RL across a broad spectrum of reasoning and agentic domains after an SFT stage.
It removes a whole class of machinery. No replay buffer to curate, no elastic-weight-consolidation penalty to tune, no per-task adapters to route between.
The failure has a leading indicator. KL drift on the new task is measurable during training and ranks runs by how much they will forget, so a regression can be caught at step 400 instead of at the eval after the run.
It reframes an SFT-versus-RL choice that is usually made on cost. SFT is cheaper per unit of new-task progress. If preserving general capability matters, that comparison is incomplete, because the two methods do not pay the same price in forgetting.

When to use it (and when not)¶

Use sequential RL to add a domain to a model whose existing behaviour is valuable and only partly covered by your evals. This is the common case for a production model getting a new capability.
Use SFT first anyway for format and cold start. RL cannot teach behaviour the policy never samples, which is exactly the gradient masking that makes it safe. A model that cannot produce the target format at all will not discover it through RL at any reasonable budget.
Instrument KL drift regardless of method. The predictor is the drift, not the label on the optimizer. A high-drift RL run forgets like SFT.
Do not treat this as a licence to skip regression evals. RL forgets less, not zero. Nemotron-Cascade 2 introduces multi-domain on-policy distillation specifically to recover benchmark regressions during the Cascade RL process, which means the regressions were real and were measured.
Do not expect it from offline RL. The argument is about on-policy data. DPO and other offline objectives train on fixed completions and are closer to SFT in this respect.

Architecture¶

flowchart TB
  BASE["Base checkpoint"] --> SFT["SFT: format and cold start"]
  SFT --> P1["RL phase 1: domain A"]
  P1 --> P2["RL phase 2: domain B"]
  P2 --> P3["RL phase 3: agentic"]
  P1 -.->|"KL drift on new task"| MON["Drift and regression monitor"]
  P2 -.-> MON
  P3 -.-> MON
  MON -->|"drift rising, prior evals falling"| MIT["Mitigate: lower LR, raise beta,<br/>mix replay prompts, on-policy distil"]
  MIT -.-> P2
  P3 --> OUT["Released checkpoint"]
  HOLD["Frozen prior-capability eval suite"] -.->|"run every phase, not at the end"| MON

How to use it¶

The mechanism is small enough to prove exactly. The block below models a shared backbone as one update vector applied in two contexts, a new task and a prior task, and compares where RL and SFT land at the same new-task accuracy.

# rl_razor_model.py -- why on-policy RL forgets less than SFT, and why KL drift is the
# quantity to instrument. A deliberately small MODEL of a shared backbone (two contexts,
# one shared update vector), not a measurement of any real model. numpy only.
import numpy as np

V = 16                                        # action vocabulary
rng = np.random.default_rng(3)

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

def kl(a, b):
    p, q = softmax(a), softmax(b)
    return float((p * np.log(p / q)).sum())

base_new = rng.normal(0, 1, V)                # base logits in the NEW-task context
base_prior = rng.normal(0, 1, V)              # base logits in a PRIOR-task context
new_ok = np.zeros(V); new_ok[[2, 5, 9, 11]] = 1.0     # 4 acceptable new-task answers
prior_ok = np.zeros(V); prior_ok[[1, 4]] = 1.0        # 2 acceptable prior-task answers
base_prior[[1, 4]] += 3.0                     # the base model is already good at the prior task

def acc(logits, ok):
    return float(softmax(logits) @ ok)

# (1) THE MECHANISM. The on-policy policy gradient for an action is proportional to that
#     action's CURRENT probability, so RL cannot push mass onto behaviour the policy never
#     samples. The SFT cross-entropy gradient is not masked that way: a demonstration on a
#     near-zero-probability action still gets a full-size update.
p = softmax(base_new)
rare = int(np.argmin(p))
onehot = np.zeros(V); onehot[rare] = 1.0
g_rl = p * (onehot - p @ onehot)              # exact policy gradient, reward = hit `rare`
g_sft = onehot - p                            # cross-entropy gradient toward `rare`
#     The ratio is exactly p(a): RL's update on an action is that action's own probability
#     times SFT's. Here p(rare) = 0.0017, so RL moves it 575 times more slowly.
assert np.isclose(g_rl[rare], p[rare] * g_sft[rare], atol=1e-15)
assert abs(g_rl[rare]) < abs(g_sft[rare]) / 500

def finetune(mode, seed, goal=0.90, lr=0.1, steps=200_000):
    """Update ONE shared delta until the new task reaches `goal` accuracy."""
    delta = np.zeros(V)
    target = None
    if mode == "sft":                          # a random demonstration set over correct answers
        w = new_ok * np.random.default_rng(seed).random(V)
        target = w / w.sum()
    for _ in range(steps):
        pr = softmax(base_new + delta)
        if pr @ new_ok >= goal:
            break
        delta += lr * (pr * (new_ok - pr @ new_ok) if mode == "rl" else target - pr)
    return delta

d_rl = finetune("rl", seed=1)
sft = [finetune("sft", seed=s) for s in range(12)]

drift_rl = kl(base_new + d_rl, base_new)
keep_rl = acc(base_prior + d_rl, prior_ok) / acc(base_prior, prior_ok)
pts = [(kl(base_new + d, base_new), acc(base_prior + d, prior_ok) / acc(base_prior, prior_ok))
       for d in sft]
drift_sft = float(np.median([x[0] for x in pts]))
keep_sft = float(np.median([x[1] for x in pts]))

# (2) AT MATCHED NEW-TASK ACCURACY, RL drifts far less and keeps far more of the prior task.
assert acc(base_new + d_rl, new_ok) >= 0.90
assert all(acc(base_new + d, new_ok) >= 0.90 for d in sft)
assert drift_rl < drift_sft / 5
assert keep_rl > keep_sft * 2

# (3) DRIFT PREDICTS FORGETTING, across BOTH methods and across SFT targets. This is what
#     makes KL drift worth instrumenting: it ranks runs by how much they will forget, rather
#     than merely labelling the method. Rank correlation, since the relationship is not linear.
def rank(v):
    order = v.argsort(); r = np.empty(len(v)); r[order] = np.arange(len(v)); return r
allpts = pts + [(drift_rl, keep_rl)]
x = np.array([a for a, _ in allpts]); y = np.array([b for _, b in allpts])
spearman = float(np.corrcoef(rank(x), rank(y))[0, 1])
assert spearman < -0.6

# (4) ADVERSARIAL: RL is not magic. SFT toward the KL-minimal target beats RL on both
#     metrics. The advantage is that on-policy RL lands near that target by construction,
#     while a demonstration set has to be built that way on purpose.
w = new_ok * p                                # base probabilities restricted to correct answers
kl_min_target = w / w.sum()
delta = np.zeros(V)
for _ in range(200_000):
    pr = softmax(base_new + delta)
    if pr @ new_ok >= 0.90:
        break
    delta += 0.1 * (kl_min_target - pr)
assert kl(base_new + delta, base_new) < drift_rl

# (5) BOUNDARY: a policy already at the goal must not be updated at all.
assert np.allclose(finetune("rl", seed=1, goal=acc(base_new, new_ok) - 1e-12), 0.0)

print("base accuracy: new task", round(acc(base_new, new_ok), 4),
      "| prior task", round(acc(base_prior, prior_ok), 4))
print("RL   drift", round(drift_rl, 4), "| prior capability kept", round(keep_rl, 4))
print("SFT  drift", round(drift_sft, 4), "| prior capability kept", round(keep_sft, 4), "(medians of 12)")
print("SFT toward the KL-minimal target: drift", round(kl(base_new + delta, base_new), 4))
print("rank correlation, drift vs capability kept:", round(spearman, 3))
print("gradient on the rarest action (p =", round(float(p[rare]), 5), "): RL",
      format(abs(g_rl[rare]), ".2e"), "| SFT", format(abs(g_sft[rare]), ".2e"),
      "| ratio", round(float(abs(g_sft[rare] / g_rl[rare])), 1))

Executed output:

base accuracy: new task 0.6893 | prior task 0.5744
RL   drift 0.1573 | prior capability kept 0.9533
SFT  drift 1.3369 | prior capability kept 0.3157 (medians of 12)
SFT toward the KL-minimal target: drift 0.1511
rank correlation, drift vs capability kept: -0.698
gradient on the rarest action (p = 0.00174 ): RL 1.74e-03 | SFT 9.98e-01 | ratio 575.0

Four results carry the argument. The gradient ratio in check (1) is exact rather than empirical: RL's update on an action is that action's own probability times SFT's, so on the rarest action here, with probability 0.0017, RL moves 575 times more slowly. Check (2) is the consequence at matched new-task accuracy: RL drifts 0.157 against SFT's median 1.337 and keeps 95.3% of prior-task capability against SFT's 31.6%. Check (3) is the operational payoff, a rank correlation of -0.698 between drift and retained capability across both methods, which is why drift is worth logging as a first-class metric rather than reconstructing after the fact. Check (4) is the honest limit: SFT aimed at the KL-minimal target beats RL on drift, 0.151 against 0.157. RL's advantage is that it lands near that target by construction, not that no other route exists.

How to develop with it¶

Freeze a prior-capability eval suite before the first RL phase and never edit it. It has to be the same suite across every phase or the comparison is worthless. Include capabilities the reward does not measure at all, since those are the ones with no gradient defending them.
Log KL to the phase-entry checkpoint, not only to the original base. In a multi-phase pipeline the relevant reference for phase N is the checkpoint that entered phase N. Cumulative drift from the original base is a separate, also useful, number.
Keep beta available even if it is set to zero. Many verifiable-reward recipes run without a KL penalty (KL regularization). Reinstating it is the first lever when drift climbs, and it should not require a code change.
Order phases so the cheapest-to-recover domain runs last. Whatever regression survives will land hardest on the earliest phases.

How to maintain it¶

Four signals, watched together, distinguish healthy skill acquisition from capability destruction:

KL drift on the new task. The predictor. Rising steadily while new-task reward is flat means the run is paying for nothing.
Prior-capability evals, every phase. The ground truth. Run them at phase boundaries at minimum, and mid-phase for long runs.
Entropy of the next-token distribution. Collapse here means the policy has stopped sampling alternatives, which ends both exploration and the gradient masking that keeps RL near the base (GRPO variants).
Reward on the new task. Necessary but not sufficient. Interpreting it without the other three is how a run ships a model that is better at one benchmark and worse at everything unmeasured.

When drift climbs and prior evals fall, the ordered responses are: lower the learning rate, reinstate or raise the KL coefficient, mix a small fraction of prior-domain prompts back into the rollout mix, and, if the regression has already happened, recover it with on-policy distillation from a checkpoint that still had the capability. That last option is what Cascade RL uses.

How to run it in production¶

Budget phases separately. Each RL phase is a full rollout-dominated run with its own fleet sizing (rollout fleet sizing), not a continuation that can reuse the previous phase's capacity plan.
Checkpoint at every phase boundary and keep them. Recovery from a regression means going back to a checkpoint that still had the capability, which is only possible if it was kept.
Treat the eval suite as production infrastructure. It gates phase promotion, so it needs the same reliability as the training job (LLM evaluation harness, evaluation integrity).
Expect the drift-forgetting relationship to be a ranking, not a formula. The correlation in the executed model is -0.698, and published evidence is empirical. Use drift to decide which runs to investigate, not to predict an exact accuracy loss.

Failure modes¶

Assuming RL cannot forget. It forgets less. Cascade RL still needed a distillation mechanism to recover measured regressions.
Evaluating only the new task. Guarantees the regression is discovered by users.
A drifting eval suite. Editing the prior-capability benchmarks between phases makes the phase-over-phase comparison meaningless.
Reaching for RL to teach a behaviour the model never samples. The gradient masking that protects prior capability also blocks acquisition. Cold-start with SFT.
Treating offline preference training as RL for this purpose. The argument depends on on-policy data.
Entropy collapse mid-pipeline. Once the policy is nearly deterministic, later phases have almost no support to reweight and the pipeline stalls.
Confusing drift from the original base with drift from the phase entry point. They diverge in a multi-phase run, and only one of them explains the current phase's behaviour.

References¶

Shenfeld et al., RL's Razor: Why Online Reinforcement Learning Forgets Less: https://arxiv.org/abs/2509.04259
NVIDIA, Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation: https://arxiv.org/abs/2603.19220
NVIDIA, Nemotron-Cascade-2-30B-A3B model card: https://huggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B
Schulman, Approximating KL Divergence: http://joschu.net/blog/kl-approx.html
Cameron R. Wolfe, Continual Learning with RL for LLMs: https://cameronrwolfe.substack.com/p/rl-continual-learning
Cameron R. Wolfe, Reinforcement Learning for LLMs: The Complete Guide: https://cameronrwolfe.substack.com/p/llm-rl