Markdown

Recursive language models: training one policy to be both parent and child¶

Scope: an agent architecture where a model handles long or wide context by recursively calling itself on sub-problems, and the training method that makes a small model good at it. The distinctive part is not the recursion but the credit assignment: one policy plays both roles, and child rollouts are never scored independently. Non-recursive planning is in agent planning and reasoning; the underlying RL algorithm is in GRPO; alternative long-context handling is in ReContext.

Source: Daniel Kim and Rehaan Ahmad, "Reinforcing Recursive Language Models", published 2026-05-13 on the alphaXiv blog. This is a blog post reporting one experiment, not a peer-reviewed paper, and it is the only source for every number below. Nothing was reproduced: no training run was performed here. The Python block is this page's own model of the objective's structural properties, executed and asserted.

flowchart TB
  ROOT["Root RLM: sees the query and a context handle"]
  ROOT -->|"rlm_query_batched(prompts, context_list)"| C1["Child RLM"]
  ROOT --> C2["Child RLM"]
  ROOT --> C3["Child RLM"]
  C1 -->|"may recurse further"| G1["Grandchild RLM"]
  C1 --> RET1["returns a string"]
  C2 --> RET2["returns a string"]
  C3 --> RET3["returns a string"]
  RET1 --> ROOT
  RET2 --> ROOT
  RET3 --> ROOT
  ROOT --> FIN["FINAL(answer) or FINAL_VAR(name)"]

What it is¶

A Recursive Language Model does not receive its context as a prompt. It receives a handle to it and a Python REPL, and it decides what to look at and what to delegate. The action space reported for this work:

rlm_query(prompt, context=None) spawns one child RLM; rlm_query_batched(prompts, context_list=None) dispatches several in parallel.
FINAL(answer) submits a literal string; FINAL_VAR(variable_name) submits the contents of a REPL variable, so a large answer never has to pass through the model's own output tokens.
list_papers(), search(), extract_section(), get_paper_abstract() inspect the context without loading all of it.

The child is the same policy as the parent. There is no separate sub-agent model, no separate reward model for sub-tasks, and no hand-written decomposition strategy. The parent learns what to delegate and the child learns to answer delegated questions, from the same weights and the same gradient.

The training problem this creates is credit assignment. A child trajectory has no reward of its own: nothing scores "was this sub-answer good?" independently. The reported solution is deliberately blunt: children uniformly inherit their parent's advantage, A_{g,i} := A_g. The authors present this as an unbiased gradient estimator and note that finer-grained credit assignment could converge faster.

The second piece is normalization. The combined objective for a root and its k_g children is 1/G * sum_g [ L_root + (1/k_g) * sum_i L_child_i ] - beta * D_KL, and for arbitrary depth the recursion is L_subtree(y, A) = L_node(y, A) + (1/k_y) * sum_i L_subtree(y_i, A). The 1/k factor is what keeps a rollout that happened to spawn many children from dominating the update.

Why the two design choices matter¶

The executed model below makes both concrete, and the second one is more interesting than it first looks.

Inheritance removes an entire scoring problem. In the modelled group, 8 root rollouts spawn 27 children, and those 27 trajectories are trained on with zero additional reward evaluations. For a task where scoring means running a rubric or a verifier, that is the difference between a tractable and an intractable method.

Normalization removes a bias that correlates with nothing. Branchiness is a property of what the policy happened to do, not of how well it did. In the modelled group the branchiest root, with 12 children, has the worst advantage at -1.605. Unnormalized, it takes 37.1% of the gradient; normalized, 14.3%. Training without the 1/k factor would spend more than a third of the update amplifying the structure of the worst rollout in the group.

The depth behaviour is the cleanest result. Under the 1/k recursion, a tree's total gradient weight grows linearly with depth, not with node count: a depth-4 branching-6 tree has 1,555 nodes and contributes weight 5.0. Without that, deep trees would swamp shallow ones by three orders of magnitude and the policy would be trained almost entirely on whatever it did at depth.

When to use it (and when not)¶

Consider a recursive architecture when:

Context is wide rather than long. The reported task is evidence selection across groups of up to 10 papers, which decomposes naturally into per-document sub-queries. Independent sub-problems are what recursion exploits.
Latency matters more than token cost. Children run in parallel, so wall clock tracks tree depth while tokens track tree size. This is the entire economic case.
You can run a small model locally. The reported result is a 4B model on one node beating an API model on latency by roughly an order of magnitude.

Do not use it when:

The task does not decompose. A genuinely sequential reasoning chain gains nothing and pays the tree's token multiple.
Tokens are the binding cost. The executed model puts a depth-2 branching-6 tree at 43x the tokens of a single pass.
You need the strongest possible answer. In the reported evaluation the fine-tuned 4B model scores about 0.58 average rubric against Claude Sonnet's 0.607. It trades a little quality for a lot of latency.

How the objective behaves¶

Run: python3 rlm.py.

import numpy as np

rng = np.random.default_rng(3)

# --- 1. GRPO advantage over a group, then inherited by children ----------
# Root rollouts are scored; the group-normalized advantage is the RLM's only
# reward signal. Children inherit their parent's advantage verbatim.
G = 8                                   # rollouts per prompt
rewards = np.array([0.9, 0.2, 0.6, 0.6, 0.1, 0.8, 0.4, 0.5])
assert len(rewards) == G


def grpo_advantage(r: np.ndarray) -> np.ndarray:
    return (r - r.mean()) / (r.std() + 1e-8)


A = grpo_advantage(rewards)
print(f"[1] rewards    {np.array2string(rewards, precision=2)}")
print(f"[1] advantages {np.array2string(A, precision=3)}")
assert np.isclose(A.mean(), 0.0, atol=1e-6), "group-normalized advantages are centered"
# Children of rollout g all take A_g. No separate reward is computed for them,
# which is what removes the need to score sub-agent trajectories.
children_per_root = np.array([1, 5, 0, 3, 12, 2, 0, 4])
inherited = [np.full(k, A[g]) for g, k in enumerate(children_per_root)]
print(f"[1] children per root {children_per_root.tolist()} "
      f"-> {int(children_per_root.sum())} child rollouts, 0 extra reward evaluations")
assert all(np.allclose(c, A[g]) for g, c in enumerate(inherited)), "uniform inheritance"

# --- 2. Why the 1/k normalization is load-bearing ------------------------
# Without it, a root that happens to spawn many children contributes
# proportionally more gradient, independent of whether it was a good rollout.
print()


def root_weight(k: np.ndarray, normalize: bool) -> np.ndarray:
    """Total gradient weight each root contributes: its own node plus children.

    With 1/k normalization the children sum to weight 1 regardless of how many
    there are. A root with no children contributes only its own node.
    """
    has_children = (k > 0).astype(float)
    return 1.0 + (has_children if normalize else k.astype(float))


w_norm = root_weight(children_per_root, True)
w_raw = root_weight(children_per_root, False)
print(f"[2] normalized 1/k : weights {np.array2string(w_norm, precision=2)} "
      f"(max/min = {w_norm.max() / w_norm.min():.2f})")
print(f"[2] no normalization: weights {np.array2string(w_raw, precision=2)} "
      f"(max/min = {w_raw.max() / w_raw.min():.2f})")
assert w_norm.max() / w_norm.min() == 2.0, "childless roots weigh 1, branching roots weigh 2"
assert w_raw.max() / w_raw.min() > 6.0, "without it, branchy roots dominate"

# The bias this removes is real: branchiness correlates with nothing useful.
# Here the branchiest root (12 children) has a BELOW-average advantage.
worst = int(np.argmax(children_per_root))
print(f"[2] root {worst} spawned the most children ({children_per_root[worst]}) "
      f"and has advantage {A[worst]:+.3f}")
share_raw = w_raw[worst] / w_raw.sum()
share_norm = w_norm[worst] / w_norm.sum()
print(f"[2] its gradient share: {100 * share_raw:.1f}% unnormalized vs "
      f"{100 * share_norm:.1f}% normalized ({share_raw / share_norm:.2f}x over-weighted)")
assert A[worst] < 0, "the branchiest rollout here is a below-average one"
assert share_raw / share_norm > 2.0, "unnormalized training over-weights it several fold"

# --- 3. Recursive subtree loss at arbitrary depth ------------------------
# L_subtree(y, A) = L_node(y, A) + (1/k_y) * sum_i L_subtree(y_i, A)
print()


def subtree_weight(depth: int, branching: int) -> float:
    """Total gradient weight of a full tree under the 1/k recursion."""
    assert depth >= 0 and branching >= 1
    if depth == 0:
        return 1.0
    return 1.0 + subtree_weight(depth - 1, branching)


for d in range(5):
    w = subtree_weight(d, branching=6)
    nodes = sum(6 ** i for i in range(d + 1))
    print(f"[3] depth {d}: {nodes:>6,} nodes in the tree, total gradient weight {w:.1f}")
assert subtree_weight(4, 6) == 5.0, "weight grows linearly in depth, not in node count"
# This is the key property: the 1/k recursion makes total gradient weight scale
# with DEPTH, not with the exponentially many nodes. Without it, a depth-4
# tree with branching 6 would contribute 1,555x the weight of a leaf.
naive_nodes = sum(6 ** i for i in range(5))
print(f"[3] a depth-4 branching-6 tree has {naive_nodes:,} nodes but contributes "
      f"weight {subtree_weight(4, 6):.1f}, not {naive_nodes:,}")
assert naive_nodes > 300 * subtree_weight(4, 6)

# --- 4. Latency: why decomposition can beat a bigger model --------------
# Children run in parallel (rlm_query_batched), so wall-clock tracks tree DEPTH
# while token cost tracks tree SIZE.
print()
SMALL_TOK_S, BIG_TOK_S = 210.0, 38.0   # tokens/s for a 4B local vs a large API model
TOK_PER_NODE = 700


def wall_clock(depth: int, tok_per_s: float) -> float:
    return (depth + 1) * TOK_PER_NODE / tok_per_s


def total_tokens(depth: int, branching: int) -> int:
    return sum(branching ** i for i in range(depth + 1)) * TOK_PER_NODE


for d in (1, 2, 3):
    print(f"[4] depth {d}: small-model tree {wall_clock(d, SMALL_TOK_S):5.1f} s wall clock, "
          f"{total_tokens(d, 6):>8,} tokens total")
print(f"[4] one large-model pass over the same budget: "
      f"{wall_clock(0, BIG_TOK_S):5.1f} s for {TOK_PER_NODE:,} tokens")
assert wall_clock(2, SMALL_TOK_S) < wall_clock(0, BIG_TOK_S) * 1.2
# The tradeoff is explicit: parallel children buy latency and spend tokens.
print(f"[4] depth 2 costs {total_tokens(2, 6) / TOK_PER_NODE:.0f}x the tokens of a single "
      f"pass to save wall clock; that trade only pays when tokens are cheap and "
      f"latency is not")
assert total_tokens(2, 6) > 40 * TOK_PER_NODE

print("\nAll assertions passed.")

Executed output:

[1] rewards    [0.9 0.2 0.6 0.6 0.1 0.8 0.4 0.5]
[1] advantages [ 1.507 -1.216  0.34   0.34  -1.605  1.118 -0.438 -0.049]
[1] children per root [1, 5, 0, 3, 12, 2, 0, 4] -> 27 child rollouts, 0 extra reward evaluations

[2] normalized 1/k : weights [2. 2. 1. 2. 2. 2. 1. 2.] (max/min = 2.00)
[2] no normalization: weights [ 2.  6.  1.  4. 13.  3.  1.  5.] (max/min = 13.00)
[2] root 4 spawned the most children (12) and has advantage -1.605
[2] its gradient share: 37.1% unnormalized vs 14.3% normalized (2.60x over-weighted)

[3] depth 0:      1 nodes in the tree, total gradient weight 1.0
[3] depth 1:      7 nodes in the tree, total gradient weight 2.0
[3] depth 2:     43 nodes in the tree, total gradient weight 3.0
[3] depth 3:    259 nodes in the tree, total gradient weight 4.0
[3] depth 4:  1,555 nodes in the tree, total gradient weight 5.0
[3] a depth-4 branching-6 tree has 1,555 nodes but contributes weight 5.0, not 1,555

[4] depth 1: small-model tree   6.7 s wall clock,    4,900 tokens total
[4] depth 2: small-model tree  10.0 s wall clock,   30,100 tokens total
[4] depth 3: small-model tree  13.3 s wall clock,  181,300 tokens total
[4] one large-model pass over the same budget:  18.4 s for 700 tokens
[4] depth 2 costs 43x the tokens of a single pass to save wall clock; that trade only pays when tokens are cheap and latency is not

All assertions passed.

Two cautions follow from that output.

Uniform inheritance is unbiased, not free. Every child of a good rollout is reinforced, including the ones that returned nothing useful, and every child of a bad rollout is penalized, including the ones that were right. The estimator is unbiased in expectation and high-variance in practice, which is exactly what the authors' own remark about finer-grained credit assignment concedes. Expect slow convergence and budget rollouts accordingly.

The latency win is a token trade, and the trade has a narrow window. Depth 2 buys a wall clock of 10.0 s against 18.4 s for one large-model pass, at 43x the tokens. That pays when you own the GPUs and the model is small, and stops paying the moment you are billed per token or the model is large enough that 43x is real money.

How to run it¶

Batch the children. rlm_query_batched is what makes wall clock scale with depth instead of node count. Spawning children one at a time discards the only advantage the architecture has.
Return by reference, not by value. FINAL_VAR exists so a large answer does not have to be regenerated token by token through the parent. Use it for anything bigger than a sentence.
Cap depth explicitly. The objective supports arbitrary depth; your budget does not. Node count is exponential in depth even though gradient weight is linear.
Keep the KL term. The reported objective retains beta * D_KL against a reference. A policy that can spawn copies of itself has an unusually rich space of degenerate behaviours to fall into, such as infinite delegation.
Watch tree shape as a training metric. Mean children per root, mean depth, and the fraction of rollouts that never recurse tell you whether the policy is learning to decompose or learning to avoid it.

How to maintain it¶

Re-check the branching distribution after any reward change. The 1/k normalization bounds the bias from branchiness but does not stop the policy from learning a degenerate branching habit if the reward happens to favour one.
Re-measure the latency claim on your own serving stack. The 7-second figure is for a 4B model on a single 8xH200 node with a particular tree shape.
Re-validate against a non-recursive baseline of the same model. The interesting comparison is not against a large API model; it is against the same 4B model given the whole context directly.

Failure modes¶

Degenerate non-recursion. The policy learns to answer directly and never spawn children, because recursion is harder and the reward does not require it. Detectable only by tracking tree shape, not by watching the loss.
Runaway delegation. The opposite failure: the policy spawns children that spawn children, exhausting the depth cap on every query.
Gradient capture by branchy rollouts. The specific failure the 1/k factor prevents, quantified above at 2.60x over-weighting for one rollout in an eight-rollout group.
High-variance training from uniform inheritance. Good children of bad parents are punished. Shows up as noisy learning curves rather than as an error.
Token blowout at depth. Exponential node count meets a per-token bill.
Reading the headline comparison as a quality win. The reported 0.58 versus 0.607 is a small quality deficit bought back with a large latency gain, and should be quoted that way.

Open questions and validation¶

The source is a blog post reporting a single experiment: Qwen3.5-4B with SFT pretraining, 1,000 synthetically generated queries over groups of up to 10 papers, a single 8xH200 node, batch size 16, 8 samples per prompt. There are no seeds, error bars, or ablations reported, and no independent replication exists that was located here.
The training rubric score is reported to improve from 0.3 to 0.6 on the training set, which is the same scale as the 0.58 evaluation figure; whether the evaluation set is disjoint in distribution is not established by the source.
The claim that uniform advantage inheritance is an unbiased estimator is stated by the authors and not verified here.
Whether the approach transfers beyond evidence selection over document groups, which is unusually decomposable, is untested.
No code release was located for this work.

References¶

Kim and Ahmad, "Reinforcing Recursive Language Models", alphaXiv blog, 2026-05-13: https://www.alphaxiv.org/blog/reinforcement-learning-for-rlms
Shao et al., "DeepSeekMath" (the GRPO objective this extends), arXiv 2402.03300: https://arxiv.org/abs/2402.03300