Markdown

Hierarchical agent decomposition¶

Scope: the orchestrator-plus-sub-agent pattern for tasks that are too long, too noisy, and too expensive for a single agent loop, told through Matryoshka Agent on machine learning engineering tasks. This page covers the three-layer split and why the Tool layer is the load-bearing part, the Solution Refinement Tree that makes training data affordable when every rollout costs a real training run, the branch-level preference labelling that avoids punishing slow-starting strategies, the ranking objective and its relationship to DPO, and an honest read of which component the published gains actually come from. It is a specific instantiation of the agent harness architecture and a sibling of multi-agent collaboration; the context-management alternative is agentic context management.

The numpy block below is executed and asserted in this page (Python 3.11, numpy 2.x). It validates the structural claims (context boundedness, branch-level versus immediate labelling, pair counts, the objective's equivalence to DPO at two candidates) and re-checks the published HumanRank table for what it actually shows. The context-size constants are illustrative, and are labelled as such; the HumanRank scores are the paper's. No agent was run.

What it is¶

Matryoshka Agent decomposes agentic problem solving into three coordinated layers:

The Orchestrator is the persistent decision layer. It maintains a compact, score-annotated view of the solution process and decides what to refine next. Critically, it "does not directly accumulate raw code, execution logs, debugging traces, or full Sub-Agent conversations." Each turn it selects a previous attempt to refine, writes a concrete implementation instruction, and may tag other previous attempts as references.
Sub-Agents are the execution layer. Each is instantiated with a fresh context containing the task specification, the selected previous attempt, the Orchestrator's instruction, and the tagged references. It does not inherit the Orchestrator's history or any other Sub-Agent's context. It runs a bounded write-execute-debug loop until it produces a valid submission or exhausts its debugging budget, then summarises.
The Tool interface mediates. It translates Orchestrator decisions into Sub-Agent contexts, manages the Sub-Agent lifecycle on allocated compute, and converts the Sub-Agent trajectory back into a compact response containing the evaluation score (if the submission was valid) and a structured summary.

The design is usually described as a hierarchy, but the mechanism is an information boundary. The Tool layer "enforce[s] the information boundary that allows the Orchestrator to reason over long-horizon progress without accumulating noisy low-level execution history." Everything else follows from that.

Why use it¶

Context stops being a function of run length. In a monolithic agent, every execution log and debugging trace lands in the same window, so a long run eventually cannot see its own beginning. Here the Orchestrator accumulates decisions and scores, and each Sub-Agent starts fresh, so peak context is bounded by the larger of two fixed quantities rather than by turn count.
Strategy and execution get optimised separately. The Orchestrator is trained with ranking-based RL on which refinement direction to pursue; Sub-Agents are improved from successful execution trajectories. These are different skills and they want different data.
Exploration becomes parallel and isolated. Multiple Sub-Agents can be launched by parallel tool calls "when parallel exploration is desired, while their contexts and execution histories remain isolated from one another." No cross-contamination between branches.
A small model can do the strategic layer. This is the headline: with training, Qwen3-4B-Instruct as Orchestrator reaches performance comparable to o4-mini in the same role.
The failure mode of one attempt is contained. A Sub-Agent that fails to produce a valid submission returns a status and diagnostic summary rather than a score, and the run continues. In a monolithic loop that failure would pollute the only context there is.

When to use it (and when not)¶

Use it when the task is long-horizon, feedback-driven, and expensive per step: ML engineering, iterative optimisation against a metric, anything where you improve a solution rather than complete a single objective.

Do not use it when:

The task is short. The hierarchy costs you an extra model call per decision and an information boundary that hides detail. Below the point where context becomes a problem, it is pure overhead.
The Orchestrator genuinely needs the details. The boundary is lossy by design. If diagnosing what to do next requires reading a stack trace, a summary will not do, and you are choosing the wrong pattern.
You cannot afford the sub-agent model. Read the results section: the published numbers say the sub-agent's capability dominates the orchestrator's. A cheap hierarchy with a weak execution layer performs badly regardless of orchestration.
You have no scalar score per attempt. The whole training paradigm rests on an environment that returns a task-specific metric. Without it there is nothing to build preferences from.

Architecture¶

flowchart TB
  TASK["task instance"] --> ROOT["root state<br/>(task spec, constraints)"]
  ROOT --> ORCH
  subgraph DEC["Decision layer (persistent)"]
    ORCH["Orchestrator<br/>score-annotated history of<br/>(instruction, summary, score)<br/>NO raw code or logs"]
  end
  ORCH -->|"tool call:<br/>1. which attempt to refine<br/>2. instruction<br/>3. tagged references"| TOOL
  subgraph MED["Tool interface (the boundary)"]
    TOOL["build fresh Sub-Agent context<br/>launch + monitor lifecycle<br/>compact the trajectory back"]
  end
  TOOL -->|"fresh context"| SA1["Sub-Agent 1"]
  TOOL -->|"fresh context"| SA2["Sub-Agent 2 (parallel)"]
  subgraph EXE["Execution layer (disposable, isolated)"]
    SA1 --> LOOP1["write code -> execute -> debug<br/>until valid submission<br/>or budget exhausted"]
    SA2 --> LOOP2["write code -> execute -> debug"]
  end
  LOOP1 --> ENV["environment E<br/>validate + score"]
  LOOP2 --> ENV
  ENV -->|"score, or failure status<br/>+ structured summary"| TOOL
  TOOL -->|"compact response only"| ORCH

How it works, validated¶

import numpy as np

# ============ 1. why the Orchestrator context stays bounded ==================
# Token counts here are ILLUSTRATIVE, chosen to show the shape of the two
# curves. The structural claim (one grows with turns, the other does not) is
# the paper's; the constants are not.
def monolithic_context(turns, per_turn_raw=9_000, task_spec=2_000):
    """A single agent accumulates code, execution logs and debugging traces."""
    return task_spec + turns * per_turn_raw


def orchestrator_context(turns, per_turn_summary=260, task_spec=2_000):
    """The Orchestrator sees only (instruction, compact summary, score) per turn."""
    return task_spec + turns * per_turn_summary


WINDOW = 128_000
mono = [monolithic_context(t) for t in range(1, 41)]
orch = [orchestrator_context(t) for t in range(1, 41)]
assert mono[12] < WINDOW <= mono[13]          # monolithic fills a 128k window by turn 14
assert orch[-1] < WINDOW / 8                  # hierarchical is nowhere near it
assert max(orch) < min(m for m in mono if m >= WINDOW)
# The Sub-Agent context does NOT grow with the run: it is rebuilt fresh each time
# from (task spec, selected previous attempt, instruction, tagged references).
def subagent_context(task_spec=2_000, prev_attempt=9_000, instruction=400, refs=6_000):
    return task_spec + prev_attempt + instruction + refs


assert all(subagent_context() == subagent_context() for _ in range(5))
assert subagent_context() < WINDOW
# So peak context is bounded by max(orchestrator, subagent), not by turn count.
assert max(max(orch), subagent_context()) < WINDOW


# ============ 2. branch-level returns vs immediate scores ====================
class Node:
    def __init__(self, score, children=None):
        self.score = score
        self.children = children or []


def downstream_return(node):
    """R(c) = max over the subtree rooted at c (the paper's labelling)."""
    return max([node.score] + [downstream_return(k) for k in node.children])


# A slow starter: branch B's first attempt scores worse than A's, but B's
# subtree finds the better solution. This is the exact case the paper says
# immediate-score labelling gets wrong.
A = Node(0.62, [Node(0.63), Node(0.61)])
B = Node(0.55, [Node(0.58), Node(0.81, [Node(0.84)])])
assert A.score > B.score                                   # immediate says A
assert downstream_return(B) > downstream_return(A)         # downstream says B
assert downstream_return(A) == 0.63 and downstream_return(B) == 0.84
immediate_pick = "A" if A.score > B.score else "B"
branch_pick = "A" if downstream_return(A) > downstream_return(B) else "B"
assert immediate_pick == "A" and branch_pick == "B"
assert immediate_pick != branch_pick                       # the labels disagree

# When there is no slow starter the two labellings agree, so branch-level
# labelling is a strict generalisation, not a different objective.
C = Node(0.70, [Node(0.72)])
D = Node(0.40, [Node(0.45)])
assert (C.score > D.score) == (downstream_return(C) > downstream_return(D))


# ============ 3. how many preference pairs a binary tree yields ==============
def internal_nodes(depth):
    """Binary expansion: 2 children per non-leaf. Pairs = internal nodes."""
    return 2 ** depth - 1


def leaves(depth):
    return 2 ** depth


for d in (1, 2, 3, 4):
    assert internal_nodes(d) == 2 ** d - 1
    assert leaves(d) == 2 ** d
# At depth 4: 15 common-prefix preference pairs from 16 executed leaves, versus
# a single (winner, rest) split under global winner-take-all labelling.
assert internal_nodes(4) == 15 and leaves(4) == 16
assert internal_nodes(4) > 1
# Each pair compares siblings that share a prefix, so the signal isolates the
# one decision that differs, rather than blaming the whole trajectory.
assert internal_nodes(4) / leaves(4) > 0.9


# ============ 4. the ranking NCE reduces to DPO at |C(u)| = 2 ================
def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))


def implicit_score(logp_theta, logp_ref, beta):
    """r_theta(s, tau) = beta * log(pi_theta / pi_ref), Eq. 3."""
    return beta * (logp_theta - logp_ref)


def ranking_nce(scores_pref_first):
    """-log softmax over candidates, with the preferred continuation at index 0."""
    s = np.asarray(scores_pref_first, dtype=float)
    return float(-(s[0] - np.log(np.exp(s).sum())))


BETA = 0.1
lp_t = np.array([-11.0, -13.5])     # updated policy logprobs (preferred, rejected)
lp_r = np.array([-12.0, -12.2])     # reference policy logprobs
r = implicit_score(lp_t, lp_r, BETA)
nce = ranking_nce(r)
dpo = float(-np.log(sigmoid(r[0] - r[1])))
assert abs(nce - dpo) < 1e-12, (nce, dpo)     # identical for two candidates

# With more than two candidates it is a genuine softmax ranking, and adding a
# strong distractor raises the loss (there is more to be discriminated against).
r3 = np.array([r[0], r[1], implicit_score(-10.5, -12.4, BETA)])
assert ranking_nce(r3) > ranking_nce(r)
# The loss falls when the preferred continuation pulls ahead of the reference.
better = implicit_score(np.array([-10.0, -13.5]), lp_r, BETA)
assert ranking_nce(better) < nce
# beta = 0 destroys the signal entirely: every score collapses to 0.
assert abs(ranking_nce(implicit_score(lp_t, lp_r, 0.0)) - np.log(2)) < 1e-12


# ============ 5. the published HumanRank table ===============================
HR = {
    "Matryoshka / Qwen3-4B (self)":        0.1878,
    "Matryoshka / o4-mini orch":           0.4832,
    "Matryoshka / o4-mini + self":         0.5465,
    "Matryoshka / Qwen3-4B + o4-mini":     0.4613,
    "Matryoshka / Qwen3-4B-SFT + o4-mini": 0.5061,
    "Matryoshka / Qwen3-4B-RL + o4-mini":  0.5360,
}
# Training the small Orchestrator is what closes the gap, in two clear steps.
sft_gain = HR["Matryoshka / Qwen3-4B-SFT + o4-mini"] - HR["Matryoshka / Qwen3-4B + o4-mini"]
rl_gain = HR["Matryoshka / Qwen3-4B-RL + o4-mini"] - HR["Matryoshka / Qwen3-4B-SFT + o4-mini"]
assert sft_gain > 0 and rl_gain > 0
assert abs(sft_gain - 0.0448) < 1e-4 and abs(rl_gain - 0.0299) < 1e-4
assert sft_gain > rl_gain              # most of the gain is from SFT, not RL
# The trained 4B Orchestrator overtakes an o4-mini Orchestrator driving the same
# sub-agents, and lands within ~1 point of the all-o4-mini configuration.
assert HR["Matryoshka / Qwen3-4B-RL + o4-mini"] > HR["Matryoshka / o4-mini orch"]
gap = HR["Matryoshka / o4-mini + self"] - HR["Matryoshka / Qwen3-4B-RL + o4-mini"]
assert 0 < gap < 0.011, gap
# But the sub-agent matters more than the orchestrator: swapping o4-mini
# sub-agents for the 4B model costs far more than any orchestrator training won.
subagent_cost = HR["Matryoshka / Qwen3-4B + o4-mini"] - HR["Matryoshka / Qwen3-4B (self)"]
assert subagent_cost > sft_gain + rl_gain
assert subagent_cost > 3 * (sft_gain + rl_gain)

print("all Matryoshka Agent assertions passed")
print("  monolithic context blows a 128k window at turn:",
      next(i + 1 for i, v in enumerate(mono) if v >= WINDOW))
print("  orchestrator context after 40 turns:", orch[-1], "tokens")
print("  preference pairs at tree depth 4:", internal_nodes(4), "from", leaves(4), "leaves")
print("  ranking-NCE == DPO at |C|=2:", round(nce, 6), "==", round(dpo, 6))
print("  HumanRank gains  SFT / RL:", round(sft_gain, 4), "/", round(rl_gain, 4))
print("  cost of downgrading sub-agents:", round(subagent_cost, 4))

Executed output:

all Matryoshka Agent assertions passed
  monolithic context blows a 128k window at turn: 14
  orchestrator context after 40 turns: 12400 tokens
  preference pairs at tree depth 4: 15 from 16 leaves
  ranking-NCE == DPO at |C|=2: 0.584745 == 0.584745
  HumanRank gains  SFT / RL: 0.0448 / 0.0299
  cost of downgrading sub-agents: 0.2735

The information boundary is the whole design¶

Block 1 makes the structural point concrete. A monolithic agent's context is spec + turns * raw_per_turn, which is linear in run length and eventually exceeds any window. The Orchestrator's is spec + turns * summary_per_turn with a summary one or two orders of magnitude smaller, and the Sub-Agent's is constant, because it is rebuilt from scratch each time. Peak context is therefore max(orchestrator, sub-agent), and neither term grows without bound.

The exact constants are illustrative and the block says so. What is not illustrative is the shape: one curve has a turn term that dominates and the other does not.

Branch-level returns exist to protect slow starters¶

The refinement tree is labelled with downstream returns, R(c) = max over subtree(c) of r(v), not with the immediate score of each child. Block 2 constructs the case that motivates it: branch A opens at 0.62 and peaks at 0.63; branch B opens at 0.55 but its subtree reaches 0.84. Immediate-score labelling prefers A and teaches the Orchestrator to avoid the direction that actually wins. Downstream-return labelling prefers B. The block asserts the two labellings genuinely disagree here, and also that they agree when there is no slow starter, so this is a strict generalisation rather than a different objective.

The paper's own framing of the two comparisons this avoids is worth quoting directly: branch-level returns "do not penalize a refinement direction merely because its first attempt has a lower score but later improves substantially", and unlike global winner-take-all labelling the tree "provides multiple common-prefix preference pairs rather than treating all non-winning trajectories as equally negative."

Block 3 quantifies the second half. Binary expansion to depth 4 executes 16 leaves and yields 15 preference pairs, one per internal node, each comparing two siblings that share an identical prefix. Winner-take-all over the same 16 rollouts yields one split. Since every leaf is a real training run in an MLE environment, that ratio is the difference between an affordable training loop and an unaffordable one.

The objective is DPO generalised to a ranking¶

The Orchestrator loss parameterises the trajectory score as the implicit reward r_theta(s, tau) = beta * log(pi_theta / pi_ref) and plugs it into a softmax ranking over candidate continuations. Block 4 confirms that with two candidates this is exactly the DPO loss, to floating-point equality. That is a useful thing to know when implementing: you can start from a DPO trainer, and the only structural change is generalising the pairwise sigmoid to a softmax over |C(u)| candidates.

Two behaviours the block also pins down: adding a strong distractor raises the loss (more to discriminate against), and beta = 0 collapses every score to zero so the loss becomes log |C| and carries no signal at all. The reference policy is the previous online iteration's policy pi^(m), not a frozen SFT checkpoint, which is what makes this an online iterative scheme rather than one-shot preference tuning.

What the results actually show¶

The published HumanRank table supports the headline, and also a caveat that is easy to miss.

Configuration	HumanRank (All)
Matryoshka, Qwen3-4B orchestrator and sub-agents	0.1878
Matryoshka, Qwen3-4B orchestrator + o4-mini sub-agents	0.4613
Matryoshka, o4-mini orchestrator	0.4832
Matryoshka, Qwen3-4B-SFT orchestrator + o4-mini sub-agents	0.5061
Matryoshka, Qwen3-4B-RL orchestrator + o4-mini sub-agents	0.5360
Matryoshka, o4-mini orchestrator and sub-agents	0.5465

The headline holds: a trained 4B Orchestrator (0.5360) beats an o4-mini Orchestrator driving the same sub-agents (0.4832) and lands within about one point of the all-o4-mini configuration (0.5465). Block 5 asserts each of those.

The caveat is the size of the effects. SFT contributes +0.0448 and RL a further +0.0299, so most of the orchestrator training gain comes from SFT, not from the ranking RL. Meanwhile, downgrading the sub-agents from o4-mini to Qwen3-4B costs 0.2735, more than three times everything orchestrator training buys. The practical ordering is therefore: get the execution layer right first, then train the orchestrator. A cheap orchestrator on a strong execution layer is a good trade; the reverse is not.

How to develop with it¶

The online loop alternates two phases:

Collect. For each task instance, build a Solution Refinement Tree from the root state using the current Orchestrator policy. At each non-leaf node, sample two alternative refinement instructions from the same parent state and execute both through Sub-Agents. Binary expansion is chosen explicitly "for tractable data collection".
Optimise. Label each internal node's two branches by downstream return, extract the common-prefix preference pairs, and update the Orchestrator with the ranking objective. Successful execution trajectories separately feed Sub-Agent improvement.

Implementation notes that follow from the design:

Keep the Tool response schema fixed and small. It is the contract between the two layers. Every field you add to it erodes the boundary that makes the pattern work.
Return failures as structured status, not as scores. An invalid submission yields a status and diagnostic summary; do not coerce it into a low score, or downstream-return labelling will treat a crash and a genuinely bad solution as the same event.
Let the Orchestrator tag references explicitly. Reference selection is how information crosses the boundary deliberately rather than by accumulation.
Isolate sibling Sub-Agents completely. Parallel exploration only produces independent evidence if the branches cannot see each other.

How to run it in production¶

Budget per Sub-Agent, not per run. The debugging loop is bounded; that bound is your cost control and your latency bound at the same time.
The parallelism is real. Multiple tool calls launch multiple Sub-Agents concurrently on allocated compute. Size the pool for the branching factor you intend, or the tree collapses to sequential.
Meter the two layers separately. They have different models, different token profiles, and different failure signatures. An Orchestrator that is looping (repeatedly refining the same attempt with no score movement) looks nothing like a Sub-Agent that cannot produce a valid submission.
Log the tree, not just the winner. The tree is the training data. Discarding non-winning branches throws away the common-prefix pairs that make the next iteration cheaper.

Failure modes¶

Boundary leakage. Once raw logs start flowing back to the Orchestrator "just for debugging", its context grows with turns again and the pattern's main benefit is gone.
Immediate-score labelling. Trains the Orchestrator away from strategies that start slow and finish well; block 2 is the counterexample.
Winner-take-all labelling. Wastes most of the executed rollouts and treats every non-winner as equally bad.
Weak sub-agents. Dominates everything else in the published table. Orchestrator training cannot compensate.
beta too small. The implicit-reward scores collapse and the ranking loss becomes constant.
Stale reference policy. The objective's reference is the previous online iteration. Pinning it to a fixed checkpoint turns the scheme into ordinary offline preference tuning and loses the online correction.
Score-free environments. Without a scalar metric per attempt there are no downstream returns and no preferences, so the training paradigm does not apply even though the runtime pattern still would.
Over-applying the hierarchy. On short tasks the extra decision hop and the lossy boundary cost more than the context savings are worth.

References¶

Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineering: https://arxiv.org/abs/2607.25090
MLE-bench (the ML engineering task family): https://arxiv.org/abs/2410.07095
DPO (the pairwise special case of the ranking objective): https://arxiv.org/abs/2305.18290
InfoNCE / noise-contrastive ranking: https://arxiv.org/abs/1807.03748