Markdown

Hierarchical agent decomposition¶

Scope: the orchestrator-plus-sub-agent pattern for tasks that are too long, too noisy, and too expensive for a single agent loop, told through Matryoshka Agent on machine learning engineering tasks. This page covers the three-layer split and why the Tool layer is the load-bearing part, the Solution Refinement Tree that makes training data affordable when every rollout costs a real training run, the branch-level preference labelling that avoids punishing slow-starting strategies, the ranking objective and its relationship to DPO, and an honest read of which component the published gains actually come from. It is a specific instantiation of the agent harness architecture and a sibling of multi-agent collaboration; the context-management alternative is agentic context management.

The numpy block below is executed and asserted in this page (Python 3.11, numpy 2.x). It validates context growth and fresh Sub-Agent boundaries, branch-level versus immediate labelling, executed-attempt and pair counts, the objective's equivalence to DPO at two candidates, and the published HumanRank labels. Context slopes and HumanRank scores are the paper's; the illustrative Sub-Agent size is labelled. No agent was run.

What it is¶

Matryoshka Agent decomposes agentic problem solving into three coordinated layers:

The Orchestrator is the persistent decision layer. It maintains a compact, score-annotated view of the solution process and decides what to refine next. Critically, it "does not directly accumulate raw code, execution logs, debugging traces, or full Sub-Agent conversations." Each turn it selects a previous attempt to refine, writes a concrete implementation instruction, and may tag other previous attempts as references.
Sub-Agents are the execution layer. Each is instantiated with a fresh context containing the task specification, the selected previous attempt, the Orchestrator's instruction, and the tagged references. It does not inherit the Orchestrator's history or any other Sub-Agent's context. It runs a bounded write-execute-debug loop until it produces a valid submission or exhausts its debugging budget, then summarises.
The Tool interface mediates. It translates Orchestrator decisions into Sub-Agent contexts, manages the Sub-Agent lifecycle on allocated compute, and converts the Sub-Agent trajectory back into a compact response containing the evaluation score (if the submission was valid) and a structured summary.

The design is usually described as a hierarchy, but the mechanism is an information boundary. The Tool layer "enforce[s] the information boundary that allows the Orchestrator to reason over long-horizon progress without accumulating noisy low-level execution history." Everything else follows from that.

Why use it¶

Execution history grows outside the persistent decision context. Each Sub-Agent starts fresh, so raw code, logs, and debugging traces do not accumulate in the Orchestrator. The Orchestrator still grows linearly with decisions, but the paper measures a lower slope than the monolithic baseline.
Strategy and execution get optimised separately. The Orchestrator is trained with ranking-based RL on which refinement direction to pursue; Sub-Agents are improved from successful execution trajectories. These are different skills and they want different data.
Exploration becomes parallel and isolated. Multiple Sub-Agents can be launched by parallel tool calls "when parallel exploration is desired, while their contexts and execution histories remain isolated from one another." No cross-contamination between branches.
A small model can do the strategic layer. This is the headline: with training, Qwen3-4B-Instruct as Orchestrator reaches performance comparable to o4-mini in the same role.
The failure mode of one attempt is contained. A Sub-Agent that fails to produce a valid submission returns a status and diagnostic summary rather than a score, and the run continues. In a monolithic loop that failure would pollute the only context there is.

When to use it (and when not)¶

Use it when the task is long-horizon, feedback-driven, and expensive per step: ML engineering, iterative optimisation against a metric, anything where you improve a solution rather than complete a single objective.

Do not use it when:

The task is short. The hierarchy costs you an extra model call per decision and an information boundary that hides detail. Below the point where context becomes a problem, it is pure overhead.
The Orchestrator genuinely needs the details. The boundary is lossy by design. If diagnosing what to do next requires reading a stack trace, a summary will not do, and you are choosing the wrong pattern.
You cannot afford capable Sub-Agents. The paper reports that small models struggle in the execution role and does not include Qwen3-4B+self in its main table. Do not infer a numerical Sub-Agent effect from configurations that also change the framework or Orchestrator.
You have no scalar score per attempt. The whole training paradigm rests on an environment that returns a task-specific metric. Without it there is nothing to build preferences from.

Architecture¶

flowchart TB
  TASK["task instance"] --> ROOT["root state<br/>(task spec, constraints)"]
  ROOT --> ORCH
  subgraph DEC["Decision layer (persistent)"]
    ORCH["Orchestrator<br/>score-annotated history of<br/>(instruction, summary, score)<br/>NO raw code or logs"]
  end
  ORCH -->|"tool call:<br/>1. which attempt to refine<br/>2. instruction<br/>3. tagged references"| TOOL
  subgraph MED["Tool interface (the boundary)"]
    TOOL["build fresh Sub-Agent context<br/>launch + monitor lifecycle<br/>compact the trajectory back"]
  end
  TOOL -->|"fresh context"| SA1["Sub-Agent 1"]
  TOOL -->|"fresh context"| SA2["Sub-Agent 2 (parallel)"]
  subgraph EXE["Execution layer (disposable, isolated)"]
    SA1 --> LOOP1["write code -> execute -> debug<br/>until valid submission<br/>or budget exhausted"]
    SA2 --> LOOP2["write code -> execute -> debug"]
  end
  LOOP1 --> ENV["environment E<br/>validate + score"]
  LOOP2 --> ENV
  ENV -->|"score, or failure status<br/>+ structured summary"| TOOL
  TOOL -->|"compact response only"| ORCH

How to use it¶

Validated mechanism and result audit¶

import numpy as np

# ============ 1. context growth: slower is not bounded ========================
# Appendix F reports empirical linear slopes of 1,201.93 tokens/round for the
# Orchestrator and 1,870.29 for the monolithic Dojo Agent.
ORCH_SLOPE = 1_201.93
DOJO_SLOPE = 1_870.29
WINDOW = 256_000


def accumulated_context(turns, tokens_per_round):
    assert turns >= 0 and tokens_per_round > 0
    return turns * tokens_per_round


def first_round_over(window, tokens_per_round):
    return int(np.floor(window / tokens_per_round)) + 1


orch_limit = first_round_over(WINDOW, ORCH_SLOPE)
dojo_limit = first_round_over(WINDOW, DOJO_SLOPE)
assert (dojo_limit, orch_limit) == (137, 213)
assert accumulated_context(100, ORCH_SLOPE) < accumulated_context(100, DOJO_SLOPE)
assert accumulated_context(orch_limit, ORCH_SLOPE) > WINDOW
# The Orchestrator still grows with run length; only the per-round Sub-Agent
# context is reset and independently bounded.
def subagent_context(task_spec=2_000, prev_attempt=9_000, instruction=400, refs=6_000):
    return task_spec + prev_attempt + instruction + refs


subagent_sizes = [subagent_context() for _ in range(100)]
assert len(set(subagent_sizes)) == 1
assert subagent_sizes[0] < WINDOW


# ============ 2. branch-level returns vs immediate scores ====================
class Node:
    def __init__(self, score, children=None):
        self.score = score
        self.children = children or []


def downstream_return(node):
    """R(c) = max over the subtree rooted at c (the paper's labelling)."""
    return max([node.score] + [downstream_return(k) for k in node.children])


# A slow starter: branch B's first attempt scores worse than A's, but B's
# subtree finds the better solution. This is the exact case the paper says
# immediate-score labelling gets wrong.
A = Node(0.62, [Node(0.63), Node(0.61)])
B = Node(0.55, [Node(0.58), Node(0.81, [Node(0.84)])])
assert A.score > B.score                                   # immediate says A
assert downstream_return(B) > downstream_return(A)         # downstream says B
assert downstream_return(A) == 0.63 and downstream_return(B) == 0.84
immediate_pick = "A" if A.score > B.score else "B"
branch_pick = "A" if downstream_return(A) > downstream_return(B) else "B"
assert immediate_pick == "A" and branch_pick == "B"
assert immediate_pick != branch_pick                       # the labels disagree

# When there is no slow starter the two labellings agree, so branch-level
# labelling is a strict generalisation, not a different objective.
C = Node(0.70, [Node(0.72)])
D = Node(0.40, [Node(0.45)])
assert (C.score > D.score) == (downstream_return(C) > downstream_return(D))


# ============ 3. tree pairs, leaves, and executed attempts ====================
def internal_nodes(depth):
    """Binary expansion: one sibling preference pair per internal node."""
    return 2 ** depth - 1


def leaves(depth):
    return 2 ** depth


def executed_attempts(depth):
    """Every non-root child node is an independently executed Sub-Agent attempt."""
    return 2 ** (depth + 1) - 2


for d in (1, 2, 3, 4):
    assert internal_nodes(d) == 2 ** d - 1
    assert leaves(d) == 2 ** d
    assert executed_attempts(d) == 2 * internal_nodes(d)
# At depth 4: 30 executed attempts produce 16 terminal leaves and 15 pairs.
assert (executed_attempts(4), leaves(4), internal_nodes(4)) == (30, 16, 15)
assert internal_nodes(4) / executed_attempts(4) == 0.5


# ============ 4. the ranking NCE reduces to DPO at |C(u)| = 2 ================
def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))


def implicit_score(logp_theta, logp_ref, beta):
    """r_theta(s, tau) = beta * log(pi_theta / pi_ref), Eq. 3."""
    return beta * (logp_theta - logp_ref)


def ranking_nce(scores_pref_first):
    """-log softmax over candidates, with the preferred continuation at index 0."""
    s = np.asarray(scores_pref_first, dtype=float)
    return float(-(s[0] - np.log(np.exp(s).sum())))


BETA = 0.1
lp_t = np.array([-11.0, -13.5])     # updated policy logprobs (preferred, rejected)
lp_r = np.array([-12.0, -12.2])     # reference policy logprobs
r = implicit_score(lp_t, lp_r, BETA)
nce = ranking_nce(r)
dpo = float(-np.log(sigmoid(r[0] - r[1])))
assert abs(nce - dpo) < 1e-12, (nce, dpo)     # identical for two candidates

# With more than two candidates it is a genuine softmax ranking, and adding a
# strong distractor raises the loss (there is more to be discriminated against).
r3 = np.array([r[0], r[1], implicit_score(-10.5, -12.4, BETA)])
assert ranking_nce(r3) > ranking_nce(r)
# The loss falls when the preferred continuation pulls ahead of the reference.
better = implicit_score(np.array([-10.0, -13.5]), lp_r, BETA)
assert ranking_nce(better) < nce
# beta = 0 destroys the signal entirely: every score collapses to 0.
assert abs(ranking_nce(implicit_score(lp_t, lp_r, 0.0)) - np.log(2)) < 1e-12


# ============ 5. the published HumanRank table ===============================
HR = {
    "Dojo / Qwen3-4B":                       0.1878,
    "Dojo / o4-mini":                        0.4832,
    "Matryoshka / o4-mini + self":           0.5465,
    "Matryoshka / Qwen3-4B + o4-mini":       0.4613,
    "Matryoshka / Qwen3-4B-SFT + o4-mini":   0.5061,
    "Matryoshka / Qwen3-4B-RL + o4-mini":    0.5360,
}
sft_gain = HR["Matryoshka / Qwen3-4B-SFT + o4-mini"] - HR["Matryoshka / Qwen3-4B + o4-mini"]
rl_gain = HR["Matryoshka / Qwen3-4B-RL + o4-mini"] - HR["Matryoshka / Qwen3-4B-SFT + o4-mini"]
assert abs(sft_gain - 0.0448) < 1e-4
assert abs(rl_gain - 0.0299) < 1e-4
assert sft_gain > rl_gain
# Controlled architecture comparison: o4-mini is both the Dojo model and both
# Matryoshka roles. This is not an isolated Sub-Agent ablation.
architecture_gain_o4 = HR["Matryoshka / o4-mini + self"] - HR["Dojo / o4-mini"]
assert abs(architecture_gain_o4 - 0.0633) < 1e-4
# With o4-mini Sub-Agents, the trained 4B Orchestrator nearly matches but does
# not exceed the all-o4-mini Matryoshka configuration.
trained_gap = HR["Matryoshka / o4-mini + self"] - HR["Matryoshka / Qwen3-4B-RL + o4-mini"]
assert abs(trained_gap - 0.0105) < 1e-4
assert HR["Matryoshka / Qwen3-4B-RL + o4-mini"] < HR["Matryoshka / o4-mini + self"]

print("all Matryoshka Agent assertions passed")
print("  first round over 256k, Dojo / Orchestrator:", dojo_limit, "/", orch_limit)
print("  fixed per-round Sub-Agent context:", subagent_sizes[0], "tokens (illustrative)")
print("  depth-4 executed attempts / leaves / pairs:",
      executed_attempts(4), "/", leaves(4), "/", internal_nodes(4))
print("  ranking-NCE == DPO at |C|=2:", round(nce, 6), "==", round(dpo, 6))
print("  HumanRank gains  SFT / RL:", round(sft_gain, 4), "/", round(rl_gain, 4))
print("  o4 architecture gain / trained-4B gap:",
      round(architecture_gain_o4, 4), "/", round(trained_gap, 4))

Executed output:

all Matryoshka Agent assertions passed
  first round over 256k, Dojo / Orchestrator: 137 / 213
  fixed per-round Sub-Agent context: 17400 tokens (illustrative)
  depth-4 executed attempts / leaves / pairs: 30 / 16 / 15
  ranking-NCE == DPO at |C|=2: 0.584745 == 0.584745
  HumanRank gains  SFT / RL: 0.0448 / 0.0299
  o4 architecture gain / trained-4B gap: 0.0633 / 0.0105

The information boundary is the whole design¶

Block 1 uses the paper's Appendix F slopes: about 1,870.29 tokens per round for the monolithic Dojo Agent and 1,201.93 for the Orchestrator. Both are linear in run length. Under a 256K budget and those rounded slopes, the model first exceeds the window at round 137 for Dojo and 213 for the Orchestrator. Hierarchy extends the horizon; it does not make persistent context independent of the horizon.

The bounded term is each fresh Sub-Agent context. The block gives that term an illustrative fixed size and asserts it does not change across 100 rounds. Production systems still need Orchestrator compaction or a hard round limit before its slower linear growth reaches the model window.

Branch-level returns exist to protect slow starters¶

The refinement tree is labelled with downstream returns, R(c) = max over subtree(c) of r(v), not with the immediate score of each child. Block 2 constructs the case that motivates it: branch A opens at 0.62 and peaks at 0.63; branch B opens at 0.55 but its subtree reaches 0.84. Immediate-score labelling prefers A and teaches the Orchestrator to avoid the direction that actually wins. Downstream-return labelling prefers B. The block asserts the two labellings genuinely disagree here, and also that they agree when there is no slow starter, so this is a strict generalisation rather than a different objective.

The paper's own framing of the two comparisons this avoids is worth quoting directly: branch-level returns "do not penalize a refinement direction merely because its first attempt has a lower score but later improves substantially", and unlike global winner-take-all labelling the tree "provides multiple common-prefix preference pairs rather than treating all non-winning trajectories as equally negative."

Block 3 quantifies the second half without hiding execution cost. Binary expansion to depth 4 has 15 internal nodes, 16 terminal leaves, and 30 non-root nodes. Every non-root node is a Sub-Agent-executed attempt, because each sampled instruction produces a scored child attempt. Those 30 attempts yield 15 common-prefix preference pairs. The tree reuses prefixes to obtain structured comparisons, but it does not obtain 15 pairs from only 16 executions.

The objective is DPO generalised to a ranking¶

The Orchestrator loss parameterises the trajectory score as the implicit reward r_theta(s, tau) = beta * log(pi_theta / pi_ref) and plugs it into a softmax ranking over candidate continuations. Block 4 confirms that with two candidates this is exactly the DPO loss, to floating-point equality. That is a useful thing to know when implementing: you can start from a DPO trainer, and the only structural change is generalising the pairwise sigmoid to a softmax over |C(u)| candidates.

Two behaviours the block also pins down: adding a strong distractor raises the loss (more to discriminate against), and beta = 0 collapses every score to zero so the loss becomes log |C| and carries no signal at all. The reference policy is the previous online iteration's policy pi^(m), not a frozen SFT checkpoint, which is what makes this an online iterative scheme rather than one-shot preference tuning.

What the results actually show¶

Table 1 separates monolithic Dojo baselines from Matryoshka configurations:

Framework and model configuration	HumanRank (All)
Dojo Agent, Qwen3-4B	0.1878
Dojo Agent, o4-mini	0.4832
Matryoshka, Qwen3-4B orchestrator + o4-mini Sub-Agents	0.4613
Matryoshka, Qwen3-4B-SFT orchestrator + o4-mini Sub-Agents	0.5061
Matryoshka, Qwen3-4B-RL orchestrator + o4-mini Sub-Agents	0.5360
Matryoshka, o4-mini orchestrator + o4-mini Sub-Agents	0.5465

The controlled architecture comparison uses o4-mini throughout: Matryoshka o4-mini+self reaches 0.5465 versus monolithic Dojo o4-mini at 0.4832, a gain of 0.0633. With the Matryoshka framework and o4-mini Sub-Agents fixed, training the 4B Orchestrator adds 0.0448 through SFT and a further 0.0299 through RL. The final 0.5360 is 0.0105 below, not above, the all-o4-mini Matryoshka configuration.

The table does not isolate a numerical Sub-Agent effect for Qwen3-4B: 0.1878 is a monolithic Dojo result, and the paper omits Qwen3-4B+self from its main experiments because the small model performed poorly as an executor. That qualitative limitation matters, but subtracting 0.1878 from a Matryoshka row would confound framework, Orchestrator, and Sub-Agent changes.

How to develop with it¶

The online loop alternates two phases:

Collect. For each task instance, build a Solution Refinement Tree from the root state using the current Orchestrator policy. At each non-leaf node, sample two alternative refinement instructions from the same parent state and execute both through Sub-Agents. Binary expansion is chosen explicitly "for tractable data collection".
Optimise. Label each internal node's two branches by downstream return, extract the common-prefix preference pairs, and update the Orchestrator with the ranking objective. Successful execution trajectories separately feed Sub-Agent improvement.

Implementation notes that follow from the design:

Keep the Tool response schema fixed and small. It is the contract between the two layers. Every field you add to it erodes the boundary that makes the pattern work.
Return failures as structured status, not as scores. An invalid submission yields a status and diagnostic summary; do not coerce it into a low score, or downstream-return labelling will treat a crash and a genuinely bad solution as the same event.
Let the Orchestrator tag references explicitly. Reference selection is how information crosses the boundary deliberately rather than by accumulation.
Isolate sibling Sub-Agents completely. Parallel exploration only produces independent evidence if the branches cannot see each other.

How to maintain it¶

Version the Tool request and response schemas; a schema change alters both inference context and the training distribution.
Track Orchestrator tokens per round and enforce compaction or a round limit before its slower linear growth reaches the model window.
Retain every scored child attempt with its parent id so downstream returns and sibling pairs can be regenerated.
Re-evaluate model-role pairings after checkpoint changes; Orchestrator and Sub-Agent capability are not interchangeable.

How to run it in production¶

Budget per Sub-Agent, not per run. The debugging loop is bounded; that bound is your cost control and your latency bound at the same time.
The parallelism is real. Multiple tool calls launch multiple Sub-Agents concurrently on allocated compute. Size the pool for the branching factor you intend, or the tree collapses to sequential.
Meter the two layers separately. They have different models, different token profiles, and different failure signatures. An Orchestrator that is looping (repeatedly refining the same attempt with no score movement) looks nothing like a Sub-Agent that cannot produce a valid submission.
Log the tree, not just the winner. The tree is the training data. Discarding non-winning branches throws away the common-prefix pairs that make the next iteration cheaper.

Failure modes¶

Boundary leakage. Once raw logs start flowing back to the Orchestrator "just for debugging", its context grows with turns again and the pattern's main benefit is gone.
Immediate-score labelling. Trains the Orchestrator away from strategies that start slow and finish well; block 2 is the counterexample.
Winner-take-all labelling. Wastes most of the executed rollouts and treats every non-winner as equally bad.
Weak Sub-Agents. The paper's failure analysis says Qwen3-4B struggles as an executor, but Table 1 does not isolate a numerical Sub-Agent effect. Benchmark each role pairing directly.
beta too small. The implicit-reward scores collapse and the ranking loss becomes constant.
Stale reference policy. The objective's reference is the previous online iteration. Pinning it to a fixed checkpoint turns the scheme into ordinary offline preference tuning and loses the online correction.
Score-free environments. Without a scalar metric per attempt there are no downstream returns and no preferences, so the training paradigm does not apply even though the runtime pattern still would.
Over-applying the hierarchy. On short tasks the extra decision hop and the lossy boundary cost more than the context savings are worth.

References¶

Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineering: https://arxiv.org/abs/2607.25090
MLE-bench (the ML engineering task family): https://arxiv.org/abs/2410.07095
DPO (the pairwise special case of the ranking objective): https://arxiv.org/abs/2305.18290
InfoNCE / noise-contrastive ranking: https://arxiv.org/abs/1807.03748