Internalizing agent experience: context, skill libraries, or weights¶
Scope: the decision of what to do with interaction trajectories an agent has already paid to collect. Putting them in the context window works but is rented, not owned; fine-tuning on them directly barely works at all; distilling them into weights or abstracting them into a retrievable skill library are the two approaches that do. This page covers the mechanisms, the sample-efficiency argument, and the implementation detail that makes distillation affordable. Token-level guidance on a student's own rollouts is in on-policy distillation; the RL alternative is in agentic RL; persistent agent state is in agent context and memory.
Two primary sources. Gou et al., "Sample-Efficient Learning from Agent Experience" (arXiv 2607.21051, submitted 2026-07-23), which introduces Experience Distillation and reports on 749 curated software-engineering tasks and six TaleSuite text-adventure games. Xia et al., "SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning" (arXiv 2602.08234, submitted 2026-02-09), evaluated on ALFWorld, WebShop, and seven search-augmented tasks. Neither was reproduced here: no agent training run was performed. The Python block is this page's own model of the mechanisms and their cost structure, executed and asserted.
flowchart TB
EXP["Collected trajectories<br/>(expensive: real environment interaction)"]
EXP --> ICL["Keep in context<br/>full gain, paid on every request"]
EXP --> SFT["SFT on raw experience<br/>recovers ~4% of the ICL gain"]
EXP --> EPD["Experience Distillation<br/>into weights, no new interaction"]
EXP --> SKILL["Abstract into a skill library<br/>retrieved per task, co-evolves with policy"]
ICL -.->|"context removed"| GONE["Gain disappears"]What it is¶
An agent that has attempted a task, failed, and tried again has produced something valuable: a trajectory containing what it tried, what the environment said, and what eventually worked. In-context learning exploits this extremely well. Put the history back in the prompt and the agent does better. The catch is that the gain is entirely rented: remove the experience from the context and it vanishes completely.
Experience Distillation converts that rented gain into owned weights. The formulation is trajectory-level context distillation: the teacher is the base model with the experience in its context, the student is the same base model without it, and the objective is to match the teacher's policy at each recorded state. Estimating that objective naively would require new teacher rollouts in the environment, which would destroy the sample efficiency the method exists to preserve. A world model could substitute, but model errors compound over long rollouts.
The resolution is to branch from real data and simulate only a short continuation, then take the limiting case of a one-step branch, k = 1. At one step the branch contains only a single teacher decision at a recorded history, so no world model is needed at all and no new environment interaction occurs. The loss reduces to a teacher-sampled forward KL over recorded histories: sample the teacher's decision at each recorded branch point, and train the student to reproduce it.
Three implementation choices carry the method:
- Experience preprocessing. Raw histories are long and noisy, interleaving useful evidence with repeated exploration and verbose environment output, and may not fit the teacher's context window. A preprocessing function rewrites, abstracts, or summarizes before teacher generation.
- Enhanced teacher reasoning. A fixed prompt instructs the teacher to examine the preprocessed experience and reason more thoroughly before deciding. The authors report empirically that eliciting longer teacher reasoning improves post-distillation performance.
- Branch packing. Instead of one training example per branch point, all branch points of one trajectory are packed into a single sequence interleaving teacher decisions with recorded actions and observations. Only the teacher decisions carry loss.
Skill libraries take the other route: keep the experience external but stop storing it raw. SkillRL distills trajectories into a hierarchical library, SkillBank, holding general skills alongside per-category task-specific skills, retrieves from it adaptively, and lets the library co-evolve with the policy during RL rather than being frozen up front. The stated motivation is that raw trajectory memory is redundant and noisy, which prevents agents from extracting the reusable behavioural patterns that generalize.
Why not just fine-tune on the experience¶
Because it does not work. On the reported software-engineering suite, supervised fine-tuning on the collected experience recovers 3.8% of the in-context learning gain, while Experience Distillation recovers at least 64.8%, roughly a seventeenfold difference.
The reason is a target mismatch. SFT trains the model to reproduce the actions that were actually recorded, including all the failed exploration that made the experience informative in the first place. Distillation instead trains it to reproduce what the model would now do having read that experience, which is the thing the in-context gain actually consists of. The recorded trajectory is the evidence; the teacher's improved decision is the lesson.
Against reinforcement learning, the claim is sample efficiency rather than final quality: in-context learning from trial-and-error experience followed by Experience Distillation matches classical RL baselines while using at least 9.6x fewer environment samples. That matters when a sample means running a test suite or waiting on a human.
When to use which¶
Choose Experience Distillation when:
- Environment interaction is the expensive axis. Long-running tests, real infrastructure, human feedback. If rollouts were free you would run RL.
- You want the context window back. The distilled model needs no experience in its prompt, which frees the budget for the actual task.
- The experience is already collected. The method requires no further interaction beyond what you have.
Choose a skill library when:
- The knowledge should stay inspectable and editable. A named skill can be read, corrected, or deleted; a weight update cannot.
- You cannot retrain. Library approaches work with a frozen policy behind an API.
- The task distribution keeps shifting. A library co-evolving with the policy adapts without a new distillation run.
Stay with plain in-context learning when:
- The experience is small and the request volume is low. Then the recurring token cost never adds up to a training run.
- You need the full gain, not two thirds of it. Distillation retains at least 64.8%, not 100%.
Do not use SFT on raw trajectories for this purpose at all. It is the one option the evidence rules out.
Architecture: the one-step branch and the packed sequence¶
sequenceDiagram
participant R as "Recorded trajectory"
participant T as "Teacher (base model + experience in context)"
participant S as "Student (base model, no context)"
R->>T: "history h_t, preprocessed experience g(tau), reasoning prompt I"
T-->>R: "one teacher decision a'_t (branch, k=1)"
Note over R,T: "no environment step is taken: the branch is one decision deep"
R->>R: "append the RECORDED action a_t and its observation o_t+1"
R->>S: "packed sequence: o_0, a'_0, a_0, o_1, a'_1, a_1, ..."
Note over S: "loss applies only to a'_t; recorded pairs are context"The packed sequence is the practical core, and it is also an approximation the authors are explicit about: because the packed context contains earlier teacher decisions, it is not an exact implementation of the one-step objective, which conditions only on recorded history. The paper reports the approximation retains distillation performance while substantially reducing generation and training time.
How to size it¶
Run: python3 expdist.py.
import numpy as np
# --- 1. Where the gain goes: four ways to use collected experience --------
# Reported on 749 curated SWE tasks (Gou et al., arXiv 2607.21051): the ICL
# gain is the reference (100%); each method recovers some fraction of it.
RECOVERY = {
"in-context (ICL), experience kept in the prompt": 1.000,
"Experience Distillation (1-step branch rollout)": 0.648,
"SFT on the raw collected experience": 0.038,
}
print("[1] fraction of the in-context learning gain retained after the context is removed")
for name, frac in RECOVERY.items():
print(f"[1] {name:48s} {100 * frac:5.1f}%")
assert RECOVERY["Experience Distillation (1-step branch rollout)"] > \
17 * RECOVERY["SFT on the raw collected experience"], "reported gap is roughly 17x"
# ICL only "retains" the gain while the experience stays in the prompt. Remove
# it and ICL retains nothing, which is the entire motivation for distilling.
print("[1] note: the ICL row holds only while the experience remains in context")
# --- 2. The inference-time cost ICL keeps paying -------------------------
# Distillation pays once at training time; ICL pays on every request forever.
print()
EXP_TOKENS = 24_000 # preprocessed experience carried in the prompt
REQ_PER_DAY = 50_000
COST_PER_MTOK = 3.00 # USD per million prompt tokens
def icl_daily_cost(exp_tokens: int, requests: int, per_mtok: float) -> float:
return exp_tokens * requests * per_mtok / 1e6
for days in (1, 30, 365):
c = icl_daily_cost(EXP_TOKENS, REQ_PER_DAY, COST_PER_MTOK) * days
print(f"[2] carrying {EXP_TOKENS:,} experience tokens for {days:>3} day(s) "
f"at {REQ_PER_DAY:,} req/day: ${c:12,.2f}")
assert icl_daily_cost(EXP_TOKENS, REQ_PER_DAY, COST_PER_MTOK) > 0
# Distillation moves this to a one-off training cost, and also frees the
# context window itself, which is the scarcer resource in long-horizon agents.
one_off = 4_000.0
break_even_days = one_off / icl_daily_cost(EXP_TOKENS, REQ_PER_DAY, COST_PER_MTOK)
print(f"[2] a ${one_off:,.0f} one-off distillation run breaks even in "
f"{break_even_days:.1f} days at this volume")
assert break_even_days < 30
# --- 3. Branch packing: why supervision density is the whole trick -------
# Naive: one training example per branch point, each re-processing the whole
# prefix. Packed: one sequence per trajectory, all branch points inside it.
print()
TURNS = 40 # interaction turns in one recorded trajectory
OBS_TOK = 350 # tokens per environment observation
ACT_TOK = 180 # tokens per model action (reasoning + action)
def naive_tokens(turns: int) -> tuple[int, int]:
"""(processed tokens, loss-bearing tokens) with one example per branch point."""
processed = loss = 0
for t in range(turns):
prefix = t * (OBS_TOK + ACT_TOK) + OBS_TOK # history up to o_t
processed += prefix + ACT_TOK # prefix plus the teacher decision
loss += ACT_TOK # only the teacher decision is supervised
return processed, loss
def packed_tokens(turns: int) -> tuple[int, int]:
"""(processed, loss-bearing) with all branches packed into one sequence."""
# o_0, a'_0, a_0, o_1, ... : each turn contributes one obs, one teacher
# decision, and one recorded action.
processed = turns * (OBS_TOK + 2 * ACT_TOK) + OBS_TOK
loss = turns * ACT_TOK
return processed, loss
np_, nl = naive_tokens(TURNS)
pp, pl = packed_tokens(TURNS)
print(f"[3] {TURNS}-turn trajectory")
print(f"[3] naive : {np_:>8,} tokens processed, {nl:>6,} supervised "
f"-> density {100 * nl / np_:5.2f}%")
print(f"[3] packed: {pp:>8,} tokens processed, {pl:>6,} supervised "
f"-> density {100 * pl / pp:5.2f}%")
print(f"[3] packing processes {np_ / pp:.1f}x fewer tokens for the same supervision")
assert nl == pl, "both forms supervise exactly the teacher decisions"
assert pp < np_, "packing must reduce processed tokens"
assert (pl / pp) > (nl / np_), "packing raises supervision density"
# The saving grows linearly with trajectory length, because the naive form is
# quadratic in turns while the packed form is linear.
ratios = [naive_tokens(t)[0] / packed_tokens(t)[0] for t in (10, 40, 160)]
print(f"[3] ratio at 10 / 40 / 160 turns: "
f"{ratios[0]:.1f}x / {ratios[1]:.1f}x / {ratios[2]:.1f}x (naive is quadratic in turns)")
assert ratios[2] > ratios[1] > ratios[0]
# Adversarial: packing is an approximation, not the exact objective. The
# packed context contains earlier TEACHER decisions a'_t, whereas the exact
# objective conditions only on the recorded history h_t. Count the drift.
print()
def contaminated_context_fraction(turns: int) -> float:
"""Share of context tokens at the final branch point that are teacher-generated."""
recorded = turns * (OBS_TOK + ACT_TOK)
teacher = turns * ACT_TOK
return teacher / (recorded + teacher)
fracs = [contaminated_context_fraction(t) for t in (5, 40, 160)]
print(f"[3] {100 * fracs[0]:.1f}% of the packed context is teacher-generated rather than "
f"recorded history,")
print(f"[3] and that share is INVARIANT to trajectory length "
f"({', '.join(f'{100 * f:.1f}%' for f in fracs)} at 5/40/160 turns):")
print(f"[3] packing does not amplify the approximation with length, but never dilutes it either")
assert np.allclose(fracs, fracs[0]), "contamination share is scale-invariant"
assert fracs[0] > 0.2, "the approximation is not negligible"
# The paper reports this approximation retains distillation performance while
# cutting generation and training time; it is a measured tradeoff, not exact.
# --- 4. Sample efficiency versus RL --------------------------------------
# Matching PPO's performance with 9.6x fewer environment samples.
print()
PPO_SAMPLES = 96_000
SPEEDUP = 9.6
epd_samples = PPO_SAMPLES / SPEEDUP
print(f"[4] PPO environment samples: {PPO_SAMPLES:,}")
print(f"[4] ICL + Experience Distillation, same performance: {epd_samples:,.0f} "
f"({SPEEDUP}x fewer)")
SEC_PER_SAMPLE = 45 # a slow SWE task rollout
for label, n in (("PPO", PPO_SAMPLES), ("ICL + EPD", epd_samples)):
hours = n * SEC_PER_SAMPLE / 3600
print(f"[4] {label:10s} -> {hours:8,.0f} environment-hours at {SEC_PER_SAMPLE}s/sample")
assert np.isclose(PPO_SAMPLES / epd_samples, SPEEDUP)
# The saving is in ENVIRONMENT interaction, which is the expensive axis when
# each sample runs a test suite or needs a human. Teacher generation and
# training still cost GPU time, and the paper does not claim otherwise.
print("[4] the saving is environment interaction only; teacher generation and "
"training still cost GPU time")
print("\nAll assertions passed.")
Executed output:
[1] fraction of the in-context learning gain retained after the context is removed
[1] in-context (ICL), experience kept in the prompt 100.0%
[1] Experience Distillation (1-step branch rollout) 64.8%
[1] SFT on the raw collected experience 3.8%
[1] note: the ICL row holds only while the experience remains in context
[2] carrying 24,000 experience tokens for 1 day(s) at 50,000 req/day: $ 3,600.00
[2] carrying 24,000 experience tokens for 30 day(s) at 50,000 req/day: $ 108,000.00
[2] carrying 24,000 experience tokens for 365 day(s) at 50,000 req/day: $1,314,000.00
[2] a $4,000 one-off distillation run breaks even in 1.1 days at this volume
[3] 40-turn trajectory
[3] naive : 434,600 tokens processed, 7,200 supervised -> density 1.66%
[3] packed: 28,750 tokens processed, 7,200 supervised -> density 25.04%
[3] packing processes 15.1x fewer tokens for the same supervision
[3] ratio at 10 / 40 / 160 turns: 3.9x / 15.1x / 59.9x (naive is quadratic in turns)
[3] 25.4% of the packed context is teacher-generated rather than recorded history,
[3] and that share is INVARIANT to trajectory length (25.4%, 25.4%, 25.4% at 5/40/160 turns):
[3] packing does not amplify the approximation with length, but never dilutes it either
[4] PPO environment samples: 96,000
[4] ICL + Experience Distillation, same performance: 10,000 (9.6x fewer)
[4] PPO -> 1,200 environment-hours at 45s/sample
[4] ICL + EPD -> 125 environment-hours at 45s/sample
[4] the saving is environment interaction only; teacher generation and training still cost GPU time
All assertions passed.
Three results are worth carrying into a design decision.
The recurring cost of in-context learning is what justifies distilling. At the modelled volume, carrying 24,000 experience tokens on every request costs about $3,600 a day, so a one-off distillation run priced at $4,000 breaks even in roughly a day. The numbers are illustrative, but the shape is not: in-context learning is a per-request tax and distillation is a one-off capital cost, so the decision is set entirely by request volume.
Branch packing is not a minor optimization; it is what makes the method affordable. The naive form processes tokens quadratically in trajectory length because every branch point re-reads the whole prefix. On a 40-turn trajectory it processes 434,600 tokens to supervise 7,200, a supervision density of 1.66%. Packing supervises exactly the same 7,200 tokens while processing 28,750, a density of 25.04% and a 15.1x reduction. The advantage grows with length: 3.9x at 10 turns, 59.9x at 160.
The approximation packing introduces is scale-invariant. About a quarter of the packed context is teacher-generated rather than recorded history, and that share does not change with trajectory length. That is reassuring in one direction, since long trajectories do not compound the deviation, and unhelpful in the other, since it never dilutes either. Treat it as a fixed known bias, and validate against the unpacked objective on a small sample before scaling a run.
How to run it¶
- Preprocess before generating, not after. The teacher's context window bounds how much experience can inform each decision, so summarization is a capacity decision, not a cleanup step.
- Give the teacher room to think. The reported finding is that longer teacher reasoning improves post-distillation performance. This is teacher-side inference cost paid once, and it is the cheapest quality lever available.
- Supervise only the teacher decisions. Recorded actions and observations are context. Applying loss to recorded actions turns the method back into SFT on raw experience, which is the 3.8% baseline.
- Keep the environment budget as the headline metric. The reported win is environment samples, not GPU hours. Teacher generation and student training still cost real compute, and the source does not claim otherwise. Reporting a total-cost win would overstate the result.
- Validate on held-out tasks, not on the tasks the experience came from. Reproducing decisions on seen tasks is close to trivial; the question is whether the behaviour transfers.
How to maintain it¶
- Re-distill when the base model changes. The teacher is the base model with context, so a new base checkpoint invalidates the distilled deltas.
- Track how much of the ICL gain you actually retain. Measure it directly by evaluating with and without experience in context; the 64.8% figure is from one paper's tasks, not a constant.
- For skill libraries, watch library size as a health signal. SkillRL evolves the library during training, which means it can grow without bound if abstraction is weak. The stated goal is reducing token footprint, so a library that grows faster than performance is failing at its own objective.
- Re-check whether experience is still worth keeping. Old trajectories describe an environment that may have changed.
Failure modes¶
- Distilling the exploration instead of the lesson. The signature is a distilled model that reproduces the failed attempts. Caused by supervising recorded actions rather than teacher decisions.
- Teacher context overflow. Long raw histories silently truncate, so the teacher decides without the evidence that made the experience valuable, and the distilled behaviour regresses toward the base model.
- Quadratic training cost from unpacked branches. Works correctly and costs 15x to 60x more than it should.
- Reading branch packing as exact. It is an approximation with a measurable, constant contamination share. Fine to use, not fine to assume away.
- Claiming a total-compute win. The evidence supports an environment-sample win.
- Skill library drift. A library that co-evolves with a policy can accumulate skills that describe the policy's current quirks rather than the task, which shows up as good in-distribution scores and poor transfer.
- Assuming the ICL ceiling is reachable. At least 64.8% retention is a strong result and is also not 100%. Budget for the residual.
Open questions and validation¶
- Nothing here was reproduced. Both papers' results are as reported by their authors on their own task suites.
- The 64.8% and 9.6x figures come from 749 curated software-engineering tasks and six text-adventure games. Whether they hold on other environments is untested.
- The cost figures in the executed model are this page's illustrative parameters, not measurements from either paper.
- Whether distillation and skill libraries compose, so that a distilled model also carries a library, is not addressed by either source.
- SkillRL's reported 15.3% improvement is against its own baselines on its own benchmarks and is not directly comparable to the Experience Distillation numbers, which use a different metric entirely (fraction of ICL gain retained).
References¶
- Gou et al., "Sample-Efficient Learning from Agent Experience", arXiv 2607.21051: https://arxiv.org/abs/2607.21051
- Xia et al., "SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning", arXiv 2602.08234: https://arxiv.org/abs/2602.08234
- Janner et al., "When to Trust Your Model: Model-Based Policy Optimization" (the branched-rollout idea Experience Distillation builds on), arXiv 1906.08253: https://arxiv.org/abs/1906.08253
Related: On-policy distillation · Agentic RL · Agent context and memory · Knowledge distillation methods · Reasoning distillation and SFT · Synthetic data generation