Internalizing agent experience: context, skill libraries, or weights¶
Scope: the decision of what to do with interaction trajectories an agent has already paid to collect. Putting them in the context window works but is rented, not owned; fine-tuning on them directly barely works at all; distilling them into weights or abstracting them into a retrievable skill library are the two approaches that do. This page covers the mechanisms, the sample-efficiency argument, and the implementation detail that makes distillation affordable. Token-level guidance on a student's own rollouts is in on-policy distillation; the RL alternative is in agentic RL; persistent agent state is in agent context and memory.
Two primary sources. Gou et al., "Sample-Efficient Learning from Agent Experience" (arXiv 2607.21051, submitted 2026-07-23), which introduces Experience Distillation and reports on 749 curated software-engineering tasks and six TaleSuite text-adventure games. Xia et al., "SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning" (arXiv 2602.08234, submitted 2026-02-09), evaluated on ALFWorld, WebShop, and seven search-augmented tasks. Neither was reproduced here: no agent training run was performed. The Python block is this page's own model of the mechanisms and their cost structure, executed and asserted.
flowchart TB
EXP["Collected trajectories<br/>(expensive: real environment interaction)"]
EXP --> ICL["Keep in context<br/>full gain, paid on every request"]
EXP --> SFT["SFT on raw experience<br/>recovers ~4% of the ICL gain"]
EXP --> EPD["Experience Distillation<br/>into weights, no new interaction"]
EXP --> SKILL["Abstract into a skill library<br/>retrieved per task, co-evolves with policy"]
ICL -.->|"context removed"| GONE["Gain disappears"]What it is¶
An agent that has attempted a task, failed, and tried again has produced something valuable: a trajectory containing what it tried, what the environment said, and what eventually worked. In-context learning exploits this extremely well. Put the history back in the prompt and the agent does better. The catch is that the gain is entirely rented: remove the experience from the context and it vanishes completely.
Experience Distillation converts that rented gain into owned weights. The formulation is trajectory-level context distillation: the teacher is the base model with the experience in its context, the student is the same base model without it, and the objective is to match the teacher's policy at each recorded state. Estimating that objective naively would require new teacher rollouts in the environment, which would destroy the sample efficiency the method exists to preserve. A world model could substitute, but model errors compound over long rollouts.
The resolution is to branch from real data and simulate only a short continuation, then take the limiting case of a one-step branch, k = 1. At one step the branch contains only a single teacher decision at a recorded history, so no world model is needed at all and no new environment interaction occurs. The loss reduces to a teacher-sampled forward KL over recorded histories: sample the teacher's decision at each recorded branch point, and train the student to reproduce it.
Three implementation choices carry the method:
- Experience preprocessing. Raw histories are long and noisy, interleaving useful evidence with repeated exploration and verbose environment output, and may not fit the teacher's context window. A preprocessing function rewrites, abstracts, or summarizes before teacher generation.
- Enhanced teacher reasoning. A fixed prompt instructs the teacher to examine the preprocessed experience and reason more thoroughly before deciding. The authors report empirically that eliciting longer teacher reasoning improves post-distillation performance.
- Branch packing. Instead of one training example per branch point, all branch points of one trajectory are packed into a single sequence interleaving teacher decisions with recorded actions and observations. Only the teacher decisions carry loss.
Skill libraries take the other route: keep the experience external but stop storing it raw. SkillRL distills trajectories into a hierarchical library, SkillBank, holding general skills alongside per-category task-specific skills, retrieves from it adaptively, and lets the library co-evolve with the policy during RL rather than being frozen up front. The stated motivation is that raw trajectory memory is redundant and noisy, which prevents agents from extracting the reusable behavioural patterns that generalize.
Why use it¶
Because on the workloads this paper evaluates, plain supervised fine-tuning on the collected trajectories recovers almost none of the benefit. On the reported software-engineering suite, supervised fine-tuning on the collected experience recovers 3.8% of the in-context learning gain, while Experience Distillation recovers at least 64.8%, roughly a seventeenfold difference.
Bound that claim to what was measured. The evidence is 749 curated software-engineering tasks and six TaleSuite text-adventure games, with a specific trajectory-collection procedure and a specific base model. It is not a general result that SFT cannot learn from agent trajectories, and the mechanism below suggests exactly where it would not apply: the gap comes from trajectories that contain substantial failed exploration, so a corpus of clean, already-successful demonstrations does not have the same target mismatch. Treat "SFT recovers 3.8%" as a measurement on exploration-heavy self-collected experience, and re-measure before assuming it holds for your trajectory distribution.
The reason is a target mismatch. SFT trains the model to reproduce the actions that were actually recorded, including all the failed exploration that made the experience informative in the first place. Distillation instead trains it to reproduce what the model would now do having read that experience, which is the thing the in-context gain actually consists of. The recorded trajectory is the evidence; the teacher's improved decision is the lesson.
Against reinforcement learning, the claim is sample efficiency rather than final quality: in-context learning from trial-and-error experience followed by Experience Distillation matches classical RL baselines while using at least 9.6x fewer environment samples. That matters when a sample means running a test suite or waiting on a human.
When to use it (and when not)¶
Choose Experience Distillation when:
- Environment interaction is the expensive axis. Long-running tests, real infrastructure, human feedback. If rollouts were free you would run RL.
- You want the context window back. The distilled model needs no experience in its prompt, which frees the budget for the actual task.
- The experience is already collected. The method requires no further interaction beyond what you have.
Choose a skill library when:
- The knowledge should stay inspectable and editable. A named skill can be read, corrected, or deleted; a weight update cannot.
- You cannot retrain. Library approaches work with a frozen policy behind an API.
- The task distribution keeps shifting. A library co-evolving with the policy adapts without a new distillation run.
Stay with plain in-context learning when:
- The experience is small and the request volume is low. Then the recurring token cost never adds up to a training run.
- You need the full gain, not two thirds of it. Distillation retains at least 64.8%, not 100%.
Do not use SFT on raw trajectories for this purpose at all. It is the one option the evidence rules out.
Architecture¶
The one-step branch and the packed sequence¶
sequenceDiagram
participant R as "Recorded trajectory"
participant T as "Teacher (base model + experience in context)"
participant S as "Student (base model, no context)"
R->>T: "history h_t, preprocessed experience g(tau), reasoning prompt I"
T-->>R: "one teacher decision a'_t (branch, k=1)"
Note over R,T: "no environment step is taken: the branch is one decision deep"
R->>R: "append the RECORDED action a_t and its observation o_t+1"
R->>S: "packed sequence: o_0, a'_0, a_0, o_1, a'_1, a_1, ..."
Note over S: "loss applies only to a'_t, recorded pairs are context"The packed sequence is the practical core, and it is also an approximation the authors are explicit about: because the packed context contains earlier teacher decisions, it is not an exact implementation of the one-step objective, which conditions only on recorded history. The paper reports the approximation retains distillation performance while substantially reducing generation and training time.
How to use it¶
Size the run¶
Run: python3 expdist.py.
import numpy as np
# --- 1. Where the gain goes after experience leaves the prompt -------------
# ICL with experience is the 100% reference. It is not an after-removal method.
ICL_REFERENCE = 1.000
AFTER_REMOVAL = {
"plain model after removing the ICL context": 0.000,
"Experience Distillation (1-step branch rollout)": 0.648,
"SFT on the raw collected experience": 0.038,
}
print("[1] ICL gain while experience remains in the prompt: 100.0% (reference)")
print("[1] fraction of that gain retained after the context is removed")
for name, frac in AFTER_REMOVAL.items():
print(f"[1] {name:48s} {100 * frac:5.1f}%")
assert AFTER_REMOVAL["Experience Distillation (1-step branch rollout)"] > \
17 * AFTER_REMOVAL["SFT on the raw collected experience"]
assert AFTER_REMOVAL["plain model after removing the ICL context"] == 0.0
# --- 2. The inference-time cost ICL keeps paying -------------------------
# Distillation pays once at training time; ICL pays on every request forever.
print()
EXP_TOKENS = 24_000 # preprocessed experience carried in the prompt
REQ_PER_DAY = 50_000
COST_PER_MTOK = 3.00 # USD per million prompt tokens
def icl_daily_cost(exp_tokens: int, requests: int, per_mtok: float) -> float:
return exp_tokens * requests * per_mtok / 1e6
for days in (1, 30, 365):
c = icl_daily_cost(EXP_TOKENS, REQ_PER_DAY, COST_PER_MTOK) * days
print(f"[2] carrying {EXP_TOKENS:,} experience tokens for {days:>3} day(s) "
f"at {REQ_PER_DAY:,} req/day: ${c:12,.2f}")
assert icl_daily_cost(EXP_TOKENS, REQ_PER_DAY, COST_PER_MTOK) > 0
# Distillation moves this to a one-off training cost, and also frees the
# context window itself, which is the scarcer resource in long-horizon agents.
one_off = 4_000.0
break_even_days = one_off / icl_daily_cost(EXP_TOKENS, REQ_PER_DAY, COST_PER_MTOK)
print(f"[2] a ${one_off:,.0f} one-off distillation run breaks even in "
f"{break_even_days:.1f} days at this volume")
assert break_even_days < 30
# --- 3. Branch packing: why supervision density is the whole trick -------
# Naive: one training example per branch point, each re-processing the whole
# prefix. Packed: one sequence per trajectory, all branch points inside it.
print()
TURNS = 40 # interaction turns in one recorded trajectory
OBS_TOK = 350 # tokens per environment observation
ACT_TOK = 180 # tokens per model action (reasoning + action)
def naive_tokens(turns: int) -> tuple[int, int]:
"""(processed tokens, loss-bearing tokens) with one example per branch point."""
processed = loss = 0
for t in range(turns):
prefix = t * (OBS_TOK + ACT_TOK) + OBS_TOK # history up to o_t
processed += prefix + ACT_TOK # prefix plus the teacher decision
loss += ACT_TOK # only the teacher decision is supervised
return processed, loss
def packed_tokens(turns: int) -> tuple[int, int]:
"""(processed, loss-bearing) with all branches packed into one sequence."""
# o_0, a'_0, a_0, o_1, ... : each turn contributes one obs, one teacher
# decision, and one recorded action.
processed = turns * (OBS_TOK + 2 * ACT_TOK) + OBS_TOK
loss = turns * ACT_TOK
return processed, loss
np_, nl = naive_tokens(TURNS)
pp, pl = packed_tokens(TURNS)
print(f"[3] {TURNS}-turn trajectory")
print(f"[3] naive : {np_:>8,} tokens processed, {nl:>6,} supervised "
f"-> density {100 * nl / np_:5.2f}%")
print(f"[3] packed: {pp:>8,} tokens processed, {pl:>6,} supervised "
f"-> density {100 * pl / pp:5.2f}%")
print(f"[3] packing processes {np_ / pp:.1f}x fewer tokens for the same supervision")
assert nl == pl, "both forms supervise exactly the teacher decisions"
assert pp < np_, "packing must reduce processed tokens"
assert (pl / pp) > (nl / np_), "packing raises supervision density"
# The saving grows linearly with trajectory length, because the naive form is
# quadratic in turns while the packed form is linear.
ratios = [naive_tokens(t)[0] / packed_tokens(t)[0] for t in (10, 40, 160)]
print(f"[3] ratio at 10 / 40 / 160 turns: "
f"{ratios[0]:.1f}x / {ratios[1]:.1f}x / {ratios[2]:.1f}x (naive is quadratic in turns)")
assert ratios[2] > ratios[1] > ratios[0]
# Adversarial: packing is an approximation, not the exact objective. The
# packed context contains earlier TEACHER decisions a'_t, whereas the exact
# objective conditions only on the recorded history h_t. Count the drift.
print()
def contaminated_context_fraction(turns: int) -> float:
"""Teacher-decision share, including the packed sequence's initial observation."""
recorded = (turns + 1) * OBS_TOK + turns * ACT_TOK
teacher = turns * ACT_TOK
return teacher / (recorded + teacher)
fracs = [contaminated_context_fraction(t) for t in (5, 40, 160)]
asymptote = ACT_TOK / (OBS_TOK + 2 * ACT_TOK)
print("[3] teacher-generated share at 5 / 40 / 160 turns:",
" / ".join(f"{100 * f:.1f}%" for f in fracs))
print(f"[3] long-trajectory asymptote: {100 * asymptote:.1f}%")
assert fracs[0] < fracs[1] < fracs[2] < asymptote
assert asymptote - fracs[2] < 0.001
assert fracs[0] > 0.2, "the approximation is not negligible"
# The paper reports this approximation retains distillation performance while
# cutting generation and training time; it is a measured tradeoff, not exact.
# --- 4. Reported sample efficiency versus PPO on curated SWE -------------
print()
PPO_SAMPLES_PER_TASK = 504.9
SAMPLE_RATIO = 9.6
EPD_SAMPLES_PER_TASK = PPO_SAMPLES_PER_TASK / SAMPLE_RATIO
PPO_PASS1 = 17.7
EPD_PASS1 = 51.4
print(f"[4] PPO: {PPO_SAMPLES_PER_TASK:.1f} environment samples/task, "
f"{PPO_PASS1:.1f}% pass@1")
print(f"[4] ICL + EPD: {EPD_SAMPLES_PER_TASK:.1f} environment samples/task, "
f"{EPD_PASS1:.1f}% pass@1")
print(f"[4] sample ratio: {PPO_SAMPLES_PER_TASK / EPD_SAMPLES_PER_TASK:.1f}x fewer for ICL + EPD")
assert np.isclose(PPO_SAMPLES_PER_TASK / EPD_SAMPLES_PER_TASK, SAMPLE_RATIO)
assert EPD_PASS1 > PPO_PASS1
# Environment samples exclude teacher generation and student training compute.
print("[4] the reported saving is environment interaction only; teacher generation "
"and training still cost GPU time")
print("\nAll assertions passed.")
Executed output:
[1] ICL gain while experience remains in the prompt: 100.0% (reference)
[1] fraction of that gain retained after the context is removed
[1] plain model after removing the ICL context 0.0%
[1] Experience Distillation (1-step branch rollout) 64.8%
[1] SFT on the raw collected experience 3.8%
[2] carrying 24,000 experience tokens for 1 day(s) at 50,000 req/day: $ 3,600.00
[2] carrying 24,000 experience tokens for 30 day(s) at 50,000 req/day: $ 108,000.00
[2] carrying 24,000 experience tokens for 365 day(s) at 50,000 req/day: $1,314,000.00
[2] a $4,000 one-off distillation run breaks even in 1.1 days at this volume
[3] 40-turn trajectory
[3] naive : 434,600 tokens processed, 7,200 supervised -> density 1.66%
[3] packed: 28,750 tokens processed, 7,200 supervised -> density 25.04%
[3] packing processes 15.1x fewer tokens for the same supervision
[3] ratio at 10 / 40 / 160 turns: 3.9x / 15.1x / 59.9x (naive is quadratic in turns)
[3] teacher-generated share at 5 / 40 / 160 turns: 23.1% / 25.0% / 25.3%
[3] long-trajectory asymptote: 25.4%
[4] PPO: 504.9 environment samples/task, 17.7% pass@1
[4] ICL + EPD: 52.6 environment samples/task, 51.4% pass@1
[4] sample ratio: 9.6x fewer for ICL + EPD
[4] the reported saving is environment interaction only; teacher generation and training still cost GPU time
All assertions passed.
Three results are worth carrying into a design decision.
The recurring cost of in-context learning is what justifies distilling. At the modelled volume, carrying 24,000 experience tokens on every request costs about $3,600 a day, so a one-off distillation run priced at $4,000 breaks even in roughly a day. The numbers are illustrative, but the shape is not: in-context learning is a per-request tax and distillation is a one-off capital cost, so the decision is set entirely by request volume.
Branch packing is not a minor optimization; it is what makes the method affordable. The naive form processes tokens quadratically in trajectory length because every branch point re-reads the whole prefix. On a 40-turn trajectory it processes 434,600 tokens to supervise 7,200, a supervision density of 1.66%. Packing supervises exactly the same 7,200 tokens while processing 28,750, a density of 25.04% and a 15.1x reduction. The advantage grows with length: 3.9x at 10 turns, 59.9x at 160.
The approximation approaches a fixed asymptote. Including the initial observation, teacher-generated decisions occupy 23.1%, 25.0%, and 25.3% of the modeled packed context at 5, 40, and 160 turns. The share approaches 25.4% rather than being exactly invariant. Long trajectories therefore stop changing the proportion materially, but they do not dilute the approximation. Validate against the unpacked objective on a small sample before scaling a run.
How to develop with it¶
- Preprocess before generating, not after. The teacher's context window bounds how much experience can inform each decision, so summarization is a capacity decision, not a cleanup step.
- Give the teacher room to think. The reported finding is that longer teacher reasoning improves post-distillation performance. This is teacher-side inference cost paid once, and it is the cheapest quality lever available.
- Supervise only the teacher decisions. Recorded actions and observations are context. Applying loss to recorded actions turns the method back into SFT on raw experience, which is the 3.8% baseline.
- Keep the environment budget as the headline metric. The reported win is environment samples, not GPU hours. Teacher generation and student training still cost real compute, and the source does not claim otherwise. Reporting a total-cost win would overstate the result.
- Validate on held-out tasks, not on the tasks the experience came from. Reproducing decisions on seen tasks is close to trivial; the question is whether the behaviour transfers.
How to maintain it¶
- Re-distill when the base model changes. The teacher is the base model with context, so a new base checkpoint invalidates the distilled deltas.
- Track how much of the ICL gain you actually retain. Measure it directly by evaluating with and without experience in context; the 64.8% figure is from one paper's tasks, not a constant.
- For skill libraries, watch library size as a health signal. SkillRL evolves the library during training, which means it can grow without bound if abstraction is weak. The stated goal is reducing token footprint, so a library that grows faster than performance is failing at its own objective.
- Re-check whether experience is still worth keeping. Old trajectories describe an environment that may have changed.
How to run it in production¶
- Keep collection, teacher generation, and student training as separately versioned jobs so a failed distillation run cannot corrupt the serving checkpoint.
- Pin the base model, teacher prompt, environment revision, trajectory corpus, and packing implementation for every run.
- Gate promotion on held-out task success, retention of the measured in-context-learning gain, and regression tests for tasks unrelated to the experience corpus.
- Serve the distilled model without the original experience context; otherwise latency and token cost still include the recurring tax that distillation is meant to remove.
Failure modes¶
- Generalising the 3.8% SFT figure. It is measured on exploration-heavy self-collected trajectories across two task families. On a corpus of clean successful demonstrations the target mismatch that produces it is much weaker, so re-measure rather than assuming SFT is hopeless.
- Distilling the exploration instead of the lesson. The signature is a distilled model that reproduces the failed attempts. Caused by supervising recorded actions rather than teacher decisions.
- Teacher context overflow. Long raw histories silently truncate, so the teacher decides without the evidence that made the experience valuable, and the distilled behaviour regresses toward the base model.
- Quadratic training cost from unpacked branches. Works correctly and costs 15x to 60x more than it should.
- Reading branch packing as exact. It is an approximation whose teacher-generated share converges toward a measurable asymptote. Fine to use, not fine to assume away.
- Claiming a total-compute win. The evidence supports an environment-sample win.
- Skill library drift. A library that co-evolves with a policy can accumulate skills that describe the policy's current quirks rather than the task, which shows up as good in-distribution scores and poor transfer.
- Assuming the ICL ceiling is reachable. At least 64.8% retention is a strong result and is also not 100%. Budget for the residual.
Open questions and validation¶
- Nothing here was reproduced. Both papers' results are as reported by their authors on their own task suites.
- The 64.8% and 9.6x figures come from 749 curated software-engineering tasks and six text-adventure games. Whether they hold on other environments is untested.
- The cost figures in the executed model are this page's illustrative parameters, not measurements from either paper.
- Whether distillation and skill libraries compose, so that a distilled model also carries a library, is not addressed by either source.
- SkillRL's reported 15.3% improvement is against its own baselines on its own benchmarks and is not directly comparable to the Experience Distillation numbers, which use a different metric entirely (fraction of ICL gain retained).
References¶
- Gou et al., "Sample-Efficient Learning from Agent Experience", arXiv 2607.21051: https://arxiv.org/abs/2607.21051
- Xia et al., "SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning", arXiv 2602.08234: https://arxiv.org/abs/2602.08234
- Janner et al., "When to Trust Your Model: Model-Based Policy Optimization" (the branched-rollout idea Experience Distillation builds on), arXiv 1906.08253: https://arxiv.org/abs/1906.08253
Related: On-policy distillation · Agentic RL · Agent context and memory · Knowledge distillation methods · Reasoning distillation and SFT · Synthetic data generation