Skip to content
Markdown

Reconstructing agent memory instead of replaying it

Scope: what happens between retrieving a past experience and acting on it, and why that gap is where memory-augmented agents fail. This page covers the applicability gap that verbatim replay ignores, the retrieve-critique-reconstruct decomposition, how a single policy learns to reject its own memories from task reward alone, and an ablation-level read of where the measured gains actually come from. The storage-side treatment is the filesystem as agent memory; the concept-level view is agent context and memory; the training machinery is GRPO.

Primary source: Wu, Fu, Wen, Yang, Zou, Mei, Wang, Zhang, Yang, Hu, Zhang, Shi, Cai, "MemHarness: Memory Is Reconstructed, Not Replayed", arXiv:2607.28272v1 [cs.AI], 30 July 2026 (Zhejiang University, Shanghai AI Laboratory, and others). Code: KnowledgeXLab/MemHarness, Apache-2.0, last pushed 2026-07-31, 7 stars as of 2026-08-02.

What this page adds. The Python block is executed and asserted (Python 3.11, numpy 2.x). It reproduces every published macro-average from the per-category cells, decomposes the headline gain across the ablation rows, and records one table-level inconsistency the paper does not address. No agent was trained or run; every number is the paper's.

Scale caveat. One backbone (Qwen2.5-7B-Instruct), two benchmarks (ALFWorld and WebShop), one run per configuration with no seed variance reported. Read the mechanism, not the leaderboard.

What it is

Most memory-augmented agents follow a retrieve-and-replay pattern: a retriever returns a stored trajectory or distilled principle, and it is inserted into the model context verbatim. That design conflates two different things, retrieval relevance and action-level applicability. A memory can be about exactly the right task and still be wrong for the current step, because it was formed under different environmental conditions.

MemHarness inserts an explicit step in between. At each decision step the same policy:

  1. Retrieves, optionally. The policy decides from its history window whether to query the memory bank at all, then emits a query and receives the top-k entries.
  2. Critiques and reconstructs. Each retrieved entry is a pair (e_i, o_src_i): an abstract strategy and the source observation it was formed under. The policy compares each source observation against the current history, then retains, revises, or rejects, emitting state-specific guidance g_t. If nothing applies it emits <EMPTY> and falls back to a self-reasoning prompt.
  3. Acts, conditioned on the reconstructed guidance rather than on the raw record.

The reconstruction has no ground-truth supervision, so it is treated as a latent variable and trained end-to-end with GRPO on task reward: 10 for success, 0 for failure, plus 0.1 x R_format where the format score equally weights one valid <think> and <action> block per step, between 1 and 5 memory retrievals per episode, and English-only output. The trajectory-level advantage is assigned to every token, so the same signal credits the thinking, the reconstruction, and the action.

Keeping o_src beside the strategy is the design decision that makes any of this possible. Without a record of the state a memory came from, "does this still apply" is not answerable.

Why use it

Because injecting retrieved memory into a trained RL agent can make it worse, and the paper measures that directly. On ALFWorld, adding a memory bank to an RL-trained agent takes it from 76.4 to 70.1. The reported headline is that MemHarness reaches 85.2 on ALFWorld and 75.6 on WebShop from a 7B backbone, against 76.4 and 66.1 for plain GRPO.

The middle ground it occupies is real and worth naming. Parametric approaches internalise experience into weights, which gives state-conditioned behaviour but makes the influence of any particular experience impossible to inspect or revise. Explicit memory banks are traceable but rigid. MemHarness keeps the bank inspectable and parameterises only the use of it, so you can still read what was stored and now also read what the policy decided to do with it.

When to use it (and when not)

Use it when your stored experiences are abstract and your states vary. The applicability gap is largest when a memory is a distilled principle rather than a concrete replayable action sequence, and when the environment presents high state variance. ALFWorld's room layouts and WebShop's page content are both in that regime.

Do not adopt it as a serving-side memory system. The ablation that matters is - w/o memory: the same trained policy with retrieval disabled at inference scores 83.0 on ALFWorld against the full system's 85.2, and 73.6 against 75.6 on WebShop. Three quarters of the improvement over plain GRPO is already in the weights. If your constraint is inference cost or retrieval-infrastructure complexity, you can drop the memory bank entirely at test time and keep most of the benefit.

Do not expect it to transfer without the RL stage. Replacing the policy's internal reconstruction with a generic instruction-tuned model of the same family, keeping the actor unchanged, drops ALFWorld from 85.2 to 77.7. Zero-shot text rewriting is not the mechanism; the trained one is.

Do not generalise "replay causes negative transfer" from this paper. The sign flips between its own two environments: raw memory costs 6.3 points on ALFWorld and gains 6.5 on WebShop against the same GRPO baseline.

Architecture

flowchart LR
  ENV["Environment observation<br/>history window w = 3"] --> DEC{"retrieve?"}
  DEC -->|"no"| ACT
  DEC -->|"yes"| RET["Retriever, BGE-M3<br/>top-3 principles<br/>each with its source observation"]
  RET --> REC["Critique and reconstruct<br/>policy compares o_src against h_t<br/>retain / revise / reject"]
  REC -->|"applicable"| G["state-specific guidance g_t"]
  REC -->|"EMPTY"| SELF["self-reasoning prompt"]
  G --> ACT["Action generation<br/>same policy, same weights"]
  SELF --> ACT
  ACT --> ENV
  ACT -.->|"episode reward 10 / 0<br/>+ 0.1 x format"| GRPO["GRPO, trajectory advantage<br/>applied to every token"]
  GRPO -.->|"one policy"| REC
  ACT -.->|"distil trajectories"| BANK["Memory bank<br/>semantic dedup, utility tracking,<br/>low-utility pruning"]
  BANK --> RET

The whole framework is one set of weights doing three jobs. That is what makes the training-time effect on the base policy unsurprising in hindsight: optimising the reconstruction head is optimising the actor.

How to use it

Executed audit of the published tables

import numpy as np

CATS = ["Pick", "Look", "Clean", "Heat", "Cool", "Pick2"]
# (ALFWorld per-category success rate, reported ALFWorld average, reported WebShop success rate)
T = {
 "GRPO":              ([90.8, 66.1, 89.3, 74.7, 72.5, 64.7], 76.4, 66.1),
 "RL + Raw Memory":   ([84.8, 61.5, 95.7, 62.5, 61.9, 54.2], 70.1, 72.6),
 "MemHarness":        ([87.0, 78.6, 97.0, 87.5, 71.4, 90.0], 85.2, 75.6),
 "- generic LLM rec": ([97.1, 52.6, 79.2, 78.6, 80.0, 78.9], 77.7, 71.8),
 "- w/o reconstr.":   ([94.1, 52.6, 79.2, 85.7, 76.7, 89.5], 79.6, 74.6),
 "- w/o memory":      ([100.0, 68.4, 91.7, 78.6, 70.0, 89.5], 83.0, 73.6),
}

# 1. Every published ALFWorld average is the unweighted mean of six categories.
for k, (row, avg, _) in T.items():
    assert abs(np.mean(row) - avg) < 0.06, k

# 2. Decompose the gain over plain GRPO into training effect and inference-time retrieval.
for i in (1, 2):
    total = T["MemHarness"][i] - T["GRPO"][i]
    train_only = T["- w/o memory"][i] - T["GRPO"][i]
    assert train_only / total > 0.74            # 75.0% on ALFWorld, 78.9% on WebShop
    assert T["MemHarness"][i] - T["- w/o memory"][i] < 2.5   # +2.2 and +2.0 from retrieval

# 3. Verbatim replay hurts on one benchmark and helps on the other.
assert T["RL + Raw Memory"][1] < T["GRPO"][1]   # ALFWorld -6.3
assert T["RL + Raw Memory"][2] > T["GRPO"][2]   # WebShop  +6.5

# 4. "RL + Raw Memory" is the reproduced EvolveR baseline, not an internal ablation.
assert T["RL + Raw Memory"][0] == [84.8, 61.5, 95.7, 62.5, 61.9, 54.2]
assert T["RL + Raw Memory"][2] == 72.6          # identical to Table 1's EvolveR row

# 5. The OOD table's rows against the in-distribution table's.
OOD = {"MemHarness": [97.1, 73.7, 87.5, 92.9, 80.0, 84.2],
       "- w/o reconstr.": [94.1, 52.6, 87.5, 92.9, 83.3, 84.2],
       "- w/o memory": [100.0, 68.4, 91.7, 78.6, 70.0, 89.5],
       "RL + Raw Memory": [91.2, 68.4, 75.0, 78.6, 86.7, 57.9]}
assert T["MemHarness"][0] != OOD["MemHarness"]                    # differs, as expected
assert T["- w/o reconstr."][0] != OOD["- w/o reconstr."]          # differs in 4 of 6 cells
assert T["- w/o memory"][0] == OOD["- w/o memory"]                # identical in ALL SIX cells
assert abs(np.mean(OOD["- w/o memory"]) - 83.0) < 0.06            # and the same average
assert np.mean(OOD["MemHarness"]) - np.mean(OOD["- w/o memory"]) < 3.0

# 6. Counterfactual state edits: the behavioural shift is small and the WebShop floor is high.
BEH = {("ALFWorld", "match"): (46.0, 53.4, 0.6), ("ALFWorld", "edit"): (37.3, 56.3, 6.4),
       ("WebShop", "match"): (0.0, 27.9, 72.1), ("WebShop", "edit"): (0.0, 21.2, 78.8)}
for v in BEH.values():
    assert abs(sum(v) - 100.0) < 0.15
assert BEH[("WebShop", "match")][2] > 70        # 72.1% of WELL-MATCHED memories are rejected
assert BEH[("WebShop", "edit")][2] - BEH[("WebShop", "match")][2] < 7
assert BEH[("ALFWorld", "edit")][2] - BEH[("ALFWorld", "match")][2] < 6

# 7. Source-state ablation: reject rate and success rate are not the same signal.
SRC = {"correct": (8.7, 85.2), "no source": (7.8, 80.0), "random source": (13.3, 84.3)}
assert abs(SRC["no source"][0] - SRC["correct"][0]) < 1.0   # reject rate barely moves
assert SRC["correct"][1] - SRC["no source"][1] > 5.0        # success falls 5.2 points
assert SRC["random source"][0] - SRC["correct"][0] > 4.0    # reject rate jumps 4.6
assert abs(SRC["random source"][1] - SRC["correct"][1]) < 1.0   # success barely moves

print("all assertions passed")

Five results.

Most of the gain is a training effect, not a memory system. Against plain GRPO, MemHarness gains 8.8 points on ALFWorld and 9.5 on WebShop. With the memory bank fully disabled at inference, the same trained policy keeps 6.6 and 7.5 of those, or 75.0% and 78.9%. Retrieval at inference is worth 2.2 and 2.0 points. The paper states this plainly in its "Intrinsic Policy Improvement" paragraph, but the abstract and title do not, and the operational consequence is significant: the retrieval infrastructure is optional at serving time and worth roughly a quarter of the benefit. The reconstruction objective behaves as latent guidance during training, which is a different and arguably more interesting claim than "memory helps".

The reject decision is not the same signal as task success. Removing the source observation from the reconstruction context leaves the reject rate essentially unchanged (8.7% to 7.8%) while success falls 5.2 points, because the policy accepts misaligned guidance rather than declining more. Replacing the source with a mismatched one from a random memory raises the reject rate by 4.6 points and moves success by 0.9. So the policy does compare states, and the reject rate is a poor proxy for whether it is doing so usefully. Monitor both.

Reconstruction responds to state edits, but the shift is small on a large base rate. Under minimal factual edits that make a retrieved memory inapplicable, ALFWorld rejection moves from 0.6% to 6.4% and WebShop from 72.1% to 78.8%. WebShop is the striking column: the policy discards 72% of memories whose state matches, and the edit adds 6.7 points on top. That is consistent with WebShop's memory contributing almost nothing at test time (73.6 without memory against 75.6 with it) and it is what "learns to reject" means in practice for that environment.

The w/o memory row is bit-identical between the in-distribution and out-of-distribution tables. All six per-category values and the 83.0 average match exactly, while the full system's row and the w/o reconstruction row both differ between the two tables. Two evaluations described as running on "unseen room layouts and object placements" cannot reproduce six independent cells exactly. Either the OOD table's w/o memory row was carried over from the in-distribution evaluation, or the two evaluations are not as different as the captions say. Whichever it is, the OOD comparison's most important reference point (the intrinsic policy with memory off, at 83.0 against MemHarness's 85.9) should be re-derived from the released code before it is relied on.

The RL + Raw Memory row is a reproduced third-party baseline. Its six ALFWorld cells and its WebShop score are identical to Table 1's EvolveR (reproduced) row. That is fine and the paper is not hiding it, but it means the comparison between w/o reconstruction (79.6) and RL + Raw Memory (70.1) is across two systems, not a clean ablation of the training objective.

How to develop with it

Store the source state with the memory or none of this works. Every mechanism on this page depends on the policy being able to compare o_src against the current history. A memory bank of bare strings cannot support critique. Retrofit this into an existing store before anything else.

Reward format compliance separately and lightly. The 0.1 x R_format term encodes the agentic protocol (one action per step, retrieval between one and five times per episode, a single well-formed reasoning block) without competing with the sparse task reward at 10 for success. The retrieval-frequency band is doing real work: it prevents both the degenerate never-retrieve policy and the degenerate retrieve-every-step one.

Expect a cold start. The cold-start model, trained only for format alignment on 200 multi-turn trajectories plus 200 trajectory-summarisation examples per benchmark, scores 7.6 on ALFWorld, below the 14.5 of the untrained base model. Format alignment alone degrades capability; only the RL stage recovers and exceeds it. Do not read a post-SFT checkpoint as evidence of anything.

Let the memory bank start empty. It is populated during RL from principles the policy summarises out of its own trajectories, with semantic deduplication, empirical utility tracking, and periodic pruning of low-utility entries. A bank seeded from a different policy's trajectories is a different experiment.

Measure retrieve, accept, and reject counts per trajectory as training curves. The two environments converge to different policies: ALFWorld settles at 2 to 3 retrievals per trajectory, while WebShop keeps retrieving frequently and rejecting persistently. Those are learned, task-specific strategies, and their shape is the clearest available signal that the reconstruction head is doing something rather than degenerating.

How to run it in production

Decide explicitly whether you are serving the memory bank at all. Given that memory contributes 2.2 and 2.0 points at inference, and that serving it costs a retriever, an embedding model, a store, and per-step latency, the honest default for a latency-sensitive deployment is to train with reconstruction and serve without it, then measure whether your workload is one of the ones where the remaining quarter matters.

If you do serve it, log the reconstruction, not just the retrieval. The whole point of keeping memory explicit rather than parametric is traceability. That is only realised if the guidance the policy actually acted on is recorded beside the raw entry it came from. Agent observability covers the trace shape.

Track the reject rate as an operational metric with a known baseline. A reject rate of 8.7% on one environment and 56.0% on another is not a bug in either; it is task-dependent. What matters is drift from your own baseline, which indicates either memory-bank rot or a distribution shift in incoming states.

Prune on measured utility, not on age or size. The bank tracks empirical utility per entry and prunes low-utility ones. That is the correct discipline, and it is the same problem the filesystem-memory literature runs into from the other direction; see the filesystem as agent memory for what happens when a store is curated without a supersession contract.

Budget the rollout cost. Every training step samples a group of trajectories through a full environment interaction loop with an extra reconstruction generation per retrieval. The generation cost per step is higher than plain GRPO by the reconstruction tokens, and the environment cost dominates either way. RL rollout fleet sizing and async RL systems cover the infrastructure.

Failure modes

  • Reading the headline as a memory-system result. Three quarters of it survives with memory switched off. Attribute the gain correctly or you will build infrastructure for a quarter of the effect.
  • Storing principles without their source state. Critique becomes impossible and the system degenerates to replay.
  • Treating a low reject rate as health. Removing the source state leaves the reject rate flat while success drops 5 points. The rate alone cannot distinguish "nothing needed rejecting" from "nothing was checked".
  • Assuming replay is always harmful. It cost 6.3 points on ALFWorld and gained 6.5 on WebShop in this paper's own tables.
  • Evaluating the cold-start checkpoint. It scores below the untrained base model. Format alignment is not capability.
  • Swapping in a general model for the reconstruction step. Costs 7.5 points on ALFWorld. The head must be trained with the actor.
  • Carrying a memory bank across policy versions. The bank is populated by a specific policy's own summaries during its own training. Reusing it under a new policy is an untested configuration.

Open questions and validation

  • Two environments, one 7B backbone, no seed variance. Single runs per configuration with per-category cells over small task counts. Several of the ablation orderings turn on one or two episodes.
  • The duplicated OOD row. Until it is re-derived from the released code, the out-of-distribution comparison's baseline is unverified.
  • The intrinsic-improvement mechanism is unexplained. That training with a reconstruction objective improves the base policy is measured; why it does is not. It is consistent with several stories (an auxiliary reasoning task, a curriculum effect, extra tokens of computation per step) and the paper distinguishes none of them.
  • No scaling evidence. The authors name larger models and open-ended environments as future work. Whether the reconstruction skill is one a bigger model already has is exactly the question.
  • Memory-bank growth is unbounded in principle. Deduplication and utility pruning are described but their long-run behaviour over many training runs is not measured.

References

Related: The filesystem as agent memory · Agent context and memory · Agentic context management · Experience distillation for agents · GRPO · GRPO variants · Agentic RL · Async RL systems · Agent observability · Agent harness architecture · Always-on agents and persistent state · RL rollout fleet sizing · Glossary