Skip to content
Markdown

Reconstructing agent memory instead of replaying it

Scope: what happens between retrieving a past experience and acting on it, and how a reconstruction step can reduce state mismatch. This page covers the retrieve-critique-reconstruct decomposition, the paper's joint GRPO objective, the released implementation's different training path, and an ablation-level read of where the measured gains come from. The storage-side treatment is the filesystem as agent memory; the concept-level view is agent context and memory; the training machinery is GRPO.

Primary source: Wu, Fu, Wen, Yang, Zou, Mei, Wang, Zhang, Yang, Hu, Zhang, Shi, Cai, "MemHarness: Memory Is Reconstructed, Not Replayed", arXiv:2607.28272v1 [cs.AI], 30 July 2026 (Zhejiang University, Shanghai AI Laboratory, and others). Code audit: KnowledgeXLab/MemHarness commit 0cafec599c80d186eed64d507bbad12716001ad2, Apache-2.0.

What this page adds. The Python block is executed and asserted (Python 3.11, numpy 2.x). It reproduces every published macro-average from the per-category cells, decomposes the headline gain across ablation rows, flags a duplicated table row, and tests the released utility-credit rule at boundary and rejection cases. No agent was trained or run; every benchmark number is the paper's.

Scale caveat. One backbone (Qwen2.5-7B-Instruct), two benchmarks (ALFWorld and WebShop), one run per configuration with no seed variance reported. Read the mechanism, not the leaderboard.

Paper and release diverge on training. The paper's equations assign the trajectory advantage to reconstruction and action tokens. The released ALFWorld and WebShop scripts instead use the actor rollout worker for reconstruction and set train_memory_adaptor=false. The rollout code stores reconstruction completions in a separate adaptor buffer only when dedicated adaptor training is enabled, so the published scripts update action trajectories but do not directly apply GRPO loss to reconstruction completions. Shared actor weights can still change reconstruction behavior indirectly. The published scores were not reproduced from the release.

What it is

Most memory-augmented agents follow a retrieve-and-replay pattern: a retriever returns a stored trajectory or distilled principle, and it is inserted into the model context verbatim. That design conflates two different things, retrieval relevance and action-level applicability. A memory can be about exactly the right task and still be wrong for the current step, because it was formed under different environmental conditions.

MemHarness inserts an explicit step in between. At each decision step the same policy:

  1. Retrieves, optionally. The policy decides from its history window whether to query the memory bank at all, then emits a query and receives the top-k entries.
  2. Critiques and reconstructs. Each retrieved entry is a pair (e_i, o_src_i): an abstract strategy and the source observation it was formed under. The policy compares each source observation against the current history, then retains, revises, or rejects, emitting state-specific guidance g_t. If nothing applies it emits <EMPTY> and falls back to a self-reasoning prompt.
  3. Acts, conditioned on the reconstructed guidance rather than on the raw record.

The paper treats reconstruction as a latent variable and specifies end-to-end GRPO with a reward of 10 for success, 0 for failure, plus 0.1 x R_format. Its stated format criteria cover one valid <think> and <action> block per step, one to five memory retrievals per episode, and English-only output. The released actor-format implementation is different: it permits any positive number of <think> blocks because max_think_segments=0, requires one action per non-empty step, requires one to five retrievals per episode in the published scripts, and rejects CJK characters rather than enforcing English. The released scripts also omit reconstruction completions from direct GRPO updates, as noted above.

Keeping o_src beside the strategy is the design decision that makes any of this possible. Without a record of the state a memory came from, "does this still apply" is not answerable.

Why use it

Because injecting retrieved memory into a trained RL agent can make it worse, and the paper measures that directly. On ALFWorld, adding a memory bank to an RL-trained agent takes it from 76.4 to 70.1. The reported headline is that MemHarness reaches 85.2 on ALFWorld and 75.6 on WebShop from a 7B backbone, against 76.4 and 66.1 for plain GRPO.

The middle ground it occupies is real and worth naming. Parametric approaches internalise experience into weights, which gives state-conditioned behaviour but makes the influence of a particular experience difficult to inspect or revise. Explicit memory banks retain attributable records but usually replay them rigidly. MemHarness keeps the stored entry inspectable and also records the reconstructed guidance used for action generation.

When to use it (and when not)

Use it when your stored experiences are abstract and your states vary. The applicability gap is largest when a memory is a distilled principle rather than a concrete replayable action sequence, and when the environment presents high state variance. ALFWorld's room layouts and WebShop's page content are both in that regime.

Evaluate a memory-free serving variant. In the - w/o memory ablation, retrieval and reconstruction are disabled at inference and the policy scores 83.0 on ALFWorld against the full system's 85.2, and 73.6 against 75.6 on WebShop. Three quarters of the point improvement over plain GRPO remains. This makes memory-free serving a candidate for latency-sensitive deployments, not a general prescription from two single-run benchmarks.

Do not assume a generic rewriter is equivalent. Replacing the policy's internal reconstruction with a generic instruction-tuned model of the same family, keeping the actor unchanged, drops ALFWorld from 85.2 to 77.7 in this experiment. This rejects that particular zero-shot substitute; it does not establish that every reconstruction module needs the paper's stated joint-token RL path.

Do not generalise "replay causes negative transfer" from this paper. The sign flips between its own two environments: raw memory costs 6.3 points on ALFWorld and gains 6.5 on WebShop against the same GRPO baseline.

Architecture

flowchart LR
  ENV["Environment observation<br/>history window w = 3"] --> DEC{"retrieve?"}
  DEC -->|"no"| ACT
  DEC -->|"yes"| RET["Retriever, BGE-M3<br/>top-3 principles<br/>each with its source observation"]
  RET --> REC["Critique and reconstruct<br/>paper: top-3 retrieved principles<br/>release adaptor default: top-1"]
  REC -->|"applicable"| G["state-specific guidance g_t"]
  REC -->|"EMPTY"| SELF["self-reasoning prompt"]
  G --> ACT["Action generation<br/>same policy, same weights"]
  SELF --> ACT
  ACT --> ENV
  ACT -.->|"paper reward 10 / 0<br/>+ 0.1 x format"| GRPO["Actor GRPO update"]
  GRPO -.->|"paper: direct joint-token credit<br/>release scripts: indirect weight sharing only"| REC
  ACT -.->|"distil trajectories"| BANK["Memory bank<br/>Milvus + BGE-M3<br/>utility tracking and pruning"]
  BANK --> RET

Decision, reconstruction, and action generation use the same policy weights; BGE-M3 remains a separate retriever. Under the paper's joint-token objective, a reconstruction-token update therefore changes the actor. The released benchmark scripts do not place reconstruction completions in that direct loss path, which is the implementation divergence audited below.

How to use it

Executed audit of the published tables

import numpy as np

CATS = ["Pick", "Look", "Clean", "Heat", "Cool", "Pick2"]
# (ALFWorld per-category success rate, reported ALFWorld average, reported WebShop success rate)
T = {
 "GRPO":              ([90.8, 66.1, 89.3, 74.7, 72.5, 64.7], 76.4, 66.1),
 "RL + Raw Memory":   ([84.8, 61.5, 95.7, 62.5, 61.9, 54.2], 70.1, 72.6),
 "MemHarness":        ([87.0, 78.6, 97.0, 87.5, 71.4, 90.0], 85.2, 75.6),
 "- generic LLM rec": ([97.1, 52.6, 79.2, 78.6, 80.0, 78.9], 77.7, 71.8),
 "- w/o reconstr.":   ([94.1, 52.6, 79.2, 85.7, 76.7, 89.5], 79.6, 74.6),
 "- w/o memory":      ([100.0, 68.4, 91.7, 78.6, 70.0, 89.5], 83.0, 73.6),
}

# 1. Every published ALFWorld average is the unweighted mean of six categories.
for k, (row, avg, _) in T.items():
    assert abs(np.mean(row) - avg) < 0.06, k

# 2. Decompose the gain over plain GRPO into training effect and inference-time retrieval.
for i in (1, 2):
    total = T["MemHarness"][i] - T["GRPO"][i]
    train_only = T["- w/o memory"][i] - T["GRPO"][i]
    assert train_only / total > 0.74            # 75.0% on ALFWorld, 78.9% on WebShop
    assert T["MemHarness"][i] - T["- w/o memory"][i] < 2.5   # +2.2 and +2.0 from retrieval

# 3. Verbatim replay hurts on one benchmark and helps on the other.
assert T["RL + Raw Memory"][1] < T["GRPO"][1]   # ALFWorld -6.3
assert T["RL + Raw Memory"][2] > T["GRPO"][2]   # WebShop  +6.5

# 4. "RL + Raw Memory" numerically matches Table 1's reproduced EvolveR row.
EVOLVER = ([84.8, 61.5, 95.7, 62.5, 61.9, 54.2], 70.1, 72.6)
assert T["RL + Raw Memory"] == EVOLVER

# 5. The OOD table's rows against the in-distribution table's.
OOD = {"MemHarness": [97.1, 73.7, 87.5, 92.9, 80.0, 84.2],
       "- w/o reconstr.": [94.1, 52.6, 87.5, 92.9, 83.3, 84.2],
       "- w/o memory": [100.0, 68.4, 91.7, 78.6, 70.0, 89.5],
       "RL + Raw Memory": [91.2, 68.4, 75.0, 78.6, 86.7, 57.9]}
assert T["MemHarness"][0] != OOD["MemHarness"]                    # differs, as expected
assert T["- w/o reconstr."][0] != OOD["- w/o reconstr."]          # differs in 4 of 6 cells
assert T["- w/o memory"][0] == OOD["- w/o memory"]                # identical in ALL SIX cells
assert abs(np.mean(OOD["- w/o memory"]) - 83.0) < 0.06            # and the same average
assert np.mean(OOD["MemHarness"]) - np.mean(OOD["- w/o memory"]) < 3.0

# 6. Counterfactual state edits: the behavioural shift is small and the WebShop floor is high.
BEH = {("ALFWorld", "match"): (46.0, 53.4, 0.6), ("ALFWorld", "edit"): (37.3, 56.3, 6.4),
       ("WebShop", "match"): (0.0, 27.9, 72.1), ("WebShop", "edit"): (0.0, 21.2, 78.8)}
for v in BEH.values():
    assert abs(sum(v) - 100.0) < 0.15
assert BEH[("WebShop", "match")][2] > 70        # 72.1% of WELL-MATCHED memories are rejected
assert BEH[("WebShop", "edit")][2] - BEH[("WebShop", "match")][2] < 7
assert BEH[("ALFWorld", "edit")][2] - BEH[("ALFWorld", "match")][2] < 6

# 7. Source-state ablation: reject rate and success rate are not the same signal.
SRC = {"correct": (8.7, 85.2), "no source": (7.8, 80.0), "random source": (13.3, 84.3)}
assert abs(SRC["no source"][0] - SRC["correct"][0]) < 1.0   # reject rate barely moves
assert SRC["correct"][1] - SRC["no source"][1] > 5.0        # success falls 5.2 points
assert SRC["random source"][0] - SRC["correct"][0] > 4.0    # reject rate jumps 4.6
assert abs(SRC["random source"][1] - SRC["correct"][1]) < 1.0   # success barely moves

# 8. Released utility accounting uses a Beta(1,1) prior and credits every retrieved ID
# in a successful trajectory, including a memory that reconstruction later rejects.
def utility_score(use_count, success_count):
    if use_count < 0 or success_count < 0 or success_count > use_count:
        raise ValueError("invalid utility counters")
    return (success_count + 1) / (use_count + 2)

assert utility_score(0, 0) == 0.5
assert utility_score(3, 0) == 0.2
assert utility_score(3, 3) == 0.8
try:
    utility_score(2, 3)
    raise AssertionError("success_count > use_count must fail")
except ValueError:
    pass

def success_credits(retrieved_ids_by_step, episode_succeeded):
    if not episode_succeeded:
        return set()
    return {memory_id for step in retrieved_ids_by_step for memory_id in step}

credited = success_credits([["accepted", "rejected"], ["accepted"]], True)
assert credited == {"accepted", "rejected"}
assert success_credits([["accepted"]], False) == set()

print("all assertions passed")

Seven results follow.

The paper's joint-token objective is not the published script path. The paper assigns the trajectory advantage to reconstruction and action tokens. At audited commit 0cafec5, both training scripts generate reconstruction with the actor worker but set train_memory_adaptor=false; the separate reconstruction buffer is populated only when dedicated adaptor training is active. The release therefore trains actions conditioned on reconstructed guidance while reconstruction changes only indirectly through shared actor weights. This discrepancy blocks a code-level reproduction of the paper's stated learning mechanism.

Most of the point gain remains without memory at inference. Against plain GRPO, MemHarness gains 8.8 points on ALFWorld and 9.5 on WebShop. The - w/o memory inference ablation retains 6.6 and 7.5 points of those gaps, or 75.0% and 78.9%. Enabling retrieval and reconstruction adds 2.2 and 2.0 points over that ablation. These are single-run point estimates; they motivate a memory-free serving comparison but do not prove a general fraction of benefit.

Reject rate is not a proxy for task success. Removing the source observation leaves the ALFWorld reject rate nearly unchanged (8.7% to 7.8%) while success falls 5.2 points. Replacing the source with a random memory raises rejection by 4.6 points while success moves by 0.9. The source state improves the measured outcome even though the no-source variant remains useful, so it should not be described as strictly necessary.

Reconstruction output responds to state edits, but the shift sits on different base rates. Minimal factual edits raise ALFWorld rejection from 0.6% to 6.4% and WebShop rejection from 72.1% to 78.8%. These are offline reconstruction probes, not executed environment episodes. WebShop rejects most nominally matched memories in both conditions, which is consistent with the small two-point inference ablation gap but does not establish causality.

The duplicated OOD row is an unresolved reporting concern, not proof of reuse. The w/o memory row has the same six category values and 83.0 average in the in-distribution and OOD tables, while other rows change. Exact equality can occur, especially with small category counts, so the table alone cannot prove carry-over. The OOD reference row should nevertheless be regenerated from predictions before using its 2.9-point gap to the full system.

RL + Raw Memory numerically matches the reproduced EvolveR row. All six ALFWorld cells and the WebShop score are identical. The paper does not explicitly state that the two labels denote one run, so the equality is evidence of shared numbers rather than enough evidence to identify the implementation. Neither row is a clean removal of reconstruction from an otherwise fixed training pipeline.

Released utility credit is coarser than acceptance. A successful episode increments the success counter for every distinct retrieved memory ID observed in the trajectory, including entries later rejected by reconstruction. The resulting Laplace-smoothed score, (successes + 1) / (uses + 2), measures association between retrieval and episode success, not the causal utility of accepted guidance. Pruning on it can preserve rejected passengers or penalize useful memories retrieved during failed episodes.

How to develop with it

Store the source state when state comparison is part of the design. A bare principle does not let the reconstructor compare historical and current conditions. The no-source ablation still scores 80.0 on ALFWorld and 73.0 on WebShop, so source state is useful rather than logically indispensable. Store the minimal state fields needed to test a principle's preconditions and apply retention controls to both fields.

Treat the paper's reward and the release's reward as separate specifications. Both published scripts use outcome weight 1.0, format weight 0.1, and require one to five retrievals per episode. The release requires one action per non-empty step, permits multiple valid <think> blocks, and zeros format reward on CJK output rather than validating English. Pin the commit and test the actual parser; prose in the paper is not an executable protocol.

Evaluate the cold-start checkpoint separately. The paper uses 200 interaction trajectories and 200 trajectory-summarisation examples per benchmark. Its cold-start checkpoint scores 7.6 on ALFWorld against 14.5 for the base checkpoint under the reported evaluation. That comparison shows the format stage is not sufficient for this task; without repeated seeds or an evaluation-isolation study, it does not establish that format alignment itself caused the loss.

Let the benchmark bank start empty if reproducing the paper. The policy summarizes its own trajectories into the store. At the audited commit, both scripts enable utility pruning every 20 global steps below a score of 0.3 after at least three uses. WebShop retains the default 0.85 embedding-similarity deduplication, while ALFWorld explicitly disables insertion and retrieval deduplication. A bank seeded from another policy or a different deduplication setting is a different experiment.

Measure retrieve, accept, and reject counts per trajectory as training curves. The two environments converge to different policies: ALFWorld settles at 2 to 3 retrievals per trajectory, while WebShop keeps retrieving frequently and rejecting persistently. Those are learned, task-specific strategies, and their shape is the clearest available signal that the reconstruction head is doing something rather than degenerating.

How to run it in production

Compare memory-on and memory-off serving on the target workload. The paper's point estimates differ by 2.2 and 2.0 points, while memory-on serving adds BGE-M3, Milvus, reconstruction generations, and store operations. The cost-benefit decision requires repeated target-workload evaluation; the two benchmark ablations do not set a universal default.

If you do serve it, log the reconstruction, not just the retrieval. The whole point of keeping memory explicit rather than parametric is traceability. That is only realised if the guidance the policy actually acted on is recorded beside the raw entry it came from. Agent observability covers the trace shape.

Track the reject rate as an operational metric with a known baseline. A reject rate of 8.7% on one environment and 56.0% on another is not a bug in either; it is task-dependent. What matters is drift from your own baseline, which indicates either memory-bank rot or a distribution shift in incoming states.

Do not interpret the released utility score as acceptance-aware credit. It increments use at retrieval and success for every distinct retrieved ID in a successful episode. Reconstruction acceptance is not part of the update. Log acceptance beside the counters and validate pruning decisions against held-out outcomes before deleting entries. The broader supersession problem is covered in the filesystem as agent memory.

Budget rollout and update cost separately. Every retrieved memory adds a reconstruction generation. In the published release scripts, those tokens add inference cost during collection but no separate reconstruction GRPO update. Enabling the repository's dedicated adaptor-training path adds its own worker group, buffer, log-probability computation, and optimizer work; that path is not the paper's published script configuration. RL rollout fleet sizing and async RL systems cover the infrastructure.

How to maintain it

Pin the paper version and repository commit together. At startup, assert the effective values of mem_adaptor.use_actor_rollout_wg, mem_adaptor.train_memory_adaptor, mem_adaptor.retrieval_top_k, the actor format limits, deduplication thresholds, and utility-pruning settings. A change in any of them changes the mechanism being evaluated.

Version memory records with the policy checkpoint, embedding model, prompt schema, source-state schema, and summarizer revision. Re-embed after an embedding-model change; do not mix vector spaces in one Milvus collection. Before migrating a bank to a new policy, run memory-on, memory-off, and raw-replay controls on a fixed validation set.

Audit pruning by sampling deleted and retained entries with their use, success, acceptance, rejection, and downstream-outcome traces. The release's success counter cannot distinguish accepted from rejected memories, so utility-score drift must be interpreted with those extra labels. Back up the store before pruning and verify row count, schema, and retrieval recall after restoration.

Re-run the format parser's self-tests and an end-to-end failed-episode test after reward changes. The useful assertions are boundary cases: zero and six retrievals must fail the one-to-five constraint, multiple action blocks must fail, and non-English text outside the implementation's CJK pattern must reveal the difference from the paper's English-only wording.

Open questions and validation

  • Two environments, one 7B backbone, no seed variance. Single runs per configuration with per-category cells over small task counts. Several of the ablation orderings turn on one or two episodes.
  • Paper and released training paths differ. The published scripts do not directly optimize reconstruction completions. A faithful joint-token implementation, exact result artifacts, or an explanation of how the reported checkpoints were trained is required to resolve the mechanism.
  • The duplicated OOD row. Until it is regenerated from underlying predictions, the equality of all six w/o memory cells remains unexplained.
  • The intrinsic-improvement mechanism is unexplained. The memory-off checkpoint outperforms plain GRPO in the reported tables, but the study does not distinguish shared-weight transfer, changed training contexts, curriculum effects, or additional per-step computation.
  • No scaling evidence. The authors name larger models and open-ended environments as future work. Whether the reconstruction skill is one a bigger model already has is exactly the question.
  • Top-k differs between paper and release. The paper states top-3 retrieval. The released memory store retrieves three, but the reconstruction adaptor consumes one by default. The configuration used for the reported scores is not established by the repository alone.
  • Memory-bank growth is unbounded in principle. Utility pruning is enabled in both scripts, but ALFWorld disables semantic deduplication and long-run growth across repeated training runs is not measured.

Failure modes

  • Claiming release-level reproduction of the joint objective. Shared weights do not mean reconstruction completions are present in the actor loss. Inspect the batch and optimizer path.
  • Reading the headline as purely a serving-memory result. Most of the point gap over plain GRPO remains in the memory-off ablation, while the uncertainty of that fraction is unreported.
  • Treating source state as strictly necessary. Removing it hurts the reported scores but leaves a functioning policy; describe the measured decrement.
  • Treating low rejection as health. The no-source ablation leaves rejection nearly flat while ALFWorld success falls 5.2 points.
  • Assuming replay is always harmful. RL + Raw Memory trails plain GRPO by 6.3 points on ALFWorld and leads it by 6.5 on WebShop.
  • Assuming utility credit reflects accepted guidance. The release credits every retrieved ID in successful episodes, including rejected entries.
  • Carrying a memory bank across policy or embedding revisions. The bank contains policy-generated summaries and model-specific vectors; reuse without migration and controls changes both retrieval and memory content.

References

Related: The filesystem as agent memory · Agent context and memory · Agentic context management · Experience distillation for agents · GRPO · GRPO variants · Agentic RL · Async RL systems · Agent observability · Agent harness architecture · Always-on agents and persistent state · RL rollout fleet sizing · Glossary