Markdown

ReContext (recursive evidence replay)¶

Scope: a training-free inference technique for long-context reasoning that preserves the original context. It accumulates question-cue attention from a fixed set of layer-head pairs, copies selected context spans as grounded evidence, and inserts the accumulated evidence between the context and question before generation. Proposed in "ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning" (Zhao et al., 2026), it complements context reduction by emphasizing selected evidence without removing unselected text.

The numpy block implements the published exponential cue-score recurrence on a fixed candidate vector and validates recursive pool bookkeeping with a different attention tensor per replay round. It does not run a transformer or reproduce the paper's benchmark. Treat the reported results as one paper's measurements and re-evaluate the released implementation on the target model and workload.

flowchart LR
  CTX["Original context C (128K tokens, retained)"] --> SCORE["Per cue: average selected heads;<br/>r(t) = Normalize(a(t) + 0.75 r(t-1))"]
  Q["Question q (cue tokens, prompt suffix)"] --> SCORE
  SCORE --> TOPK["Top-K new positions"]
  TOPK --> SPAN["Materialize to containing sentence"]
  SPAN -->|"dedup vs pool"| POOL["Evidence pool phi(E)"]
  POOL -->|"round < R: rescan C excluding pool"| SCORE
  POOL -->|"round = R"| REPLAY["x+ = [C ; phi(E) ; q]"]
  CTX --> REPLAY
  Q --> REPLAY
  REPLAY --> GEN["Generation"]

What it is¶

ReContext adds read passes and a replay scaffold to a normal long-context call, using the same frozen model. For each of the last L <= w cue positions, with w = 8 in the paper, it averages attention over a fixed set of selected layer-head pairs H:

a_i^(t) = (1 / |H|) * sum over (l,h) in H of A^(l,h)_(t,i)
r^(t) = Normalize(a^(t) + lambda * r^(t-1)), lambda = 0.75

After the cue sequence, candidate positions are ranked by r. Selected tokens are mapped to containing sentences or local spans and copied into an ordered evidence pool phi(E). The final prompt is x+ = [C ; phi(E) ; q]: context first, replayed evidence near the question, then the question. The original context is retained rather than pruned or summarized.

Why use it¶

No context removal. The full original context stays in the prompt. Selection can still miss relevant evidence, so retaining the context is not the same as lossless evidence identification.
No training, no external index. The scoring signal is the model's own attention on the current prompt; there is no retriever to build, no embedding index to maintain, and no fine-tuning step.
Measured gains across model scale. Averaged over eight accuracy metrics (one per dataset) across three backbones, mean accuracy rises from 0.24 (vanilla full-context decoding) to 0.30, a 24.6 percent relative gain, and ReContext posts the best average rank of any method tested on all three backbones: 1.00 on Qwen3-4B, 1.46 on Qwen3-8B, 1.29 on Llama3-8B (of six methods each).
Lower measured runtime than the strongest competing accuracy baseline. On the paper's CLIPPER run with Llama3-8B at 128K, ReContext took 62 minutes, vanilla took 44 minutes, and DySCO took 2 hours 13 minutes. These are end-to-end benchmark runtimes, not per-request decode latency.

When to use it (and when not)¶

Use it when the model implementation exposes the required attention readout, the task is long-context QA or evidence-grounded reasoning, and the measured accuracy gain justifies extra read passes and a longer prompt. The paper's 62-versus-44-minute CLIPPER result is a workload-level runtime ratio, not a general latency multiplier.
Do not use it against closed-source hosted APIs that expose only text completions: the paper states plainly that ReContext "requires access to model-internal relevance signals, which limits its direct use with closed-source APIs that do not expose attention or similar scoring information."
Do not expect it to help if the deployment already truncates or heavily compresses context before the model sees it; ReContext's gain comes from resurfacing signal already present in an intact long context, not from making a short context smarter.
Evidence caveat. Demonstrated on Qwen3-4B, Qwen3-8B, and Llama3-8B at a fixed 128K context length across eight QA-style benchmarks (Natural Questions, TriviaQA, HotpotQA, PopQA, NarrativeQA, InfBench QA, InfBench MC, CLIPPER). It is one 2026 paper; validate the head-set choice and gain on your own model family and task before trusting the reported margins.

Architecture¶

Scoring and replay share the frozen generation weights. The released configuration pins 16 layer-head pairs per tested backbone, but neither the paper nor repository documents a general procedure for selecting those heads. Each round reruns the readout on [C; pool_so_far; q]; sentence-text deduplication prevents an already inserted span from growing the pool again. The paper describes R = 2 as the main recursive setting, while the released Qwen3-4B sweep uses dataset-specific values from 1 to 3.

sequenceDiagram
  participant Q as Question (cue tokens)
  participant M as Model (frozen weights)
  participant C as Original context
  participant P as Evidence pool
  Q->>M: forward pass over [C; pool_so_far; q]
  M-->>M: read attention A^(l,h) for fixed head set H
  M->>C: score every position i in C (pool excluded)
  C-->>P: top-K positions, mapped to containing sentences, deduped
  P->>M: round < R: repeat scoring with pool excluded
  P->>M: round = R: replay [C; pool; q], generate final answer

Core mechanism (runnable): scoring, needle recovery, and the recursive pool¶

This numpy-only block implements r(t) = Normalize(a(t) + lambda*r(t-1)) on a fixed context candidate vector and checks it against a slow reference. It also validates sentence deduplication, empty and uniform inputs, informative versus noise heads, and two recursive rounds supplied with distinct attention tensors. It validates the scoring arithmetic and pool controller, not transformer attention extraction or benchmark accuracy. Run: python3 recontext_evidence.py.

# recontext_evidence.py -- core mechanism, runnable (numpy only).
# ReContext averages selected heads at each cue token, then applies
# r(t) = Normalize(a(t) + lambda*r(t-1)) across the cue sequence. Top-K
# context positions are materialized to sentences and replayed after the
# original context. A transformer would produce a new attention tensor each round.
import numpy as np

def relevance_scores(attn, head_set, decay_factor=0.75):
    # attn: (heads, cue_tokens, context_len) attention weights, each row sums to 1.
    # head_set flattens the selected layer-head pairs H to one axis in this model.
    assert 0.0 <= decay_factor <= 1.0
    assert len(head_set) > 0
    _, _, ctx = attn.shape
    if ctx == 0:
        return np.zeros(0)
    per_cue = attn[head_set].mean(axis=0)          # (cue_tokens, context_len)
    scores = np.zeros(ctx)
    for current in per_cue:
        scores = current + decay_factor * scores
        total = scores.sum()
        assert total > 0.0
        scores /= total
    return scores

def relevance_scores_reference(attn, head_set, decay_factor=0.75):
    # Slow reference: explicit head averaging, recurrence, and normalization per cue.
    _, cues, ctx = attn.shape
    if ctx == 0:
        return np.zeros(0)
    scores = np.zeros(ctx)
    for u in range(cues):
        current = np.zeros(ctx)
        for h in head_set:
            current += attn[h, u]
        current /= len(head_set)
        scores = current + decay_factor * scores
        scores /= scores.sum()
    return scores

def sentence_of(pos, boundaries):
    # boundaries: sorted list of (start, end) exclusive spans partitioning the context.
    for start, end in boundaries:
        if start <= pos < end:
            return start, end
    raise ValueError(f"position {pos} not covered by any sentence span")

def exclude_spans_of(exclude_positions, boundaries):
    return {sentence_of(int(p), boundaries) for p in exclude_positions} if exclude_positions else set()

def select_evidence(attn, head_set, boundaries, exclude, top_k):
    # One ReContext round: score context-only positions, pick top_k NEW positions
    # (paper ablation: "context only" beats "full prompt" as the scoring source),
    # materialize each to its containing sentence, dedup against the running pool.
    scores = relevance_scores(attn, head_set)
    order = np.argsort(-scores)
    excluded_spans = exclude_spans_of(exclude, boundaries)
    new_spans = []
    for pos in order:
        if len(new_spans) >= top_k:
            break
        span = sentence_of(int(pos), boundaries)
        if span not in new_spans and span not in excluded_spans:
            new_spans.append(span)
    return new_spans, scores

def recursive_rounds(attn_rounds, head_set, boundaries, top_k):
    # Each supplied tensor represents a fresh read pass over [C; pool_so_far; q].
    pool_positions, pool_spans, per_round_pools = set(), [], []
    for attn in attn_rounds:
        new_spans, _ = select_evidence(attn, head_set, boundaries, pool_positions, top_k)
        for span in new_spans:
            if span not in pool_spans:
                pool_spans.append(span)
                pool_positions.update(range(span[0], span[1]))
        per_round_pools.append(list(pool_spans))
    return per_round_pools

rng = np.random.default_rng(0)
CTX_LEN, CUE_TOKENS, HEADS = 400, 8, 12
INFORMATIVE_HEADS = [2, 5, 9]             # the paper's "fixed set of selected layer-head pairs" H
NEEDLE_POS = 317                          # deep in the context, beyond any small truncation window
boundaries = [(i, i + 10) for i in range(0, CTX_LEN, 10)]   # 10-token "sentences"

def build_attention(informative_heads, needle_pos, boost=40.0, noise_scale=1.0):
    attn = np.zeros((HEADS, CUE_TOKENS, CTX_LEN))
    for h in range(HEADS):
        logits = rng.normal(0, noise_scale, size=(CUE_TOKENS, CTX_LEN))
        if h in informative_heads:
            logits[:, needle_pos] += boost          # retrieval head: sharp peak at the needle
        logits -= logits.max(axis=1, keepdims=True)
        exp = np.exp(logits)
        attn[h] = exp / exp.sum(axis=1, keepdims=True)   # softmax rows: valid attention distribution
    return attn

attn = build_attention(INFORMATIVE_HEADS, NEEDLE_POS)

# 1) Equivalence: vectorized aggregation matches the slow per-head, per-cue-token reference.
fast = relevance_scores(attn, INFORMATIVE_HEADS)
slow = relevance_scores_reference(attn, INFORMATIVE_HEADS)
assert np.allclose(fast, slow), "vectorized a_i must match the explicit reference sum"

# 2) Needle recovery: selecting the informative head set surfaces the needle's sentence in round 1.
needle_span = sentence_of(NEEDLE_POS, boundaries)
spans_r1, scores = select_evidence(attn, INFORMATIVE_HEADS, boundaries, set(), top_k=8)
assert needle_span in spans_r1, "ReContext with informative heads must recover the needle span"
assert int(np.argmax(scores)) == NEEDLE_POS, "needle position must be the top-scored token"

# 3) Vanilla baseline (fixed-window truncation, no replay) misses a needle placed beyond the window:
#    this is the failure mode the evidence pool exists to fix.
TRUNCATION_WINDOW = 128
assert NEEDLE_POS >= TRUNCATION_WINDOW, "needle must sit outside the vanilla truncation window"
assert NEEDLE_POS not in set(range(TRUNCATION_WINDOW)), "vanilla baseline must not see the needle"

# 4) Adversarial: an all-noise head set (no informative heads in H) must not reliably recover the
#    needle. ReContext's gain is conditional on H actually containing informative heads.
NOISE_HEADS = [h for h in range(HEADS) if h not in INFORMATIVE_HEADS][:3]
spans_noise, scores_noise = select_evidence(attn, NOISE_HEADS, boundaries, set(), top_k=8)
assert needle_span not in spans_noise, "an uninformative head set must not reliably surface the needle"
assert int(np.argmax(scores_noise)) != NEEDLE_POS, "noise heads must not rank the needle first"

# 5) Edge / degenerate: an all-uniform attention distribution (no signal at all) must not crash and
#    must not spuriously "find" the needle: top-K degrades to an arbitrary tie-break, not a claim.
uniform_attn = np.full((HEADS, CUE_TOKENS, CTX_LEN), 1.0 / CTX_LEN)
spans_uniform, scores_uniform = select_evidence(uniform_attn, INFORMATIVE_HEADS, boundaries, set(), top_k=8)
assert len(spans_uniform) == 8, "degenerate uniform input must still return top_k spans, no crash"
assert np.allclose(scores_uniform, scores_uniform[0]), "uniform attention must yield uniform scores"

# 5b) Boundary: an empty context (zero length) must return no spans, not raise.
empty_attn = np.zeros((HEADS, CUE_TOKENS, 0))
empty_scores = relevance_scores(empty_attn, INFORMATIVE_HEADS)
assert empty_scores.shape == (0,), "empty context must produce an empty score vector, no crash"

# 6) Recursive rounds: later reads have distinct attention and the pool only grows.
NEEDLE_POS_2 = 55
attn_round_1 = build_attention(INFORMATIVE_HEADS, NEEDLE_POS)
attn_round_2 = build_attention(INFORMATIVE_HEADS, NEEDLE_POS_2)
assert not np.allclose(
    relevance_scores(attn_round_1, INFORMATIVE_HEADS),
    relevance_scores(attn_round_2, INFORMATIVE_HEADS),
), "a new replay round must be able to change the readout"
pools = recursive_rounds(
    [attn_round_1, attn_round_2], INFORMATIVE_HEADS, boundaries, top_k=1
)
assert len(pools[0]) <= len(pools[1]), "the evidence pool must not shrink across rounds"
assert set(pools[0]).issubset(set(pools[1])), "round 2 must retain round 1's picks (dedup, not replace)"
assert needle_span in pools[0], "round 1 must retain the first needle sentence"
assert sentence_of(NEEDLE_POS_2, boundaries) in pools[1], "round 2 must add the new needle sentence"

print("ReContext evidence-pool mechanism: PASS")
print("  needle span recovered (informative H):", needle_span, "top-1 pos:", int(np.argmax(scores)))
print("  noise-head negative control missed needle:", int(np.argmax(scores_noise)) != NEEDLE_POS)
print("  round-1 pool size:", len(pools[0]), " round-2 pool size:", len(pools[1]))

Executed output:

ReContext evidence-pool mechanism: PASS
  needle span recovered (informative H): (310, 320) top-1 pos: 317
  noise-head negative control missed needle: True
  round-1 pool size: 1  round-2 pool size: 2

How it works¶

For every cue token, Equation 1 averages attention across the fixed head set H; Equation 2 then combines that vector with the normalized result from the preceding cue using lambda = 0.75. The implementation reruns the read pass on [C; pool_so_far; q] at each round, ranks candidate positions, maps selected positions to sentences, and removes sentence-text duplicates before extending the pool. The final generation prompt is [C; phi(E); q]. Table 4 favors original-context candidates in its ablation, but some released dataset configurations specify full_prompt; that upstream inconsistency requires dataset-level validation rather than a universal context-only rule.

How to use it¶

The paper body and released scripts establish the following starting points, not portable defaults:

Cue recurrence: use at most the last eight cue tokens and lambda = 0.75, matching the paper.
Head set: released configurations list 16 layer-head pairs for each tested backbone. No general head-selection algorithm is published, so a new model requires its own evidence-recovery evaluation.
Rounds and K: the paper describes R = 2 and commonly reports K = 8. The released Qwen3-4B sweep varies R from 1 to 3 and K from 8 to 32 by dataset.
Candidate source: prefer the paper's context-only ablation as an initial setting, but reproduce both context and full_prompt because the released scripts use both.
Version pin: record the repository commit, model revision, head pairs, dataset prompt template, R, K, w, and candidate source with every result.

How to integrate with it¶

ReContext is a prompt-construction wrapper around an existing model, not a new trainer or a new serving engine.

Use a model implementation that exposes the required attention. The released repository supplies custom Transformers model classes. A stock text-generation or vLLM API is insufficient unless a tested integration reproduces the same layer-head readout and prompt bookkeeping.
Budget R read passes plus final generation. Each round obtains new attention from [C; pool_so_far; q]; it is not a loop over one stored attention tensor.
No parameter training or external retrieval index is required. Head selection and workload evaluation remain deployment work.
Composes with, does not replace, context reduction. Context and memory's hierarchical reduction (measure, compact, summarize) shrinks what the model sees; ReContext runs on the unshrunk context and adds a curated duplicate of the relevant parts. Use reduction when the context genuinely exceeds the window; use ReContext when the context fits but the model still under-uses the middle of it (the lost-in-the-middle effect that page documents).

How to run it in production¶

Budget and measure the runtime premium. The paper reports 62 minutes for ReContext and 44 minutes for vanilla on its CLIPPER Llama3-8B 128K evaluation. That end-to-end benchmark ratio is not a per-request or decode-latency guarantee.
It will not work behind a closed API. If your serving path is a hosted completions endpoint with no attention access, ReContext cannot run at all; the paper states this as a hard limitation, not a tuning problem.
Cap R and K per route. Every additional round adds a read pass, and every selected sentence lengthens final prefill. Use measured bounds from the route's latency and accuracy evaluation.
Monitor pool growth, duplicate rate, span lengths, and answer quality. Repeatedly irrelevant or duplicate evidence can indicate weak head transfer, unsuitable prompt boundaries, or a workload on which replay provides no benefit.

How to maintain it¶

Treat H, w, K, and R as version-specific. They come from one paper's experiments on Qwen3-4B/8B and Llama3-8B at 128K; a different model family, context length, or task distribution should re-derive them, not inherit them by default.
Keep a vanilla full-context baseline. ReContext can legitimately trail it on a different workload; a regression is evidence to inspect selection quality and overhead, not proof of misconfiguration.
Test recurrence and controller invariants on every change. The numpy reference checks normalization, vectorized equivalence, distinct round readouts, sentence deduplication, and monotonic pool growth.

Results¶

Average rank across six methods (Vanilla, AttnSharp, DySCO, A-MEM, DAC, ReContext) on eight long-context QA benchmarks at 128K tokens, per backbone (lower rank is better; ReContext is best on all three):

Backbone	Vanilla	AttnSharp	DySCO	A-MEM	DAC	ReContext
Qwen3-4B	4.39	4.25	4.00	3.57	3.79	1.00
Qwen3-8B	3.96	4.50	3.25	4.21	3.61	1.46
Llama3-8B	3.25	3.29	4.57	3.43	5.18	1.29

Selected per-dataset accuracy, Qwen3-4B, 128K tokens (Vanilla vs ReContext): Natural Questions 0.02 to 0.08, TriviaQA 0.04 to 0.30, HotpotQA 0.00 to 0.08, PopQA 0.00 to 0.07, NarrativeQA 0.02 to 0.07, InfBench QA 0.09 to 0.12, InfBench MC 0.51 to 0.55, CLIPPER 0.38 to 0.52. Averaged over all eight accuracy metrics and all three backbones, mean accuracy rises from 0.24 (vanilla) to 0.30 (ReContext), a 24.6 percent relative gain.

Efficiency (CLIPPER, Llama3-8B, 128K tokens): Vanilla 44 min, AttnSharp 46 min, DAC 34 min, A-MEM 50 min, DySCO 2h 13min, ReContext 62 min. ReContext is slower than vanilla, AttnSharp, and DAC, but well under half of DySCO's runtime for the strongest-competing accuracy.

Failure modes¶

Closed-source API deployments. No attention access means no scoring signal; the method does not degrade gracefully to something weaker, it simply cannot run.
Misidentified head set H. The synthetic negative test above shows the mechanism's dependency: uninformative heads can produce plausible but irrelevant evidence.
Unbounded R or K. Larger settings add read work and final-prefill tokens without a guaranteed accuracy gain.
Candidate-source drift. Table 4 favors context-only scoring, while released configurations include full_prompt. Copying either setting without recording and evaluating it makes results irreproducible.
Runtime extrapolation. The reported 62-to-44-minute CLIPPER comparison is one benchmark runtime, not a general latency multiplier.

References¶

Zhao, Qiu, Wei, Bei, Liu, Chen, Lourentzou, Tong, He, ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning (arXiv 2607.02509): https://arxiv.org/abs/2607.02509
ReContext reference implementation, audited commit ea14e9e45e9dac7f333b754abf16521bf3a86e4e: https://github.com/Yanjun-Zhao/ReContext/tree/ea14e9e45e9dac7f333b754abf16521bf3a86e4e
Liu et al., Lost in the Middle: How Language Models Use Long Contexts (the effect ReContext targets): https://arxiv.org/abs/2307.03172