Copy for agents Markdown

ReContext (recursive evidence replay)

Scope: a training-free inference technique for long-context reasoning that preserves the original context. It accumulates question-cue attention from a fixed set of layer-head pairs, copies selected context spans as grounded evidence, and inserts the accumulated evidence between the context and question before generation. Proposed in "ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning" (Zhao et al., 2026), it complements context reduction by emphasizing selected evidence without removing unselected text.

Executed here: the numpy block (cue-score recurrence and recursive pool bookkeeping on synthetic tensors), a real CPU attention-extraction run on Qwen/Qwen2.5-0.5B-Instruct with torch==2.13.0+cpu and transformers==5.14.1 (eager attention readout, offset-mapping span alignment, cue scoring, top-k span selection, per-step generation attention), and, in the production adapter section, a real scorer HTTP service (wraps the toy-model extraction demo behind http.server, tested with a real client round trip) and a real reloadable rollback config (file-mtime-polled, tested by rewriting the file from the same process and confirming the change takes effect with no restart). Not executed here: the paper's 128K benchmarks, its pinned retrieval-head sets, and the generate() call to a live vLLM engine, which remains a reference template since it needs a GPU-backed server this sandbox does not have. Treat the reported results as one paper's measurements and re-evaluate the released implementation on the target model and workload.

Independently verified beyond the numpy model: the released repository was cloned at the pinned commit and its real attention-extraction code was run end to end. transformers==5.14.1 and 4.57.1 both fail to import the repo's custom model classes (TypeError: check_model_inputs() missing 1 required positional argument: 'func', a real version incompatibility); the repository's own env.yml pins transformers==4.57.3, and importing against that exact pin succeeds. With that pin, a tiny randomly-initialized 2-layer RescaleQwen3ForCausalLM was constructed and model.rescale_generate(input_ids, rescale_config=...) was called for real on CPU: it ran a real forward pass with output_attentions=True, extracted and decayed real per-head attention via the repository's own _aggregate_head_attention/_apply_importance_decay functions, materialized sentence spans via NLTK's sent_tokenize, replayed them, and returned real generated token ids plus a real 5-field logging dict. This validates the released mechanism's code path on a toy model, not the paper's accuracy claims on a real 4B/8B backbone; no pretrained weights were downloaded. See "Executed: the released attention-extraction and replay code" below for the full transcript.

flowchart LR
  CTX["Original context C (128K tokens, retained)"] --> SCORE["Per cue: average selected heads;<br/>r(t) = Normalize(a(t) + 0.75 r(t-1))"]
  Q["Question q (cue tokens, prompt suffix)"] --> SCORE
  SCORE --> TOPK["Top-K new positions"]
  TOPK --> SPAN["Materialize to containing sentence"]
  SPAN -->|"dedup vs pool"| POOL["Evidence pool phi(E)"]
  POOL -->|"round < R: rescan C excluding pool"| SCORE
  POOL -->|"round = R"| REPLAY["x+ = [C ; phi(E) ; q]"]
  CTX --> REPLAY
  Q --> REPLAY
  REPLAY --> GEN["Generation"]

What it is

ReContext adds read passes and a replay scaffold to a normal long-context call, using the same frozen model. For each of the last L <= w cue positions, with w = 8 in the paper, it averages attention over a fixed set of selected layer-head pairs H:

a_i^(t) = (1 / |H|) * sum over (l,h) in H of A^(l,h)_(t,i)
r^(t) = Normalize(a^(t) + lambda * r^(t-1)), lambda = 0.75

After the cue sequence, candidate positions are ranked by r. Selected tokens are mapped to containing sentences or local spans and copied into an ordered evidence pool phi(E). The final prompt is x+ = [C ; phi(E) ; q]: context first, replayed evidence near the question, then the question. The original context is retained rather than pruned or summarized.

Why use it

• No context removal. The full original context stays in the prompt. Selection can still miss relevant evidence, so retaining the context is not the same as lossless evidence identification.
• No training, no external index. The scoring signal is the model's own attention on the current prompt; there is no retriever to build, no embedding index to maintain, and no fine-tuning step.
• Measured gains across model scale. Averaged over eight accuracy metrics (one per dataset) across three backbones, mean accuracy rises from 0.24 (vanilla full-context decoding) to 0.30, a 24.6 percent relative gain, and ReContext posts the best average rank of any method tested on all three backbones: 1.00 on Qwen3-4B, 1.46 on Qwen3-8B, 1.29 on Llama3-8B (of six methods each).
• Lower measured runtime than the strongest competing accuracy baseline. On the paper's CLIPPER run with Llama3-8B at 128K, ReContext took 62 minutes, vanilla took 44 minutes, and DySCO took 2 hours 13 minutes. These are end-to-end benchmark runtimes, not per-request decode latency.

When to use it (and when not)

• Use it when the model implementation exposes the required attention readout, the task is long-context QA or evidence-grounded reasoning, and the measured accuracy gain justifies extra read passes and a longer prompt. The paper's 62-versus-44-minute CLIPPER result is a workload-level runtime ratio, not a general latency multiplier.
• Do not use it against closed-source hosted APIs that expose only text completions: the paper states plainly that ReContext "requires access to model-internal relevance signals, which limits its direct use with closed-source APIs that do not expose attention or similar scoring information."
• Do not expect it to help if the deployment already truncates or heavily compresses context before the model sees it; ReContext's gain comes from resurfacing signal already present in an intact long context, not from making a short context smarter.
• Evidence caveat. Demonstrated on Qwen3-4B, Qwen3-8B, and Llama3-8B at a fixed 128K context length across eight QA-style benchmarks (Natural Questions, TriviaQA, HotpotQA, PopQA, NarrativeQA, InfBench QA, InfBench MC, CLIPPER). It is one 2026 paper; validate the head-set choice and gain on your own model family and task before trusting the reported margins.

Architecture

Scoring and replay share the frozen generation weights. The released configuration pins 16 layer-head pairs per tested backbone, but neither the paper nor repository documents a general procedure for selecting those heads. Each round reruns the readout on [C; pool_so_far; q]; sentence-text deduplication prevents an already inserted span from growing the pool again. The paper describes R = 2 as the main recursive setting, while the released Qwen3-4B sweep uses dataset-specific values from 1 to 3.

sequenceDiagram
  participant Q as Question (cue tokens)
  participant M as Model (frozen weights)
  participant C as Original context
  participant P as Evidence pool
  Q->>M: forward pass over [C; pool_so_far; q]
  M-->>M: read attention A^(l,h) for fixed head set H
  M->>C: score every position i in C (pool excluded)
  C-->>P: top-K positions, mapped to containing sentences, deduped
  P->>M: round < R: repeat scoring with pool excluded
  P->>M: round = R: replay [C; pool; q], generate final answer

Core mechanism (runnable): scoring, needle recovery, and the recursive pool

This numpy-only block implements r(t) = Normalize(a(t) + lambda*r(t-1)) on a fixed context candidate vector and checks it against a slow reference. It also validates sentence deduplication, empty and uniform inputs, informative versus noise heads, and two recursive rounds supplied with distinct attention tensors. It validates the scoring arithmetic and pool controller, not transformer attention extraction or benchmark accuracy. Run: python3 recontext_evidence.py.

# recontext_evidence.py -- core mechanism, runnable (numpy only).
# ReContext averages selected heads at each cue token, then applies
# r(t) = Normalize(a(t) + lambda*r(t-1)) across the cue sequence. Top-K
# context positions are materialized to sentences and replayed after the
# original context. A transformer would produce a new attention tensor each round.
import numpy as np

def relevance_scores(attn, head_set, decay_factor=0.75):
    # attn: (heads, cue_tokens, context_len) attention weights, each row sums to 1.
    # head_set flattens the selected layer-head pairs H to one axis in this model.
    assert 0.0 <= decay_factor <= 1.0
    assert len(head_set) > 0
    _, _, ctx = attn.shape
    if ctx == 0:
        return np.zeros(0)
    per_cue = attn[head_set].mean(axis=0)          # (cue_tokens, context_len)
    scores = np.zeros(ctx)
    for current in per_cue:
        scores = current + decay_factor * scores
        total = scores.sum()
        assert total > 0.0
        scores /= total
    return scores

def relevance_scores_reference(attn, head_set, decay_factor=0.75):
    # Slow reference: explicit head averaging, recurrence, and normalization per cue.
    _, cues, ctx = attn.shape
    if ctx == 0:
        return np.zeros(0)
    scores = np.zeros(ctx)
    for u in range(cues):
        current = np.zeros(ctx)
        for h in head_set:
            current += attn[h, u]
        current /= len(head_set)
        scores = current + decay_factor * scores
        scores /= scores.sum()
    return scores

def sentence_of(pos, boundaries):
    # boundaries: sorted list of (start, end) exclusive spans partitioning the context.
    for start, end in boundaries:
        if start <= pos < end:
            return start, end
    raise ValueError(f"position {pos} not covered by any sentence span")

def exclude_spans_of(exclude_positions, boundaries):
    return {sentence_of(int(p), boundaries) for p in exclude_positions} if exclude_positions else set()

def select_evidence(attn, head_set, boundaries, exclude, top_k):
    # One ReContext round: score context-only positions, pick top_k NEW positions
    # (paper ablation: "context only" beats "full prompt" as the scoring source),
    # materialize each to its containing sentence, dedup against the running pool.
    scores = relevance_scores(attn, head_set)
    order = np.argsort(-scores)
    excluded_spans = exclude_spans_of(exclude, boundaries)
    new_spans = []
    for pos in order:
        if len(new_spans) >= top_k:
            break
        span = sentence_of(int(pos), boundaries)
        if span not in new_spans and span not in excluded_spans:
            new_spans.append(span)
    return new_spans, scores

def recursive_rounds(attn_rounds, head_set, boundaries, top_k):
    # Each supplied tensor represents a fresh read pass over [C; pool_so_far; q].
    pool_positions, pool_spans, per_round_pools = set(), [], []
    for attn in attn_rounds:
        new_spans, _ = select_evidence(attn, head_set, boundaries, pool_positions, top_k)
        for span in new_spans:
            if span not in pool_spans:
                pool_spans.append(span)
                pool_positions.update(range(span[0], span[1]))
        per_round_pools.append(list(pool_spans))
    return per_round_pools

rng = np.random.default_rng(0)
CTX_LEN, CUE_TOKENS, HEADS = 400, 8, 12
INFORMATIVE_HEADS = [2, 5, 9]             # the paper's "fixed set of selected layer-head pairs" H
NEEDLE_POS = 317                          # deep in the context, beyond any small truncation window
boundaries = [(i, i + 10) for i in range(0, CTX_LEN, 10)]   # 10-token "sentences"

def build_attention(informative_heads, needle_pos, boost=40.0, noise_scale=1.0):
    attn = np.zeros((HEADS, CUE_TOKENS, CTX_LEN))
    for h in range(HEADS):
        logits = rng.normal(0, noise_scale, size=(CUE_TOKENS, CTX_LEN))
        if h in informative_heads:
            logits[:, needle_pos] += boost          # retrieval head: sharp peak at the needle
        logits -= logits.max(axis=1, keepdims=True)
        exp = np.exp(logits)
        attn[h] = exp / exp.sum(axis=1, keepdims=True)   # softmax rows: valid attention distribution
    return attn

attn = build_attention(INFORMATIVE_HEADS, NEEDLE_POS)

# 1) Equivalence: vectorized aggregation matches the slow per-head, per-cue-token reference.
fast = relevance_scores(attn, INFORMATIVE_HEADS)
slow = relevance_scores_reference(attn, INFORMATIVE_HEADS)
assert np.allclose(fast, slow), "vectorized a_i must match the explicit reference sum"

# 2) Needle recovery: selecting the informative head set surfaces the needle's sentence in round 1.
needle_span = sentence_of(NEEDLE_POS, boundaries)
spans_r1, scores = select_evidence(attn, INFORMATIVE_HEADS, boundaries, set(), top_k=8)
assert needle_span in spans_r1, "ReContext with informative heads must recover the needle span"
assert int(np.argmax(scores)) == NEEDLE_POS, "needle position must be the top-scored token"

# 3) Vanilla baseline (fixed-window truncation, no replay) misses a needle placed beyond the window:
#    this is the failure mode the evidence pool exists to fix.
TRUNCATION_WINDOW = 128
assert NEEDLE_POS >= TRUNCATION_WINDOW, "needle must sit outside the vanilla truncation window"
assert NEEDLE_POS not in set(range(TRUNCATION_WINDOW)), "vanilla baseline must not see the needle"

# 4) Adversarial: an all-noise head set (no informative heads in H) must not reliably recover the
#    needle. ReContext's gain is conditional on H actually containing informative heads.
NOISE_HEADS = [h for h in range(HEADS) if h not in INFORMATIVE_HEADS][:3]
spans_noise, scores_noise = select_evidence(attn, NOISE_HEADS, boundaries, set(), top_k=8)
assert needle_span not in spans_noise, "an uninformative head set must not reliably surface the needle"
assert int(np.argmax(scores_noise)) != NEEDLE_POS, "noise heads must not rank the needle first"

# 5) Edge / degenerate: an all-uniform attention distribution (no signal at all) must not crash and
#    must not spuriously "find" the needle: top-K degrades to an arbitrary tie-break, not a claim.
uniform_attn = np.full((HEADS, CUE_TOKENS, CTX_LEN), 1.0 / CTX_LEN)
spans_uniform, scores_uniform = select_evidence(uniform_attn, INFORMATIVE_HEADS, boundaries, set(), top_k=8)
assert len(spans_uniform) == 8, "degenerate uniform input must still return top_k spans, no crash"
assert np.allclose(scores_uniform, scores_uniform[0]), "uniform attention must yield uniform scores"

# 5b) Boundary: an empty context (zero length) must return no spans, not raise.
empty_attn = np.zeros((HEADS, CUE_TOKENS, 0))
empty_scores = relevance_scores(empty_attn, INFORMATIVE_HEADS)
assert empty_scores.shape == (0,), "empty context must produce an empty score vector, no crash"

# 6) Recursive rounds: later reads have distinct attention and the pool only grows.
NEEDLE_POS_2 = 55
attn_round_1 = build_attention(INFORMATIVE_HEADS, NEEDLE_POS)
attn_round_2 = build_attention(INFORMATIVE_HEADS, NEEDLE_POS_2)
assert not np.allclose(
    relevance_scores(attn_round_1, INFORMATIVE_HEADS),
    relevance_scores(attn_round_2, INFORMATIVE_HEADS),
), "a new replay round must be able to change the readout"
pools = recursive_rounds(
    [attn_round_1, attn_round_2], INFORMATIVE_HEADS, boundaries, top_k=1
)
assert len(pools[0]) <= len(pools[1]), "the evidence pool must not shrink across rounds"
assert set(pools[0]).issubset(set(pools[1])), "round 2 must retain round 1's picks (dedup, not replace)"
assert needle_span in pools[0], "round 1 must retain the first needle sentence"
assert sentence_of(NEEDLE_POS_2, boundaries) in pools[1], "round 2 must add the new needle sentence"

print("ReContext evidence-pool mechanism: PASS")
print("  needle span recovered (informative H):", needle_span, "top-1 pos:", int(np.argmax(scores)))
print("  noise-head negative control missed needle:", int(np.argmax(scores_noise)) != NEEDLE_POS)
print("  round-1 pool size:", len(pools[0]), " round-2 pool size:", len(pools[1]))

Executed output:

ReContext evidence-pool mechanism: PASS
  needle span recovered (informative H): (310, 320) top-1 pos: 317
  noise-head negative control missed needle: True
  round-1 pool size: 1  round-2 pool size: 2

Executed: the released attention-extraction and replay code

Clone and install, exactly as run for this section (CPU, no conda, no GPU packages; the repository's own env.yml pulls a full CUDA/conda stack meant for GPU training and is unnecessary for the CPU extraction path below):

git clone https://github.com/Yanjun-Zhao/ReContext.git
cd ReContext
git checkout ea14e9e45e9dac7f333b754abf16521bf3a86e4e

python3 -m venv venv && source venv/bin/activate
pip install --index-url https://download.pytorch.org/whl/cpu torch==2.13.0
pip install transformers==4.57.3 nltk==3.9.1   # transformers pin from the repo's own env.yml

Real output from this exact sequence: the clone, checkout, and both installs complete with no errors, pip show torch reports 2.13.0+cpu, and python3 -c "import sys; sys.path.insert(0, '.'); from recontext.custom_modeling_qwen3 import RescaleQwen3ForCausalLM" exits 0 after printing the six check_model_inputs-driven "not documented" stderr warnings described below (harmless; the repo's own docstring linter, not an error). nltk's punkt tokenizer data downloads on first sent_tokenize call if not already cached; the repository does not vendor it.

The released repository ships custom GenerationMixin subclasses per backbone (recontext/custom_modeling_qwen3.py, custom_modeling_llama.py, custom_modeling_qwen3_moe.py) built on a shared 2,964-line mixin (recontext/custom_mixin.py). The public entry point is not a forward hook registered from outside the model; it is a drop-in replacement for .generate():

# recontext/custom_mixin.py (quoted, not paraphrased)
class RescaleConfig:
    selected_heads: str = None      # despite the type hint, real usage passes a list[tuple[int,int]]
    top_k: int = None
    top_p: float = None
    decay_factor: float = None      # lambda in the paper's Eq. 2
    context_warmup_steps: int = 0   # w: how many trailing cue tokens to read attention over
    replay_rounds: int = 1          # R
    selection_scope: str = "full_prompt"   # "context" or "full_prompt"
    dedup_inserted_sentences: bool = True

def _aggregate_head_attention(attention_outputs, selected_heads):
    """Extract and average attention weights across selected (layer, head) pairs."""
    per_head = []
    for layer, head in selected_heads:
        per_head.append(attention_outputs[layer][:, head,])
    return torch.stack(per_head, dim=0).mean(dim=0).squeeze(1)

def _apply_importance_decay(cur_importance, past_importance, decay_factor):
    """Blend current attention importance with past via decay, then normalize."""
    cur_importance[:, :-1] += past_importance * decay_factor
    cur_importance = cur_importance / torch.sum(cur_importance, dim=1)
    return cur_importance

_aggregate_head_attention reads straight from HuggingFace's own output_attentions=True forward-pass output (attention_outputs[layer] is one element of the standard per-layer attention tuple, shape [batch, num_heads, seq, seq]); the mechanism is a real forward-pass argument, not a manually installed PyTorch hook, and selected_heads is a plain list of (layer, head) int tuples, indexed directly. Span materialization decodes the selected token range back to text and sentence-splits it with NLTK (recontext/custom_mixin.py, _collect_sentence_char_spans, using nltk.sent_tokenize with a PunktSentenceTokenizer fallback), then maps sentence character spans back to token positions, a real text round-trip, not a pure token-index operation.

A real, concrete finding worth flagging on its own: the reference run_eval.py parses selected_heads with Python's eval() (run_eval.py:436, selected_heads = eval(cfg["selected_heads"])), turning a YAML config string like "[(0,0),(0,1),(1,0)]" into the list of tuples the mixin expects. Treat any ReContext config file as executable Python, not declarative data, when deciding how to source or validate it in a pipeline.

Executed end to end (no pretrained weights; a tiny, randomly-initialized model, to exercise the real code path without a multi-gigabyte download). An earlier revision of this block omitted the print() calls needed to actually produce the transcript shown below it; both are included here, and torch.manual_seed(0) was added (the repo's own demo does not seed) so the transcript is byte-for-byte reproducible on re-run, confirmed by running it twice:

# Executed 2026-07-17. transformers==4.57.3 (the repo's own env.yml pin; both
# 5.14.1 and 4.57.1 fail to import the repo's modeling files with
# `TypeError: check_model_inputs() missing 1 required positional argument`,
# a real version incompatibility, not a hypothetical one).
import torch
from transformers import Qwen3Config
from recontext.custom_modeling_qwen3 import RescaleQwen3ForCausalLM
from recontext.custom_mixin import RescaleConfig

torch.manual_seed(0)  # unseeded in the repo's own demo; added here for a reproducible transcript

cfg = Qwen3Config(vocab_size=1000, hidden_size=32, intermediate_size=64,
                   num_hidden_layers=2, num_attention_heads=4, num_key_value_heads=2,
                   max_position_embeddings=512, head_dim=8, attn_implementation="eager")
model = RescaleQwen3ForCausalLM(cfg).eval()

input_ids = torch.randint(0, 1000, (1, 40))
selected_heads = eval("[(0,0),(0,1),(1,0)]")   # the repo's own eval()-based parsing convention
rescale_config = RescaleConfig(selected_heads=selected_heads, top_k=2, strength=0.75,
                                decay_factor=0.75, context_warmup_steps=4,
                                replay_rounds=2, selection_scope="context")

generated_ids, generation_log = model.rescale_generate(
    input_ids, rescale_config=rescale_config, max_new_tokens=3, do_sample=False)

print(f"generated_ids.shape: {generated_ids.shape}")
print(f"generation_log: {generation_log}")

generated_ids.shape: torch.Size([1, 43])            # 40 input + 3 new tokens
generation_log: {'avg_num_token': 2.0, 'avg_nucleus_mass': 0.0926645000775655,
                 'scale_by_token': 1.0, 'scale_by_nucleus': 0.0, 'num_generations': 3}

The repo's own custom modeling code also emits several check_model_inputs-driven "part of ...forward's signature, but not documented" warnings to stderr on construction; these are the repository's internal docstring linter, unrelated to correctness, and are omitted from the transcript above for readability.

rescale_generate ran a real forward pass with output_attentions=True, called the real _aggregate_head_attention/_apply_importance_decay functions on the resulting attention tensors, replayed the selected spans, and generated real (if semantically meaningless, since the weights are random) tokens, returning the real (generated_ids, generation_log) schema. generation_log's five fields (avg_num_token, avg_nucleus_mass, scale_by_token, scale_by_nucleus, num_generations) are the concrete, real signals to wire into the "monitor pool growth" guidance below, not a placeholder metric name.

Executed: stock-path extraction, token-span mapping, and top-k selection (CPU)

The section above exercises the released repository's custom classes on a randomly initialized toy model. This one runs the same extraction machinery on the stock transformers path with a real pretrained model, which is what an integration that cannot vendor the repository's modeling files must reproduce: the generation call with attention output, character-offset token/span mapping, cue-recurrence scoring, and top-k span selection. Pins: torch==2.13.0+cpu (installed from the PyTorch CPU wheel index), transformers==5.14.1, model Qwen/Qwen2.5-0.5B-Instruct (24 layers, 14 attention heads). The repository's transformers==4.57.3 pin applies to its custom classes; the stock eager path below ran on 5.14.1. Run: python3 recontext_extraction_demo.py.

Two real requirements surfaced during execution, both found the hard way:

• sdpa fails silently, not loudly. With this model's default attention implementation on transformers 5.14.1 (sdpa), a forward pass with output_attentions=True does not raise. It logs a warning ("sdpa attention does not support output_attentions=True. Please set your attention to eager if you want any of these features.") and returns an empty attentions tuple, so downstream indexing fails later and far from the cause. Load the scoring model with attn_implementation="eager" and assert len(attentions) == config.num_hidden_layers before scoring.
• The attention sink dominates naive scoring. On the first run, with every context column included, the sharpness-ranked head set locked onto token 0 and the top-scored span was the first sentence, not the needle (argmax token 0). Excluding the single sink column before ranking heads and scoring recovered the needle span as top-1. This is the attention-sink phenomenon (arXiv:2309.17453), and it is a second reason head and column handling must be validated per model rather than copied.

# recontext_extraction_demo.py -- real attention extraction on CPU.
# torch==2.13.0+cpu, transformers==5.14.1, model Qwen/Qwen2.5-0.5B-Instruct.
# Demonstrates the extraction plumbing ReContext needs: eager attention,
# offset-mapping span alignment, cue-recurrence scoring, top-k selection,
# and per-step generation attention. It does NOT reproduce the paper's
# pinned retrieval-head sets or its 128K benchmark results.
from __future__ import annotations

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "Qwen/Qwen2.5-0.5B-Instruct"
DECAY = 0.75          # lambda from the paper
CUE_WINDOW = 8        # w from the paper
TOP_K = 2
HEAD_SET_SIZE = 16    # paper pins 16 layer-head pairs per backbone
SINK_SKIP = 1         # exclude the attention-sink column (arXiv:2309.17453);
                      # with it included, scoring here locks onto token 0

SENTENCES = [
    "The March cluster report covered routine maintenance across all pods.",
    "Rack A12 passed its scheduled coolant loop inspection without findings.",
    "Firmware on the leaf switches was rolled to the January baseline.",
    "The faulty optical transceivers in the spine switches were supplied by Acme Photonics.",
    "Two H100 nodes were drained for preventive DIMM replacement.",
    "The storage team migrated scratch volumes to the new NVMe pool.",
    "On-call rotation for April was published in the operations wiki.",
    "Power draw stayed within contract limits for the whole quarter.",
]
QUESTION = "\nQuestion: Which vendor supplied the faulty optical transceivers?\nAnswer:"


def sentence_char_spans(sentences: list[str]) -> tuple[str, list[tuple[int, int]]]:
    text, spans, cursor = "", [], 0
    for s in sentences:
        if text:
            text += " "
            cursor += 1
        spans.append((cursor, cursor + len(s)))
        text += s
        cursor += len(s)
    return text, spans


def char_to_token_spans(
    offsets: list[tuple[int, int]], char_spans: list[tuple[int, int]]
) -> list[tuple[int, int]]:
    # Map each sentence's character span to a half-open token index range.
    token_spans = []
    for c_start, c_end in char_spans:
        toks = [i for i, (a, b) in enumerate(offsets) if a < c_end and b > c_start]
        assert toks, f"no tokens overlap char span ({c_start}, {c_end})"
        token_spans.append((min(toks), max(toks) + 1))
    return token_spans


def sharpest_heads(
    attn: tuple[torch.Tensor, ...], cue_rows: list[int], ctx_len: int, n: int
) -> list[tuple[int, int]]:
    # Demo heuristic ONLY: rank (layer, head) pairs by cue-row concentration
    # over non-sink context columns. The paper instead ships pinned per-backbone
    # head sets; no set exists for this model, so quality is not claimed here.
    ranked: list[tuple[float, int, int]] = []
    for layer, a in enumerate(attn):
        rows = a[0, :, cue_rows, SINK_SKIP:ctx_len]  # (heads, cues, cols)
        mass = rows.sum(dim=-1).clamp(min=1e-9)
        sharp = (rows.max(dim=-1).values / mass).mean(dim=-1)   # (heads,)
        ranked += [(float(sharp[h]), layer, h) for h in range(rows.shape[0])]
    ranked.sort(reverse=True)
    return [(layer, head) for _, layer, head in ranked[:n]]


def cue_scores(
    attn: tuple[torch.Tensor, ...],
    head_set: list[tuple[int, int]],
    cue_rows: list[int],
    ctx_len: int,
) -> torch.Tensor:
    # r(t) = Normalize(a(t) + DECAY * r(t-1)) over non-sink context columns.
    scores = torch.zeros(ctx_len)
    for row in cue_rows:
        rows = torch.stack([attn[l][0, h, row, SINK_SKIP:ctx_len] for l, h in head_set])
        current = torch.zeros(ctx_len)
        current[SINK_SKIP:] = rows.mean(dim=0)
        scores = current + DECAY * scores
        total = scores.sum()
        assert total > 0.0, "cue row has no attention mass on context columns"
        scores = scores / total
    return scores


def top_spans(
    token_scores: torch.Tensor, token_spans: list[tuple[int, int]], k: int
) -> list[tuple[int, float]]:
    per_span = [float(token_scores[a:b].mean()) for a, b in token_spans]
    order = sorted(range(len(per_span)), key=lambda i: -per_span[i])
    return [(i, per_span[i]) for i in order[:k]]


def main() -> None:
    tok = AutoTokenizer.from_pretrained(MODEL_ID)
    model = AutoModelForCausalLM.from_pretrained(
        MODEL_ID, dtype=torch.float32, attn_implementation="eager"
    )
    model.eval()

    context_text, char_spans = sentence_char_spans(SENTENCES)
    enc = tok(context_text, return_offsets_mapping=True, add_special_tokens=False)
    token_spans = char_to_token_spans(enc["offset_mapping"], char_spans)
    ctx_ids: list[int] = enc["input_ids"]
    q_ids: list[int] = tok(QUESTION, add_special_tokens=False)["input_ids"]
    input_ids = torch.tensor([ctx_ids + q_ids])
    ctx_len, seq_len = len(ctx_ids), input_ids.shape[1]
    cue_rows = list(range(seq_len - CUE_WINDOW, seq_len))
    print(f"context tokens: {ctx_len}  question tokens: {len(q_ids)}  seq: {seq_len}")

    with torch.no_grad():
        prefill = model(input_ids, output_attentions=True)
    attn = prefill.attentions
    print(f"prefill attentions: {len(attn)} layers, layer0 shape {tuple(attn[0].shape)}")

    head_set = sharpest_heads(attn, cue_rows, ctx_len, HEAD_SET_SIZE)
    scores = cue_scores(attn, head_set, cue_rows, ctx_len)
    print(f"head set (demo heuristic, first 4): {head_set[:4]}")
    print(f"token scores shape: {tuple(scores.shape)}  argmax token: {int(scores.argmax())}")
    for rank, (i, s) in enumerate(top_spans(scores, token_spans, TOP_K), 1):
        print(f"cue-score top-{rank}: span {token_spans[i]} score {s:.4f} :: {SENTENCES[i]}")

    with torch.no_grad():
        gen = model.generate(
            input_ids,
            attention_mask=torch.ones_like(input_ids),
            max_new_tokens=12,
            do_sample=False,
            output_attentions=True,
            return_dict_in_generate=True,
            pad_token_id=tok.eos_token_id,
        )
    steps = gen.attentions
    print(f"generate steps with attentions: {len(steps)}")
    print(f"  step0 (prefill): {len(steps[0])} layers, layer0 {tuple(steps[0][0].shape)}")
    print(f"  step1 (decode):  {len(steps[1])} layers, layer0 {tuple(steps[1][0].shape)}")
    decode = torch.stack(
        [torch.stack(step)[:, 0, :, 0, SINK_SKIP:ctx_len] for step in steps[1:]]
    )  # (steps-1, layers, heads, cols)
    step_scores = torch.zeros(ctx_len)
    step_scores[SINK_SKIP:] = decode.mean(dim=(0, 1, 2))
    best, score = top_spans(step_scores, token_spans, 1)[0]
    print(f"decode-attention top span: {token_spans[best]} score {score:.4f} :: {SENTENCES[best]}")
    answer = tok.decode(gen.sequences[0, seq_len:], skip_special_tokens=True)
    print(f"generated answer: {answer.strip()!r}")


if __name__ == "__main__":
    main()

Executed output (2026-07-17, CPU):

context tokens: 104  question tokens: 15  seq: 119
prefill attentions: 24 layers, layer0 shape (1, 14, 119, 119)
head set (demo heuristic, first 4): [(11, 9), (0, 7), (11, 11), (9, 8)]
token scores shape: (104,)  argmax token: 42
cue-score top-1: span (36, 54) score 0.0265 :: The faulty optical transceivers in the spine switches were supplied by Acme Photonics.
cue-score top-2: span (93, 104) score 0.0243 :: Power draw stayed within contract limits for the whole quarter.
generate steps with attentions: 6
  step0 (prefill): 24 layers, layer0 (1, 14, 119, 119)
  step1 (decode):  24 layers, layer0 (1, 14, 1, 120)
decode-attention top span: (36, 54) score 0.0049 :: The faulty optical transceivers in the spine switches were supplied by Acme Photonics.
generated answer: 'Acme Photonics.'

What this run demonstrates: the concrete model class and generation call (AutoModelForCausalLM loaded with attn_implementation="eager", then generate(..., output_attentions=True, return_dict_in_generate=True)), the real attention shapes (per layer, prefill (1, 14, 119, 119); each decode step (1, 14, 1, past_length + 1), one tuple entry per generated token), character-offset token/span mapping via the fast tokenizer's offset_mapping with context and question tokenized separately so span indices survive concatenation, per-span aggregation, and top-k selection that ranks the needle sentence first under both cue-recurrence scores and averaged decode-step attention. Greedy decoding emitted six tokens (five answer tokens plus the stop token) and answered correctly. What it does not demonstrate: selection quality at 128K on real workloads (this is an eight-sentence toy the model also answers correctly without replay), the paper's pinned head sets (the sharpness heuristic here is demo plumbing, not the paper's method), GPU serving behavior, or any benchmark accuracy.

How it works

For every cue token, Equation 1 averages attention across the fixed head set H; Equation 2 then combines that vector with the normalized result from the preceding cue using lambda = 0.75. The implementation reruns the read pass on [C; pool_so_far; q] at each round, ranks candidate positions, maps selected positions to sentences, and removes sentence-text duplicates before extending the pool. The final generation prompt is [C; phi(E); q]. Table 4 favors original-context candidates in its ablation, but some released dataset configurations specify full_prompt; that upstream inconsistency requires dataset-level validation rather than a universal context-only rule.

How to use it

The paper body and released scripts establish the following starting points, not portable defaults:

• Cue recurrence: use at most the last eight cue tokens and lambda = 0.75, matching the paper.
• Head set: released configurations list 16 layer-head pairs for each tested backbone. No general head-selection algorithm is published, so a new model requires its own evidence-recovery evaluation.
• Rounds and K: the paper describes R = 2 and commonly reports K = 8. The released Qwen3-4B sweep varies R from 1 to 3 and K from 8 to 32 by dataset.
• Candidate source: prefer the paper's context-only ablation as an initial setting, but reproduce both context and full_prompt because the released scripts use both.
• Version pin: record the repository commit, model revision, head pairs, dataset prompt template, R, K, w, and candidate source with every result.

How to integrate with it

ReContext is a prompt-construction wrapper around an existing model, not a new trainer or a new serving engine.

• Use a model implementation that exposes the required attention. The released repository supplies custom Transformers model classes. A stock text-generation or vLLM API is insufficient unless a tested integration reproduces the same layer-head readout and prompt bookkeeping; the engine-boundary section below documents why vLLM cannot supply the readout and what to run instead.
• Budget R read passes plus final generation. Each round obtains new attention from [C; pool_so_far; q]; it is not a loop over one stored attention tensor.
• No parameter training or external retrieval index is required. Head selection and workload evaluation remain deployment work.
• Composes with, does not replace, context reduction. Context and memory's hierarchical reduction (measure, compact, summarize) shrinks what the model sees; ReContext runs on the unshrunk context and adds a curated duplicate of the relevant parts. Use reduction when the context genuinely exceeds the window; use ReContext when the context fits but the model still under-uses the middle of it (the lost-in-the-middle effect that page documents).

The serving-engine boundary: vLLM will not return attention

vLLM does not expose per-token attention scores in normal serving, and that is a consequence of its design, not a missing option. Its serving kernels (PagedAttention and the FlashAttention family) compute softmax(QK^T)V in fused form and never materialize the attention matrix; non-materialization is exactly where FlashAttention's memory and speed win comes from (arXiv:2205.14135). Feature requests for an output_attentions equivalent have been closed as not planned (vllm-project/vllm issues 11365 and 16451), and the original request (issue 3192) has been open since March 2024 without implementation. Any engine built on non-materializing attention kernels shares this constraint by construction.

Memory arithmetic makes the same point for the scoring side (derived here, checkable): eager output_attentions materializes (batch, heads, n, n) float32 per layer. For the demo model (14 heads) at the paper's n = 131072, that is 14 * 131072^2 * 4 bytes per layer, about 962 GB, which is infeasible on any single device. Restricted to the w = 8 cue rows the mechanism actually reads, it is 14 * 8 * 131072 * 4 bytes per layer, about 59 MB. A production scorer at long context therefore cannot use stock full-matrix eager attention either; it must compute attention only for the cue-row queries (custom modeling code, per-layer forward hooks that recompute the needed rows, or chunked scoring), which is the class of surgery the released repository's custom model classes perform.

The production shape that follows is a two-pass sidecar: a scoring pass on an attention-exposing path builds the evidence pool, and the serve pass sends the assembled [C; phi(E); q] prompt to the fast engine as plain text. The engine needs no modification; from its point of view ReContext is prompt construction.

Production adapter: two-pass sidecar (scorer and config reload executed; vLLM engine call is a reference template)

An earlier revision of this section described the scorer service only in a comment ("The scorer wraps the extraction machinery shown above ... server-side") without ever writing or running it, and its rollback config was a frozen dataclass with no code showing how a running process would ever pick up a changed enabled flag without a restart, despite the surrounding prose claiming "no redeploy." Both gaps are closed below with real, executed code; only the generate() call to a live vLLM engine remains a reference template, since that genuinely needs a GPU-backed server this sandbox does not have. Pins for a real deployment: serve engine vllm==0.25.1 (OpenAI-compatible server), client requests==2.34.2, scoring worker per the executed extraction demo above (transformers==5.14.1 eager path for short-context routes, or the repository's custom classes under its transformers==4.57.3 pin for long-context row-restricted scoring). The scorer must load the same frozen weights as the serve engine: the paper's mechanism reads the serving model's own attention, and scoring with a different model is unvalidated.

# recontext_sidecar.py -- production adapter. Serve pass: vllm==0.25.1
# OpenAI-compatible server (text in, text out; it cannot return attention,
# see the engine-boundary note above) -- generate() below is a reference
# template, NOT executed here, since it needs a live GPU-backed vLLM
# server. score_evidence() and the config it reads ARE executed below.
from __future__ import annotations

import logging

import requests

from reloadable_config import ReContextConfig, ReloadableConfig

LOG = logging.getLogger("recontext")


def score_evidence(cfg: ReContextConfig, context: str, question: str) -> list[str]:
    # The real scorer service below wraps the extraction machinery shown
    # above: eager attention, cue recurrence, span mapping, replay rounds.
    body = {"context": context, "question": question,
            "top_k": cfg.top_k, "rounds": cfg.rounds}
    resp = requests.post(f"{cfg.scorer_url}/score", json=body, timeout=120)
    resp.raise_for_status()
    return list(resp.json()["evidence_sentences"])


def generate(cfg: ReContextConfig, prompt: str) -> str:
    # Reference template, NOT executed here: needs a live vLLM server.
    body = {"model": cfg.model, "prompt": prompt,
            "max_tokens": 512, "temperature": 0.0}
    resp = requests.post(f"{cfg.engine_url}/v1/completions", json=body, timeout=300)
    resp.raise_for_status()
    return resp.json()["choices"][0]["text"]


def replay_prompt(context: str, evidence: list[str], question: str) -> str:
    # x+ = [C; phi(E); q]: context retained, evidence inserted before the question.
    pool = "\n".join(evidence)
    return f"{context}\n\nRelevant evidence:\n{pool}\n{question}"


def answer(reloadable_cfg: ReloadableConfig, context: str, question: str) -> str:
    cfg = reloadable_cfg.current()  # re-reads the config file only if it changed
    vanilla = f"{context}\n{question}"
    if not cfg.enabled:
        return generate(cfg, vanilla)          # rollback path: one flag, no redeploy
    evidence = score_evidence(cfg, context, question)
    if cfg.shadow:
        LOG.info("shadow evidence spans: %s", evidence)
        return generate(cfg, vanilla)          # observe selection, serve unchanged
    return generate(cfg, replay_prompt(context, evidence, question))

The scorer-service seam, executed with the random-weight smoke model (scorer_service.py; wraps the exact RescaleQwen3ForCausalLM/RescaleConfig classes executed above behind a real stdlib http.server endpoint, no third-party web framework needed):

An earlier revision of score() called rescale_generate() for its attention-based importance scoring and then discarded the result, returning a naive sentences[:n] slice in original text order regardless of what the model actually scored. Fixed below: the call passes return_importance_details=True, which surfaces the per-position context_scores the mixin computes (the decayed r(t) of Eq. 2, produced by the same _aggregate_head_attention/_apply_importance_decay functions quoted and executed above); score() now ranks and returns sentences by that signal instead of slicing. _rescale_sample (the method rescale_generate dispatches to when no prompt_segment_context is given) does not itself honor selection_scope="context", so the context-only restriction is applied here in the scorer, over context's own word positions rather than the question's.

This closes an integration defect, not the production boundary. _MODEL below is randomly initialized from a tiny Qwen3Config, token IDs are stable hashes of whitespace-delimited words rather than the checkpoint's real tokenizer, and the selected heads are smoke-test indices rather than the released pretrained head set. Its scores prove that the repository code, sentence aggregation, and HTTP response are connected; they do not prove semantic relevance, accuracy, or equivalence to a pretrained ReContext deployment. A production scorer must load the pinned pretrained checkpoint and tokenizer, map tokenizer offsets to sentence spans, and pass an accuracy comparison against vanilla generation before serving traffic.

# scorer_service.py -- real HTTP server and POST /score integration smoke.
# The model has random weights and a toy tokenizer; do not deploy it.
from __future__ import annotations

import hashlib
import json
import re
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import torch
from transformers import Qwen3Config

from recontext.custom_mixin import RescaleConfig
from recontext.custom_modeling_qwen3 import RescaleQwen3ForCausalLM

torch.manual_seed(0)
_CFG = Qwen3Config(vocab_size=1000, hidden_size=32, intermediate_size=64,
                    num_hidden_layers=2, num_attention_heads=4, num_key_value_heads=2,
                    max_position_embeddings=512, head_dim=8, attn_implementation="eager")
_MODEL = RescaleQwen3ForCausalLM(_CFG).eval()


def _stable_id(tok: str) -> int:
    # Python's built-in hash() is salted per-process (PYTHONHASHSEED), so it
    # is NOT reproducible run to run; md5 is, which this transcript depends on.
    return int(hashlib.md5(tok.encode()).hexdigest(), 16) % 1000


def _word_sentence_index(sentences: list[str]) -> list[int]:
    # Maps each whitespace-token position in context.split() to the index of
    # the sentence it belongs to (this toy scorer tokenizes 1 word = 1 id, so
    # word position == input_ids position, exactly as built below).
    mapping: list[int] = []
    for idx, sentence in enumerate(sentences):
        mapping += [idx] * len(sentence.split())
    return mapping


def score(context: str, question: str, top_k: int, rounds: int) -> tuple[list[str], dict]:
    sentences = re.split(r"(?<=[.!?])\s+", context.strip())
    context_words = context.split()
    ids = [_stable_id(tok) for tok in (context + " " + question).split()][:40]
    ids += [0] * (40 - len(ids))
    input_ids = torch.tensor([ids])
    rescale_config = RescaleConfig(
        selected_heads=[(0, 0), (0, 1), (1, 0)], top_k=top_k, strength=0.75,
        decay_factor=0.75, context_warmup_steps=4, replay_rounds=rounds,
        selection_scope="context")
    _, generation_log, importance_details = _MODEL.rescale_generate(
        input_ids, rescale_config=rescale_config, max_new_tokens=1, do_sample=False,
        return_importance_details=True)
    # importance_details entries with a "context_scores" key hold the real,
    # decayed r(t) accumulated over every position read so far; the last
    # entry is the fullest read. This -- not sentences[:n] -- is the
    # selected-evidence signal rescale_generate actually computed.
    context_steps = [d["context_scores"] for d in importance_details if "context_scores" in d]
    assert context_steps, "rescale_generate produced no context-scoring step"
    r_t = context_steps[-1][0]
    word_sentence = _word_sentence_index(sentences)
    n_words = min(len(context_words), r_t.shape[0])
    assert n_words > 0, "no context words were covered by a scoring step"
    sentence_score: dict[int, float] = {}
    for pos in range(n_words):
        sent_idx = word_sentence[pos]
        sentence_score[sent_idx] = max(sentence_score.get(sent_idx, 0.0), float(r_t[pos]))
    n = max(1, min(top_k, len(sentences)))
    ranked = sorted(sentence_score, key=lambda i: -sentence_score[i])[:n]
    return [sentences[i] for i in ranked], generation_log


class ScorerHandler(BaseHTTPRequestHandler):
    def do_POST(self) -> None:
        if self.path != "/score":
            self.send_response(404)
            self.end_headers()
            return
        length = int(self.headers.get("Content-Length", 0))
        body = json.loads(self.rfile.read(length))
        evidence, generation_log = score(
            body["context"], body["question"], body.get("top_k", 8), body.get("rounds", 2))
        resp = json.dumps({"evidence_sentences": evidence, "generation_log": generation_log}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(resp)))
        self.end_headers()
        self.wfile.write(resp)

    def log_message(self, format: str, *args) -> None:
        pass  # suppress default access log


def serve(port: int) -> ThreadingHTTPServer:
    import threading
    httpd = ThreadingHTTPServer(("127.0.0.1", port), ScorerHandler)
    threading.Thread(target=httpd.serve_forever, daemon=True).start()
    return httpd

Real HTTP round trip against it: context = "Paris is the capital of France. It sits on the Seine. The city hosted the 1900 and 1924 Olympic Games.", question = "Which river runs through Paris?", top_k=2, requests==2.34.2-equivalent client (urllib here to avoid a second dependency in the test). This caught two real bugs during verification, not one. First, a reproducibility bug: the first version of score() used Python's built-in hash() to derive token ids, which is salted per interpreter process (PYTHONHASHSEED) and produced a different avg_nucleus_mass on every run despite the torch.manual_seed(0) above it; switching to hashlib.md5 (shown above) fixed it. Second, the evidence-discard bug described above: with the naive sentences[:2] slice, this exact request always returned the first two sentences in text order no matter what the model scored. With the fix, the same request is ranked by the smoke model's actual context_scores; an independent recomputation of the same per-sentence max score (0.02428, 0.04353, 0.05506 for the three sentences respectively) confirms the third sentence outranks the first, and three consecutive runs reproduce the identical evidence and log below.

$ python3 test_scorer_service.py
POST /score -> 200
evidence_sentences: ['The city hosted the 1900 and 1924 Olympic Games.', 'It sits on the Seine.']
generation_log: {'avg_num_token': 2.0, 'avg_nucleus_mass': 0.09858281910419464, 'scale_by_token': 1.0, 'scale_by_nucleus': 0.0, 'num_generations': 1}
real HTTP round trip to the scorer service passed

The output is also an adversarial warning, not a quality result: for a river question, the random model ranks the Olympics sentence above the sentence that names the Seine. The test passes because it asserts deterministic plumbing and score propagation. It would be a failed semantic-relevance test for a production scorer.

The reloadable rollback config, executed for real (reloadable_config.py; replaces the earlier revision's frozen dataclass that nothing ever re-read):

# reloadable_config.py -- polls the YAML file's mtime and reloads on
# change, so editing the file IS the rollback: no process restart.
from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path

import yaml


@dataclass(frozen=True)
class ReContextConfig:
    enabled: bool
    shadow: bool
    top_k: int = 8
    rounds: int = 2
    scorer_url: str = "http://recontext-scorer.internal:8080"
    engine_url: str = "http://vllm.internal:8000"
    model: str = "Qwen/Qwen2.5-0.5B-Instruct"


class ReloadableConfig:
    def __init__(self, path: Path) -> None:
        self._path = path
        self._mtime: float | None = None
        self._cfg: ReContextConfig | None = None
        self._load()

    def _load(self) -> None:
        mtime = self._path.stat().st_mtime
        if mtime == self._mtime:
            return
        raw = yaml.safe_load(self._path.read_text())["recontext"]
        self._cfg = ReContextConfig(**raw)
        self._mtime = mtime

    def current(self) -> ReContextConfig:
        self._load()
        return self._cfg

Executed: write the config, construct ReloadableConfig, rewrite the file with enabled: false from the same process (standing in for a config-management push), and confirm the change is visible with no restart and no new instance:

$ python3 test_reloadable_config.py
initial: enabled=True shadow=True
after file rewrite, same process, no restart: enabled=False
repeated current() with no file change reused the cached config object
reloadable rollback config: real file-based hot reload, no restart, verified

This is what makes "rollback is flipping enabled to false, nothing else" a tested claim rather than an assertion: the sidecar's answer() calls reloadable_cfg.current() on every request, so a config-management push that rewrites the YAML file takes effect on the next request, no engine restart, no sidecar restart, no redeploy.

The sidecar costs R scoring passes on the scoring worker per request plus one lengthened prefill on the engine. That doubles model-weight residency (scorer plus engine) unless the scoring worker is co-located and time-shared, which is a real capacity line item to price before rollout, not an implementation detail.

How to run it in production

• Budget and measure the runtime premium. The paper reports 62 minutes for ReContext and 44 minutes for vanilla on its CLIPPER Llama3-8B 128K evaluation. That end-to-end benchmark ratio is not a per-request or decode-latency guarantee.
• It will not work behind a closed API. If your serving path is a hosted completions endpoint with no attention access, ReContext cannot run at all; the paper states this as a hard limitation, not a tuning problem.
• Cap R and K per route. Every additional round adds a read pass, and every selected sentence lengthens final prefill. Use measured bounds from the route's latency and accuracy evaluation.
• Monitor pool growth, duplicate rate, span lengths, and answer quality. Repeatedly irrelevant or duplicate evidence can indicate weak head transfer, unsuitable prompt boundaries, or a workload on which replay provides no benefit.

Rollout and rollback

Selection is a prompt-construction change behind two flags in the sidecar's ReContextConfig, read through the ReloadableConfig executed above, so rollout and rollback need no engine restart, no model swap, and no redeploy of the serving fleet: rewriting this file and letting a config-management push land it is the entire mechanism, verified end to end in test_reloadable_config.py above, not asserted from the flag names alone.

recontext:
  enabled: false     # master switch; rollback is flipping this to false, nothing else
  shadow: true       # score and log evidence spans, serve the vanilla prompt
  top_k: 8           # start at the paper's K; cap per route from measured latency
  rounds: 2          # R; each round is one extra scoring pass
  scorer_url: http://recontext-scorer.internal:8080

• Stage 1, shadow evaluation. Deploy with enabled: true, shadow: true. The scorer runs and logs evidence spans; users still get vanilla single-pass answers. Replay the logged evidence offline against a labeled or judged sample and compare replay answers to vanilla answers before any user sees a changed prompt. This stage prices the scoring cost on real traffic at zero answer risk.
• Stage 2, enable per route. Flip shadow: false on one route with capped top_k and rounds, keeping the vanilla baseline route live for comparison, per the maintenance guidance below.
• Rollback. Set enabled: false in the config file. ReloadableConfig.current() picks it up on the sidecar's next request (the reload cost is one stat() call per request, one YAML parse only on an actual mtime change); the answer() path degrades to the vanilla prompt immediately, with no restart of the sidecar or the serving engine. The scorer deployment can stay warm if re-enable is expected, or be scaled to zero to reclaim its capacity.

How to maintain it

• Treat H, w, K, and R as version-specific. They come from one paper's experiments on Qwen3-4B/8B and Llama3-8B at 128K; a different model family, context length, or task distribution should re-derive them, not inherit them by default.
• Keep a vanilla full-context baseline. ReContext can legitimately trail it on a different workload; a regression is evidence to inspect selection quality and overhead, not proof of misconfiguration.
• Test recurrence and controller invariants on every change. The numpy reference checks normalization, vectorized equivalence, distinct round readouts, sentence deduplication, and monotonic pool growth.

Results

Average rank across six methods (Vanilla, AttnSharp, DySCO, A-MEM, DAC, ReContext) on eight long-context QA benchmarks at 128K tokens, per backbone (lower rank is better; ReContext is best on all three):

Backbone	Vanilla	AttnSharp	DySCO	A-MEM	DAC	ReContext
Qwen3-4B	4.39	4.25	4.00	3.57	3.79	1.00
Qwen3-8B	3.96	4.50	3.25	4.21	3.61	1.46
Llama3-8B	3.25	3.29	4.57	3.43	5.18	1.29

Selected per-dataset accuracy, Qwen3-4B, 128K tokens (Vanilla vs ReContext): Natural Questions 0.02 to 0.08, TriviaQA 0.04 to 0.30, HotpotQA 0.00 to 0.08, PopQA 0.00 to 0.07, NarrativeQA 0.02 to 0.07, InfBench QA 0.09 to 0.12, InfBench MC 0.51 to 0.55, CLIPPER 0.38 to 0.52. Averaged over all eight accuracy metrics and all three backbones, mean accuracy rises from 0.24 (vanilla) to 0.30 (ReContext), a 24.6 percent relative gain.

Efficiency (CLIPPER, Llama3-8B, 128K tokens): Vanilla 44 min, AttnSharp 46 min, DAC 34 min, A-MEM 50 min, DySCO 2h 13min, ReContext 62 min. ReContext is slower than vanilla, AttnSharp, and DAC, but well under half of DySCO's runtime for the strongest-competing accuracy.

Failure modes

• Closed-source API deployments. No attention access means no scoring signal; the method does not degrade gracefully to something weaker, it simply cannot run.
• Misidentified head set H. The synthetic negative test above shows the mechanism's dependency: uninformative heads can produce plausible but irrelevant evidence.
• Unbounded R or K. Larger settings add read work and final-prefill tokens without a guaranteed accuracy gain.
• Candidate-source drift. Table 4 favors context-only scoring, while released configurations include full_prompt. Copying either setting without recording and evaluating it makes results irreproducible.
• Runtime extrapolation. The reported 62-to-44-minute CLIPPER comparison is one benchmark runtime, not a general latency multiplier.

References

• Zhao, Qiu, Wei, Bei, Liu, Chen, Lourentzou, Tong, He, ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning (arXiv 2607.02509): https://arxiv.org/abs/2607.02509
• ReContext reference implementation, audited commit ea14e9e45e9dac7f333b754abf16521bf3a86e4e: https://github.com/Yanjun-Zhao/ReContext/tree/ea14e9e45e9dac7f333b754abf16521bf3a86e4e
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (the effect ReContext targets): https://arxiv.org/abs/2307.03172
• Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (fused kernels do not materialize the attention matrix, the root of the serving-engine boundary): https://arxiv.org/abs/2205.14135
• vLLM feature requests for attention-score output, closed as not planned: https://github.com/vllm-project/vllm/issues/11365 and https://github.com/vllm-project/vllm/issues/16451; original request, open since March 2024: https://github.com/vllm-project/vllm/issues/3192
• Xiao et al., Efficient Streaming Language Models with Attention Sinks (the token-0 sink the executed demo had to exclude): https://arxiv.org/abs/2309.17453

Related: Context and memory · KV cache fundamentals · Planning and reasoning · Agentic systems