Markdown

Parametric long-term memory modules¶

Scope: separating a language model's stored knowledge from its reasoning by training a second decoder to imitate a nearest-neighbour retriever, then mixing the two output distributions at every decoded token. This page covers the offline retrieval pipeline that produces the training signal, the sparse storage format that makes it affordable, the serving contract, and an audit of the published scaling results. The algorithmic neighbours are knowledge distillation and RAG vs CAG; the serving-side cost lands in inference serving.

The numbers below are from Memory Decoder at Scale (arXiv 2607.27919, 30 July 2026) and were re-derived from the paper's own tables. The Python blocks were executed with numpy. Faiss and Megatron-LM snippets are unexecuted reference templates.

What it is¶

A decoder-only model keeps everything it knows in one parameter set, so the only way to add knowledge is to make the whole model bigger and retrain it. A parametric memory splits that. A second decoder M_psi, sharing the backbone's tokenizer and vocabulary, is pretrained on a different target: instead of predicting the observed next token, it predicts the distribution a kNN retriever would have produced for that context.

Building the target is a separate offline job. A frozen base model encodes every context in the corpus into its final-layer hidden state, giving a key-value store of (hidden state, observed next token). For each training context, the K nearest keys are retrieved, weighted by softmax(-d/tau) over their distances, and the weights of neighbours sharing a target token are summed. The result is a sparse distribution over the vocabulary with support at most K.

The memory is trained against that distribution with L = beta * KL(p_ret || p_psi) + (1 - beta) * CE(x_t). The KL term teaches it to imitate retrieval; the cross-entropy term keeps it anchored to the corpus. At inference the retriever is gone. Backbone and memory read the same context in parallel and their distributions are mixed:

p_final(y) = (1 - alpha) * p_theta(y) + alpha * p_psi(y)

The backbone is never updated. The memory is a swappable component: one general memory, or a domain memory per corpus, attached to a backbone that stays frozen.

Why use it¶

Knowledge scales without touching the backbone. Pairing a frozen Pythia-410M with a 6.9B memory raises its 17-task average from 29.86 to 37.34, above the frozen Pythia-12B at 37.24, at 7.31B total parameters against 12B.
No retrieval at serving time. Unlike kNN-LM or RAG, nothing is searched, fetched, or prepended during decode. The retrieval cost is paid once, offline, when the supervision is built.
Domain adaptation without catastrophic forgetting. The backbone's weights are untouched, so a domain memory cannot degrade general capability the way continued pretraining can. Reported gains over frozen Qwen3 backbones across biology, law, and finance average 9.09 to 10.00 points at every scale from 0.6B to 14B.
One memory, many backbones. A memory trained against one backbone family transfers to others by replacing the embedding layer and language-model head, then continuing training at 20% of the standard budget.

When to use it (and when not)¶

Use it when the gap is knowledge, not reasoning, but size the expectation honestly. The largest single gains are knowledge-intensive (TriviaQA rises 8.81 points at the 2.8B pairing, 2WikiMultiHopQA 5.68 at 1.4B), yet averaged over the whole table knowledge tasks gain only about 1.7x what the rest do. Three quarters of the non-knowledge results move under 2 points, with WinoGrande and GPQA-main the exceptions at the smallest scale (3.31 and 3.37).
Use it when you must keep a specific backbone frozen: a certified checkpoint, a shared base behind several products, or a model you do not control.
Prefer RAG when the knowledge changes faster than you can rebuild a datastore and retrain a memory. A memory bakes the corpus in at training time and has no update path short of retraining.
Prefer LoRA or continued pretraining when serving capacity, not parameter count, is the binding constraint. Two models must be resident and stepped together per token.
Do not use it where verbatim training content is sensitive. Section 6.6 of the paper measures the opposite of privacy: on the same corpus and budget, the memory model reproduces training suffixes more reliably than a continued-pretraining baseline (see Failure modes).
Do not expect it to help factual calibration. TruthfulQA is the one benchmark that regresses, and it regresses at every scale tested.

Architecture¶

flowchart TB
  subgraph OFF["Offline: build the supervision (once per corpus)"]
    C["Corpus contexts"] --> H["Frozen base model final-layer hidden states"]
    H --> OPQ["OPQ256: 4096-d to 256-d"]
    OPQ --> IVF["IVF partition, IndexHNSW quantizer"]
    IVF --> SH["IndexIVFPQ shards over contiguous centroid ranges"]
    Q["Training queries"] --> R1["Route: probe centroids, group by shard"]
    R1 --> SH
    SH --> MRG["Writer workers merge candidates by query id"]
    MRG --> SP["Sparse p_ret rows: offsets, token ids, FP16 probs"]
  end
  subgraph TRAIN["Memory pretraining"]
    SP --> KL["beta * KL(p_ret || p_psi) + (1 - beta) * CE"]
    KL --> MEM["Memory decoder M_psi"]
  end
  subgraph SERVE["Serving: no retrieval"]
    CTX["Prompt"] --> BASE["Frozen backbone p_theta"]
    CTX --> MEM
    BASE --> MIX["p_final = (1 - alpha) p_theta + alpha p_psi"]
    MEM --> MIX
    MIX --> TOK["Next token"]
  end

The offline half is the expensive half, and it is a cluster job, not a preprocessing script. A corpus of N tokens yields N keys and N queries, so a naive build is quadratic. The deduplicated Pile used here is 207B tokens of 4096-dimensional keys.

How to use it¶

Three properties decide whether the pipeline is correct, and all three are checkable without a GPU. The block below models the sharded search, the distribution construction, and the storage format, and it breaks each one on purpose.

# knn_supervision.py: executed model of the sharded retrieval and sparse storage
# that make kNN-distribution supervision affordable at pretraining scale.
import numpy as np

VOCAB, DIM, K, TAU, EPS = 50_304, 32, 16, 1.0, 1e-4
rng = np.random.default_rng(0)
keys = rng.standard_normal((20_000, DIM)).astype(np.float32)
values = rng.integers(0, VOCAB, keys.shape[0])


def knn_distribution(query, keys, values, k=K, tau=TAU):
    """Equations 4 and 5: softmax(-d/tau) over the K nearest, summed per target."""
    d = np.linalg.norm(keys - query, axis=1)
    nn = np.argsort(d, kind="stable")[:k]
    w = np.exp(-(d[nn] - d[nn].min()) / tau)
    w /= w.sum()
    ids, inv = np.unique(values[nn], return_inverse=True)
    return ids, np.bincount(inv, weights=w)


def sharded_knn(query, keys, values, shard_of, probed, k=K, tau=TAU):
    """Route to probed shards, search each locally, merge candidates by distance."""
    cand = np.concatenate([np.flatnonzero(shard_of == s) for s in probed])
    d = np.linalg.norm(keys[cand] - query, axis=1)
    nn = cand[np.argsort(d, kind="stable")[:k]]
    dn = np.linalg.norm(keys[nn] - query, axis=1)
    w = np.exp(-(dn - dn.min()) / tau)
    w /= w.sum()
    ids, inv = np.unique(values[nn], return_inverse=True)
    return ids, np.bincount(inv, weights=w)


# An IVF partition: assign every key to its nearest centroid, shard by centroid id.
centroids = keys[rng.choice(keys.shape[0], 64, replace=False)]
cell = np.argmin(((keys[:, None, :] - centroids[None]) ** 2).sum(-1), axis=1)
shard_of = cell // 8  # 8 contiguous centroid ranges -> 8 IndexIVFPQ shards

queries = rng.standard_normal((200, DIM)).astype(np.float32)
cell_d = np.linalg.norm(queries[:, None, :] - centroids[None], axis=2)


def recall_at(nprobe):
    """Fraction of the exact top-K recovered when only nprobe centroids are probed."""
    hits = total = 0
    for q, cd in zip(queries, cell_d):
        exact, _ = knn_distribution(q, keys, values)
        probed = np.unique(np.argsort(cd, kind="stable")[:nprobe] // 8)
        got, _ = sharded_knn(q, keys, values, shard_of, probed)
        hits += np.intersect1d(exact, got).size
        total += exact.size
    return hits / total


# Probing every centroid touches every shard: the merge is then exact by construction.
exhaustive = recall_at(64)
assert exhaustive == 1.0, exhaustive
# Probing too few centroids silently returns a different distribution. Nothing errors.
narrow = recall_at(1)
assert narrow < 0.7, narrow
assert recall_at(8) > narrow and recall_at(32) > recall_at(8)

# Merging on the wrong key is the classic sharded-search defect: taking each shard's
# best candidate without re-ranking globally by distance.
q = queries[0]
exact_ids, exact_p = knn_distribution(q, keys, values)
bad = []
for s in range(8):
    idx = np.flatnonzero(shard_of == s)
    bad.append(idx[np.argmin(np.linalg.norm(keys[idx] - q, axis=1))])
assert not np.array_equal(np.unique(values[np.array(bad)[:K]]), exact_ids)

# The distribution is a normalised, sparse vote: support <= K, mass sums to one.
ids, p = knn_distribution(q, keys, values)
assert ids.size <= K and abs(p.sum() - 1.0) < 1e-6
dup_keys = np.repeat(keys[:64], 4, axis=0)
dup_vals = np.repeat(values[:64], 4)
dids, dp = knn_distribution(q, dup_keys, dup_vals)
assert dids.size == K // 4 and abs(dp.sum() - 1.0) < 1e-6  # duplicates fold together

# Two lossy steps sit between the retriever and the KL target: epsilon truncation
# and FP16 storage. Neither preserves normalisation, so the retained probabilities
# have to be renormalised in FP32 on the GPU before the divergence is taken.
keep = p > EPS
stored = p[keep].astype(np.float16)
assert stored.sum(dtype=np.float64) != 1.0
renorm = stored.astype(np.float32) / stored.astype(np.float32).sum()
assert abs(renorm.sum(dtype=np.float64) - 1.0) < 1e-6
# Skipping the renormalisation biases the KL term by the dropped mass.
skewed = np.array([0.9, 0.05, 0.03, 0.02], np.float32)
kept = skewed[skewed > 0.04]
assert abs(kept.sum() - 1.0) > 0.04 and abs((kept / kept.sum()).sum() - 1.0) < 1e-6

# Sparse storage: INT64 row offset, INT32 label, then INT32 id + FP16 prob per pair.
PAIRS_PER_ROW = 64.95
dense_row = VOCAB * 2                       # FP16 over the full vocabulary
sparse_row = 8 + 4 + PAIRS_PER_ROW * (4 + 2)
ratio = dense_row / sparse_row
assert 249 < ratio < 251, ratio             # the paper's "approximately 250x"
# The saving is a vocabulary effect, not a compression trick: halve V, halve the ratio.
assert abs((VOCAB // 2 * 2) / sparse_row - ratio / 2) < 1e-9
# It also collapses once K approaches the vocabulary size.
assert (VOCAB * 2) / (8 + 4 + 5000 * (4 + 2)) < 4

print(
    f"nprobe recall 1/8/32/64 = {narrow:.2f}/{recall_at(8):.2f}/"
    f"{recall_at(32):.2f}/{exhaustive:.2f} | support={ids.size} "
    f"mass={p.sum():.6f} | dense/sparse={ratio:.1f}x"
)

Executed output:

nprobe recall 1/8/32/64 = 0.29/0.89/1.00/1.00 | support=16 mass=1.000000 | dense/sparse=250.5x

Two things follow. First, nprobe is a silent quality knob: at one probed centroid the pipeline recovers 29% of the true neighbours and reports no error, so the memory is trained on a target that is not the retriever's answer. Second, the 250x storage win is a vocabulary effect. It holds because 64.95 retained pairs per row is tiny against 50,304 vocabulary slots, and it shrinks proportionally on a smaller vocabulary and collapses entirely if K grows.

How to develop with it¶

The offline build is the part that needs engineering. The reference shape of the index, from the paper's Figure 3:

# Reference template (unexecuted): faiss-gpu index construction per shard.
# Pin faiss and validate recall against a brute-force sample before committing a build.
import faiss

DIM_IN, DIM_OUT, NLIST = 4096, 256, 1 << 18

opq = faiss.OPQMatrix(DIM_IN, DIM_OUT)          # 4096-d hidden state -> 256-d
quantizer = faiss.IndexHNSWFlat(DIM_OUT, 32)    # HNSW routes vectors and queries
ivfpq = faiss.IndexIVFPQ(quantizer, DIM_OUT, NLIST, 64, 8)
index = faiss.IndexPreTransform(opq, ivfpq)

Sequence the build in this order, because each step's cost depends on the previous one:

Emit keys with the frozen base model. One forward pass over the corpus, final-layer hidden state at each position. This is a full inference sweep over the whole corpus and it is not optional.
Train OPQ on a sample, then transform. Compression comes before indexing, so both the index and the distance computations work in 256 dimensions rather than 4096.
Learn the IVF partition and split it into shards by contiguous centroid range. Contiguity is what lets a shard answer a query batch with no cross-shard random access.
Route, then search. The HNSW quantizer picks the probed centroids and therefore the shards; queries are grouped per shard and searched in parallel on GPU. Writer workers merge candidates by query id into the final top K.
Validate recall against brute force on a held-out sample before you build the full index. This is the step teams skip. The pipeline cannot tell you it retrieved the wrong neighbours.
Write sparse rows, not dense ones. Offsets in INT64, token ids in INT32, probabilities in FP16, in sharded flat arrays with no padding.

For training itself, the memory is an ordinary decoder with a non-ordinary target. Preprocessing attaches the distribution row range to each packed example so shuffling keeps tokens and supervision aligned; workers memory-map the shards and read contiguous slices; the collator passes p_ret as sparse coordinate triplets alongside the usual language-modelling labels. Renormalise in FP32 on device before the KL, for the reason the executed block demonstrates.

The published run used 256 NVIDIA A800 80GB GPUs with Megatron-LM, training 1.4B, 2.8B, and 6.9B memories for 300B tokens (roughly 1.5 epochs over the datastore) at peak learning rates of 3e-4, 2.5e-4, and 2e-4, with AdamW (beta 0.9/0.95), weight decay 0.01, cosine decay to 10% of peak, and 2,000 warmup steps.

How to maintain it¶

The interpolation weight alpha is not a constant you set once. It is tuned per benchmark on a validation split and then fixed for test. The second executed block re-derives the published table and quantifies what that tuning is worth.

# interpolation_audit.py: executed audit of the inference rule and the published table.
import numpy as np

# Table 1 of the paper: (base, base+memory) per task at three matched scales,
# plus the frozen 12B backbone the headline compares against.
TABLE = {
    "ARC-Easy": ((61.83, 62.42), (63.64, 65.82), (68.39, 70.12), 70.58),
    "ARC-Challenge": ((27.39, 27.56), (30.03, 30.29), (33.11, 35.67), 33.45),
    "LAMBADA": ((61.91, 63.30), (65.09, 65.75), (68.76, 69.94), 70.99),
    "LogiQA": ((22.43, 22.73), (21.66, 21.97), (23.04, 23.04), 21.97),
    "PIQA": ((72.20, 73.23), (74.05, 74.92), (76.01, 76.17), 76.28),
    "SciQ": ((86.30, 88.00), (88.10, 90.20), (90.90, 91.50), 92.70),
    "WinoGrande": ((56.12, 59.43), (58.56, 61.25), (62.98, 64.72), 65.59),
    "MMLU": ((23.76, 24.75), (24.72, 24.87), (26.26, 26.95), 25.67),
    "NQ-Open": ((2.80, 3.88), (3.82, 5.15), (4.68, 6.32), 6.32),
    "TriviaQA": ((8.49, 10.08), (8.30, 17.11), (22.21, 25.58), 26.80),
    "PopQA": ((10.65, 11.90), (11.45, 12.83), (14.00, 15.13), 14.95),
    "2WikiMultiHopQA": ((16.89, 22.57), (18.96, 21.64), (20.60, 21.11), 20.57),
    "Bamboogle": ((1.60, 2.40), (0.80, 2.40), (4.00, 5.60), 2.40),
    "HotpotQA": ((6.75, 8.44), (8.83, 9.49), (8.97, 11.78), 11.11),
    "GPQA-main": ((24.53, 27.90), (24.91, 25.92), (24.11, 26.36), 25.49),
    "TruthfulQA": ((30.63, 30.46), (28.47, 27.92), (28.49, 27.81), 26.91),
    "HaluEval": ((42.67, 45.13), (44.68, 45.83), (40.53, 44.59), 41.23),
}
assert len(TABLE) == 17

pairs = np.array([[row[s] for s in range(3)] for row in TABLE.values()])  # 17 x 3 x 2
avg = pairs.mean(axis=0)
for got, want in zip(avg.ravel(), [32.76, 34.36, 33.89, 35.49, 36.30, 37.79]):
    assert abs(got - want) < 0.005, (got, want)
assert abs(np.array([r[3] for r in TABLE.values()]).mean() - 37.24) < 0.005

delta = pairs[:, :, 1] - pairs[:, :, 0]
improved, matched, regressed = (delta > 0).sum(), (delta == 0).sum(), (delta < 0).sum()
assert (improved, matched, regressed) == (47, 1, 3)  # the paper's 47-of-51 claim
# Every regression is the same benchmark, at every scale. It is not noise.
names = list(TABLE)
assert {names[i] for i, _ in zip(*np.where(delta < 0))} == {"TruthfulQA"}
assert (delta[names.index("TruthfulQA")] < 0).all()
assert delta[names.index("LogiQA")][2] == 0.0

# The gains are concentrated on knowledge tasks, but not exclusively so.
KNOWLEDGE = ["MMLU", "NQ-Open", "TriviaQA", "PopQA", "2WikiMultiHopQA", "Bamboogle",
             "HotpotQA", "GPQA-main"]
know = np.array([delta[names.index(n)] for n in KNOWLEDGE])
rest = np.array([delta[names.index(n)] for n in names if n not in KNOWLEDGE])
assert 1.6 < know.mean() / rest.mean() < 1.8       # 1.7x, not an order of magnitude
assert abs(delta[names.index("TriviaQA")][1] - 8.81) < 0.005      # largest single gain
assert abs(delta[names.index("2WikiMultiHopQA")][0] - 5.68) < 0.005
# Two general-task exceptions at the smallest scale, both over three points.
assert abs(delta[names.index("WinoGrande")][0] - 3.31) < 0.005
assert abs(delta[names.index("GPQA-main")][0] - 3.37) < 0.005
assert (np.abs(rest) < 2.0).sum() / rest.size > 0.7               # the rest move little

# The headline parameter saving is real, and it is the mixed-scale pairing that earns it.
assert abs((12.0 - (0.41 + 6.9)) / 12.0 - 0.39) < 0.005
# The matched-scale pairing at the top of the table is larger than the model it beats.
assert (6.9 + 6.9) > 12.0 and avg[2][1] > 37.24
# Only the 1.4B pairing is both same-size and better than the next backbone up.
assert (1.4 + 1.4) == 2.8 and avg[0][1] > avg[1][0]


def interpolate(p_base, p_mem, alpha):
    """Equation 2, applied to the full vocabulary at every decoded token."""
    return (1.0 - alpha) * p_base + alpha * p_mem


rng = np.random.default_rng(7)
p_base = rng.dirichlet(np.ones(8))
p_mem = rng.dirichlet(np.ones(8))
for a in (0.0, 0.25, 0.5, 0.75, 1.0):
    assert abs(interpolate(p_base, p_mem, a).sum() - 1.0) < 1e-12
assert np.allclose(interpolate(p_base, p_mem, 0.0), p_base)
assert np.allclose(interpolate(p_base, p_mem, 1.0), p_mem)

# A mixture is bounded by its components, so memory cannot raise a token above the
# larger of the two beliefs. It shifts the argmax only by demoting the base's choice.
mixed = interpolate(p_base, p_mem, 0.5)
assert (mixed <= np.maximum(p_base, p_mem) + 1e-12).all()
confident = np.zeros(8)
confident[0] = 1.0
gap = p_mem.max() - p_mem[0]          # memory's margin over the backbone's choice
flip = 1.0 / (1.0 + gap)              # alpha where the mixture's argmax changes
assert interpolate(confident, p_mem, flip - 0.01).argmax() == 0
assert interpolate(confident, p_mem, flip + 0.01).argmax() == p_mem.argmax() != 0
# Overriding a confident backbone needs alpha above 1/(1+gap), and the memory's own
# margin is small, so that threshold sits near the top of the range.
assert gap < 0.25 and flip > 0.8

# alpha is tuned per benchmark on a validation split. That tuning is load-bearing:
# two tasks with opposite base/memory reliability have opposite optima, and one
# shared alpha gives up accuracy on both.
def accuracy(base_right, mem_right, alpha, n=4000, seed=1):
    """Fraction correct when a two-way vote is resolved by the interpolation weight."""
    g = np.random.default_rng(seed)
    b = g.random(n) < base_right
    m = g.random(n) < mem_right
    picks_mem = alpha > 0.5
    return np.where(b == m, b, m if picks_mem else b).mean()


grid = np.linspace(0.0, 1.0, 21)
knowledge = np.array([accuracy(0.30, 0.62, a) for a in grid])   # memory-dominant task
reasoning = np.array([accuracy(0.71, 0.44, a) for a in grid])   # base-dominant task
assert grid[knowledge.argmax()] > 0.5 > grid[reasoning.argmax()]
per_task = knowledge.max() + reasoning.max()
shared = (knowledge + reasoning).max()
assert per_task - shared > 0.15                                 # >15 points given up

print(
    f"AVG {avg.ravel().round(2).tolist()} vs 12B 37.24 | "
    f"improved/matched/regressed = {improved}/{matched}/{regressed} "
    f"(all regressions TruthfulQA) | tuned-vs-shared alpha gap "
    f"{per_task - shared:.3f}"
)

Executed output:

AVG [32.76, 34.36, 33.89, 35.49, 36.3, 37.79] vs 12B 37.24 | improved/matched/regressed = 47/1/3 (all regressions TruthfulQA) | tuned-vs-shared alpha gap 0.286

Three maintenance consequences. The published table reproduces exactly, including its own "improves 47 of 51" claim, and every one of the three regressions is TruthfulQA at a different scale, so treat calibration as a known cost rather than variance. The parameter-efficiency headline belongs to the mixed-scale pairing (410M backbone plus 6.9B memory, 7.31B against 12B); the matched-scale pairing at the top of the table, 6.9B plus 6.9B, is 13.8B and therefore larger than the 12B model it beats. And because alpha optima move in opposite directions for knowledge-heavy and reasoning-heavy traffic, a single production alpha across mixed traffic gives back a large share of the benefit. Route by task, or accept the loss and measure it.

How to run it in production¶

Serving is the part the parameter count hides. Both models are resident, both are stepped on every token, and the mix happens after both have produced a distribution over the full vocabulary.

Budget two models, not one. The pair holds two sets of weights and two KV caches. The 410M-plus-6.9B configuration is 39% fewer parameters than a 12B, but it is 6% more than a 6.9B backbone alone, and it needs two engine instances.
The two forwards can overlap, so latency need not be the sum. The paper notes this. Realising it means running the backbone and memory on separate devices or streams, with a synchronisation point per decoded token.
That per-token barrier is the risk. Speculative decoding, chunked prefill, and continuous batching all assume a single logits producer. Confirm your engine can join two per-step distributions before you commit to the design.
Version the pair. A memory is trained against a specific backbone's hidden-state geometry. Deploy backbone and memory as one artifact with one version, and reject a mismatched pair at load time rather than at decode.
Refreshing knowledge means rebuilding the datastore and retraining. The corpus is compiled into weights. Plan the cadence around the offline build (a full inference sweep plus an index build over the corpus), not around the training run.
Domain memories are swappable at inference, but each needs its own alpha. Carry the tuned value with the memory artifact.

Failure modes¶

Silent retrieval degradation. A low nprobe, a badly trained OPQ transform, or an unbalanced centroid partition returns plausible neighbours that are not the true top K. Nothing raises. The only detection is periodic brute-force recall on a held-out sample.
Merge-by-wrong-key. Taking each shard's best candidate without re-ranking globally by distance yields a different distribution, as the executed block shows. Merge by query id and then re-rank.
Unnormalised targets. Epsilon truncation plus FP16 storage both drop mass. Feeding the stored values straight into a KL biases the objective by the dropped remainder.
Increased extractable memorization. On a 1.7B biology memory against a continued-pretraining baseline trained on the same data and budget, strict exact-match completion of 16-token suffixes rose from 42.4% to 49.7%, and a domain-anchor completion probe rose from 22.6% to 56.5%. The paper frames this as traceability. For a corpus with licensing or privacy constraints it is the same measurement read as a leak, and it argues against training memories on sensitive text.
Calibration regression. TruthfulQA falls at every scale tested. If your product depends on refusal or hedging behaviour, gate on it explicitly.
A shared alpha across mixed traffic. Modelled above: opposite optima for knowledge-heavy and reasoning-heavy tasks, and a large gap between per-task and shared tuning.
Treating this as a drop-in for RAG. There is no retrieval at inference and therefore no citation, no provenance, and no way to revoke a document short of retraining.
Vocabulary lock-in. Memory and backbone must share a tokenizer and output vocabulary. Crossing families needs an embedding and head replacement plus further training, not a config change.

References¶

Wei, Cao, Wang, Zhang, Guo, Zhou and Lin, "Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory" (arXiv 2607.27919, 30 July 2026): https://arxiv.org/abs/2607.27919
Reference implementation, LUMIA-Group/MemoryDecoder-at-Scale: https://github.com/LUMIA-Group/MemoryDecoder-at-Scale
Cao et al., "Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models" (the smaller-scale predecessor): https://arxiv.org/abs/2508.09874
Wei et al., "MLP Memory: A Retriever-Pretrained Memory for Large Language Models": https://arxiv.org/abs/2508.01832
Khandelwal, Levy, Jurafsky, Zettlemoyer and Lewis, "Generalization through Memorization: Nearest Neighbor Language Models" (kNN-LM, the retriever being imitated): https://arxiv.org/abs/1911.00172
Douze et al., "The Faiss Library": https://arxiv.org/abs/2401.08281
Malkov and Yashunin, "Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs" (the IVF quantizer): https://arxiv.org/abs/1603.09320