Markdown

Multi-agent KV reuse: prefill under diverging prefixes¶

Scope: eliminating redundant prefill in LLM multi-agent systems, where several agents process heavily overlapping context but ordinary prefix caching does not apply because each agent sees that shared text behind a different prefix. This page covers the offset-variance problem, the anchor-pool approach of KVCOMM (arXiv 2510.12872), the workload shape that conversational multi-agent frameworks such as AutoGen (arXiv 2308.08155) produce, and the arithmetic that bounds what any reuse rate can buy. Reuse across requests to one model with an identical prefix is prefix caching; reuse across different models is cross-model KV transfer; the agent-team design patterns themselves are multi-agent collaboration and the orchestration control plane.

The numpy blocks below are runnable, self-checking validations of the mechanics and the arithmetic this page teaches: the exactness of RoPE realignment for a segment that moves position, how the reusable share of prefill grows with agent count in a fully-connected topology, the ceiling a given reuse rate places on speedup, and anchor-pool hit rate under frequency-biased eviction. Each was executed on a stock python3 with numpy and asserts its result; the pasted output is the real output. The cost model is a first-order FLOP proxy, not a simulation of any serving stack. No accuracy, reuse-rate or TTFT number from the cited papers was reproduced; those are quoted and attributed.

What it is¶

A conversational multi-agent system runs several LLM agents that exchange messages. AutoGen established the dominant shape: customizable, conversable agents that talk to each other, mixing LLMs, humans and tools, with conversation patterns programmable in natural language or code.² The pattern is now standard, and it has an infrastructure consequence that the framework literature does not address.

Every message an agent processes goes through prefill first. Agents in these systems share a great deal of context: the same retrieved passages, the same task statement, the same peer outputs. But they do not share a prefix. Agent 3 sees the system prompt, then agent 1's output, then agent 2's output, then its own instructions. Agent 4 sees a different arrangement. The shared content sits at a different offset in every agent's context, and often behind different preceding text.

That breaks prefix caching, which requires an exact matching prefix. The result is that multi-agent systems recompute KV for the same tokens over and over. KVCOMM names the obstacle offset variance and addresses it with a training-free framework that estimates what a cached segment's KV would have been under a different prefix, by referencing a pool of previously observed deviations called anchors.¹

Why use it¶

The redundancy is structural, not incidental, and it grows with the size of the agent team. In a topology where each agent sees all its predecessors' outputs plus a shared prefix, the fraction of prefill tokens that are in principle reusable rises steadily with agent count.

import numpy as np

def token_volume(n_agents, own=1024, prefix=512):
    """Tokens each agent must prefill when every agent sees all predecessors."""
    total, shared = 0, 0
    for i in range(n_agents):
        total += prefix + own + i * own       # shared prefix, own input, predecessors
        shared += prefix + i * own            # everything except its own input
    return total, shared

for n in (2, 3, 5, 8):
    tot, sh = token_volume(n)
    print(f"{n} agents: {tot:6d} prefill tokens, {sh:6d} reusable ({sh/tot:.1%})")
assert token_volume(5)[1] / token_volume(5)[0] > token_volume(2)[1] / token_volume(2)[0]

2 agents:   4096 prefill tokens,   2048 reusable (50.0%)
3 agents:   7680 prefill tokens,   4608 reusable (60.0%)
5 agents:  17920 prefill tokens,  12800 reusable (71.4%)
8 agents:  40960 prefill tokens,  32768 reusable (80.0%)

This is why the technique is worth more to a large agent team than a small one, and it is also why published headline numbers are usually drawn from the largest configuration measured. KVCOMM reports over 70% reuse across retrieval-augmented generation, math reasoning and collaborative coding, with under 2.5% accuracy drop, and a TTFT reduction from roughly 430 ms to roughly 55 ms for the best-placed agent in a five-agent run.¹

When to use it (and when not)¶

Use it when your agents genuinely share long context and you are latency-sensitive on time to first token. RAG pipelines where several agents read the same retrieved passages are the clearest case.

Do not use it when:

Agents share little context. A pipeline where each agent gets a short, distinct instruction has nothing to reuse.
Contexts are short. The measured mean speedup falls to 2.24x at the shortest prefix setting tested (64 tokens).¹
You cannot tolerate approximate KV. This is an approximation method: it estimates cache deviations rather than computing them. It is not lossless, and the reported quality claim is "no degradation" only within the measured 2.5% band on the measured tasks.
Your bottleneck is decode, not prefill. This addresses TTFT. Total latency in a long-generation agent loop is often dominated by decode, which this does not touch.

What a reuse rate can and cannot buy¶

Reuse rate is the number these systems advertise, but it is not the speedup. The relationship is bounded, and it is worth deriving before setting expectations.

def prefill_cost(n_new, n_ctx, d_model=4096, n_layer=32):
    """First-order prefill FLOPs: projections over new tokens + attention over context."""
    linear = n_new * d_model * d_model * 12 * n_layer
    attn = n_new * n_ctx * d_model * 2 * 2 * n_layer
    return linear + attn

ctx = 4096
full = prefill_cost(ctx, ctx)
print(f"{'reuse':>6} {'speedup':>9}")
for rho in (0.5, 0.7, 0.876, 0.95):
    new = round(ctx * (1 - rho))
    sp = full / prefill_cost(new, ctx)
    print(f"{rho:6.1%} {sp:8.2f}x")
    # Both terms are linear in n_new at fixed context, so the saving is exactly the
    # token ratio: skipped tokens buy nothing extra from the quadratic term, because
    # the tokens still prefilled attend over the whole context anyway.
    assert abs(sp - ctx / new) < 1e-9
    assert sp <= 1 / (1 - rho) * (1 + 1e-3)

 reuse   speedup
 50.0%     2.00x
 70.0%     3.33x
 87.6%     8.06x
 95.0%    19.98x

Two consequences. First, prefix reuse saves exactly in proportion to the tokens skipped; there is no extra credit from attention being quadratic, because the tokens you still prefill must attend over the whole context regardless. Second, a 70% reuse rate cannot produce more than a 3.33x prefill speedup. KVCOMM's reported reuse rate spans 70 to 87.6%, and its headline 7.8x sits at the top of that range, not the advertised floor: at 87.6% this model gives 8.06x. Both published numbers are real, but they describe different operating points, and the abstract pairs the reuse floor with the speedup peak in a single sentence. The paper's own means are the honest planning numbers: 2.24x at the shortest setting, 6.72x at the longest, and about 6.7x average in the three-agent configuration.¹

A related detail worth knowing if you compare against the first preprint: the v2 revision states that "in the original submission, the TTFT calculation for KVCOMM omitted the first token's decoding latency", so v1's speedups were measured on a more favourable definition.¹ Cite v2.

Architecture¶

flowchart TB
  REQ["agent request<br/>(shared segment + own prefix)"] --> MATCH
  subgraph POOL["anchor pool (size V, online)"]
    A1["anchor: base KV<br/>+ observed offsets per prefix"]
  end
  MATCH["anchor matching<br/>by embedding proximity<br/>+ length compatibility"] --> DEC{"shareable?"}
  POOL -.lookup.-> MATCH
  DEC -->|yes| ALIGN["RoPE de-rotate / re-rotate<br/>to the new offset"]
  ALIGN --> EST["add estimated content offset<br/>from matched anchors"]
  EST --> SKIP["skip prefill for the segment"]
  DEC -->|no| FULL["dense prefill"]
  FULL --> NEW["register as a new anchor,<br/>measure its offsets"]
  NEW -.update.-> POOL
  SKIP --> POOL

The alignment splits into two parts, and only one of them is hard.

The positional part is exactly recoverable. A segment that moves from offset 100 to offset 356 has had a known rotation applied to its keys; undoing it and re-applying the new one is exact, because RoPE is orthogonal.

The content part is not recoverable in closed form. The same text preceded by different content produces genuinely different keys and values, because attention over the preceding tokens differs. This residual is what the anchor pool estimates from previously observed deviations. The block below separates the two.

import numpy as np
rng = np.random.default_rng(3)

def rope(x, pos, base=10000.0):
    d = x.shape[1]
    inv = base ** (-np.arange(0, d, 2) / d)
    ang = pos[:, None] * inv[None, :]
    c, s = np.cos(ang), np.sin(ang)
    xe, xo = x[:, 0::2], x[:, 1::2]
    out = np.empty_like(x)
    out[:, 0::2] = xe * c - xo * s
    out[:, 1::2] = xe * s + xo * c
    return out

seg = rng.standard_normal((128, 64))
k_at_100 = rope(seg, np.arange(100, 228))     # same text after a 100-token prefix
k_at_356 = rope(seg, np.arange(356, 484))     # same text after a 356-token prefix

raw = np.abs(k_at_100 - k_at_356).max()
realigned = np.abs(rope(rope(k_at_100, -np.arange(100, 228)),
                        np.arange(356, 484)) - k_at_356).max()
print(f"same segment, different offset: raw max|dK| = {raw:.3f}")
print(f"after de-rotate / re-rotate:    {realigned:.2e}")
assert raw > 1.0 and realigned < 1e-9

same segment, different offset: raw max|dK| = 7.100
after de-rotate / re-rotate:    8.88e-16

The positional component is large enough that ignoring it corrupts the cache outright, and it costs essentially nothing to remove. The anchor machinery exists for what remains.

The anchor pool is a cache with an eviction policy¶

Anchors are stored per placeholder segment, holding a base KV plus the observed deviations under various prefixes. Admission is governed by a criterion combining sequence length against existing anchors and an entropy term over embedding-distance weights, with a threshold gamma. The pool is capped at size V (20 in the experiments) and evicts "the least frequently accessed anchor among the earliest-added entries".¹

That makes pool sizing an ordinary cache-tuning problem, with the usual shape.

def hit_rate(pool_size, n_req=4000, n_distinct=60, zipf=1.2, seed=3):
    r = np.random.default_rng(seed)
    p = 1.0 / np.arange(1, n_distinct + 1) ** zipf
    p /= p.sum()
    pool, freq, hits = [], {}, 0
    for req in r.choice(n_distinct, size=n_req, p=p):
        if req in pool:
            hits += 1
            freq[req] = freq.get(req, 0) + 1
            continue
        if len(pool) >= pool_size:
            victim = min(pool, key=lambda x: freq.get(x, 0))
            pool.remove(victim); freq.pop(victim, None)
        pool.append(req); freq[req] = 1
    return hits / n_req

for v in (5, 10, 20, 40):
    print(f"anchor pool V={v:3d}: hit rate {hit_rate(v):.1%}")
assert hit_rate(40) > hit_rate(5)

anchor pool V=  5: hit rate 54.7%
anchor pool V= 10: hit rate 71.1%
anchor pool V= 20: hit rate 80.6%
anchor pool V= 40: hit rate 92.4%

The default V=20 is not a law of nature. If your workload has more distinct shared segments than the pool holds, raising V is the first thing to try, and the cost is GPU memory holding anchor KV.

How to run it in production¶

Report the mean, not the best agent. The per-agent spread is large by construction, since later agents in a pipeline have more reusable context. A fleet-level TTFT p50 and p95 are the numbers that matter.
Keep dense prefill as the fallback and instrument its rate. The design already falls back when a segment is judged unshareable; that rate is your best early-warning signal that the workload has drifted away from the anchor pool.
Treat request order as a variable. The paper reports performance is correlated with request order, because the admission criterion depends on which samples arrive first and become anchors.¹
Gate on task accuracy, not reconstruction fidelity. The comparison baseline here is instructive: CacheBlend, an alternative reuse scheme, dropped GSM8K accuracy from 82.0% to 57.1% in the same harness where KVCOMM stayed within 1.9 points.¹ Approximate KV reuse can fail catastrophically and silently on reasoning tasks while looking fine on aggregate metrics.
Budget the anchor pool as GPU memory. It holds KV, and it grows with the diversity of your shared segments.

How to maintain it¶

Re-tune V and the entropy threshold gamma when the agent topology or the retrieval corpus changes; both control admission and therefore reuse rate.
Re-measure after any model change. Offsets are a property of the model's attention, so anchors do not transfer across checkpoints.
Track reuse rate and accuracy together on the same dashboard. Reuse rate rising while accuracy falls is the signature of an admission threshold that has become too permissive.
Revisit whether you need this at all if your framework gains real prefix-sharing support. The problem is created by how conversational frameworks assemble contexts; a topology that keeps a stable shared prefix in front for every agent can use ordinary prefix caching and skip this machinery entirely.

Failure modes¶

Quoting the peak speedup as the expected speedup. 7.8x is one agent in the largest configuration; the reported means run 2.24x to 6.72x.
Expecting a 70% reuse rate to give more than 3.33x. The bound is arithmetic.
Ignoring the v1-to-v2 metric change. The original submission omitted the first token's decoding latency from TTFT.
Assuming lossless reuse. It is an estimator. The accuracy budget is real and task-dependent.
Leaving V at its default on a workload with many distinct shared segments. Hit rate, and therefore reuse rate, degrades quietly.
Benchmarking with a fixed request order. Results depend on it, so a single ordering can flatter or penalize the system.
Applying it where decode dominates. This is a TTFT optimization; a long-generation agent loop may barely notice it.

Open questions and validation¶

Nothing here reproduces the paper's reuse rates, TTFT figures or accuracy numbers; code is published at https://github.com/FastMAS/KVCOMM but was not run for this page.
The interaction between approximate KV reuse and long agent chains is unmeasured: errors introduced at agent 2 propagate into agent 5's context, and the published evaluations run short pipelines.
Whether anchor-based estimation composes with token eviction or quantized caches, where the base KV is itself lossy, is not addressed.
Whether this composes with cross-model transfer in a heterogeneous agent team, where agents run different models, is open in both literatures.

References¶

Ye et al., KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems (arXiv 2510.12872, NeurIPS 2025): https://arxiv.org/abs/2510.12872 and https://github.com/FastMAS/KVCOMM
Wu et al., AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation (arXiv 2308.08155): https://arxiv.org/abs/2308.08155
Yao et al., CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion (EuroSys 2025), the reuse baseline KVCOMM compares against: https://arxiv.org/abs/2405.16444
Gim et al., Prompt Cache: Modular Attention Reuse for Low-Latency Inference (MLSys 2024): https://arxiv.org/abs/2311.04934

Ye, Gao, Ma, Wang, Fu, Chung, Lin, Liu, Zhang, Zhuo and Chen, "KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems", arXiv 2510.12872v2 (v1 14 Oct 2025, v2 1 Nov 2025), Duke University with MIT and NVIDIA co-authors, NeurIPS 2025; code at https://github.com/FastMAS/KVCOMM. Problem framing: multi-agent pipelines reprocess overlapping context because KV caching requires unchanged prefixes, and agent-specific context extensions create diverging prefixes; the named obstacle is offset variance of KV-caches across agents. Method: training-free; an anchor pool stores, per placeholder segment, a base KV plus observed cache deviations under varying prefixes, keyed by an embedding and length. Admission criterion P_anchor(phi) fires when the sample is longer than every existing anchor or when an entropy term over embedding-distance softmax weights exceeds gamma log|A_phi|. Reuse path performs anchor matching, RoPE de-rotation and re-rotation to the new offset, then adds an estimated content offset from matched anchors; unshareable segments fall back to dense prefill and are registered as new anchors. Pool is capped at V (20 in experiments) with eviction of "the least frequently accessed anchor among the earliest-added entries". Setup: Llama-3.1-8B-Instruct on a single NVIDIA H100. Reported results: reuse rate 70 to 87.6%, declining as agent count grows; accuracy within 2.5% of the original workload, MMLU 64.7 to 69.9%, GSM8K declining by 1.9%, against CacheBlend which fell from 82.0% to 57.1% on GSM8K. TTFT (Table 2, each agent receiving 1K user tokens with 512 prefix and 512 output tokens, five agents): prefill reduced to 26.9 to 38.6 ms, up to 7.82x speedup on Agent 5, with the abstract quoting roughly 430 ms down to roughly 55 ms. Table 3 scalability over prefix lengths 64 to 1K gives mean speedups from 2.24x (shortest) to 6.72x (longest); the conclusion quotes about 6.7x average prefilling speedup in the three-agent setting. A footnote to the v2 revision states: "In the original submission, the TTFT calculation for KVCOMM omitted the first token's decoding latency." Robustness study reports performance correlates with request order because of the anchor prediction criterion. Ablations cover the three alignment steps and hyperparameter sensitivity to gamma and V. None of these numbers were reproduced for this page. ↩↩↩↩↩↩↩↩
Wu, Bansal, Zhang, Wu, Li, Zhu, Jiang, Zhang, Zhang, Liu, Awadallah, White, Burger and Wang, "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation", arXiv 2308.08155 (v1 16 Aug 2023, v2 3 Oct 2023), Microsoft Research with Penn State and University of Washington co-authors. Open-source framework for building LLM applications from multiple conversing agents. Agents are customizable, conversable, and can operate in modes combining LLMs, human input and tools; interaction behaviour is developer-defined, and both natural language and code can program conversation patterns. Positioned as generic infrastructure spanning mathematics, coding, question answering, operations research, online decision-making and entertainment. Cited here for the workload shape rather than for any performance claim: the conversable-agent pattern is what produces the overlapping-but-not-prefix-matching contexts that KVCOMM addresses, and the paper itself makes no claims about prefill cost or KV-cache reuse. ↩