Copy for agents Markdown

Centralized KV cache placement: when a shared store pays and when it cannot

Scope: where the KV cache should live in distributed and decentralized LLM inference. Covers the shared (centralized) KV store pattern behind Mooncake, LMCache, and BanaServe, the impractical per-token KV-streaming variant, the low-latency-fabric DistAttention variant, and an executed first-order model for deciding whether a one-shot prefix fetch can beat local recomputation. Complements disaggregated inference (the intra-datacenter pattern that uses a KV handoff) and cross-WAN model-parallel inference (the decentralized pattern that never moves KV at all). Classification: a decision and reference page for KV-placement economics, not itself an installable technology; the HOW sections below describe how to apply the break-even model to your own fabric, not how to install a specific product (that install path lives in LMCache's own docs).

Verdict up front: a centralized KV store is useful when reusable KV is transferred once per request and the measured end-to-end fetch time is below local recomputation time. The network-only boundary is net_bw > kv_bytes_per_token x prefill_tok_s / compression, roughly 13 Gb/s for Llama-3.1-70B at FP16 in the executed example below. Streaming the full KV history per generated token is impractical. Shipping attention queries to remote KV is a different design that can work on a tightly coupled, low-latency GPU fabric, but it is unsuitable for WAN decode.

What it is

A centralized KV cache is a store that holds KV blocks outside the GPU that will consume them: a dedicated cache server (LMCache server mode), a pool built from the cluster's idle DRAM and SSD (Mooncake¹), or a global store that any prefill or decode instance can read (BanaServe⁴). Serving instances fetch KV blocks from it instead of recomputing them with a prefill pass.

The design space has three interaction patterns, and they are not variations of one idea; they live on different timescales and have opposite economics:

1. Once-per-request prefix fetch. A request arrives whose prefix (system prompt, shared document, earlier turns of the conversation) already has KV in the store. The serving instance pulls those blocks once, then decodes with a fully local cache. This is Mooncake, LMCache, the vLLM disaggregated prefill handoff over NIXL, and CacheGen's compressed streaming². The executed model below gives a network-only boundary; storage, queueing, serialization, and codec costs must be measured separately.
2. Per-token KV streaming. Decode runs on a node that does not hold the KV; every generated token pulls the attention history over the network. No shipping system does this, for the reason quantified below: at 8k context on a 70B model it needs 2.6 GB of reads per token.
3. Per-token attention offload (ship the query to the KV). The KV stays in a pooled GPU-memory tier and attention queries are sent to the nodes holding the relevant cache. Infinite-LLM's DistAttention³ implements this pattern inside a 32-GPU A100 cluster. Its query transfers are small (roughly 0.075 to 0.36 ms in the paper's Figure 4) and overlap with local attention work; the paper reports 1.35x to 3.4x cluster throughput improvement. Those results depend on a low-latency datacenter fabric and do not justify putting layer-wise attention RPCs on a WAN.

Why use it

• Prefix deduplication across replicas. Fleet-wide, most cache-worthy bytes are shared prefixes (system prompts, RAG documents, multi-turn history). A central store lets any replica serve a conversation without having previously seen it, so the router can balance purely on load. BanaServe makes this decoupling explicit: its Global KV Cache Store lets routing ignore cache placement and still reports 1.2x to 3.9x higher throughput than vLLM and 1.1x to 2.8x over DistServe⁴.
• Capacity beyond HBM. Mooncake builds its pool from "the underutilized CPU, DRAM, and SSD resources of the GPU cluster", reporting up to a 525% throughput increase in simulated long-context scenarios and 75% more requests handled under Kimi's real workload¹. The win is reuse that HBM alone could not hold, not faster decode.
• Churn survival in decentralized serving. In a cross-WAN pipeline, a healed stage currently pays a full re-prefill of the accumulated history. A central session store turns that into one fetch, which is the TTFT-shaped benefit a high-bandwidth central node genuinely can deliver to decentralized decode nodes.
• Slow-network reuse becomes viable with compression. CacheGen compresses KV 3.5x to 4.3x and streams it, cutting total context-loading delay 3.2x to 3.7x versus fetching raw KV or re-prefilling, with adaptive quality under varying bandwidth². Compression moves the break-even bandwidth down by the same factor, which is what makes the pattern reach beyond the datacenter.

When to use it (and when not)

Use a centralized KV store when all of these hold:

• The workload has real prefix reuse across instances or across time (multi-turn chat, shared system prompts, RAG over a common corpus). A store nobody hits twice is pure overhead.
• The fetch path clears the break-even: effective bandwidth above kv_bytes_per_token x prefill_tok_s / compression (executed below; about 13 Gb/s uncompressed for Llama-3.1-70B FP16, about 3.7 Gb/s with CacheGen-class compression). Below that line, recomputing the prefix locally is faster than fetching it.
• The store sits on the same fast fabric as the decode pool, or the use is explicitly TTFT-shaped (session resume, churn recovery) where a WAN fetch still beats a WAN re-prefill.

Do not use it when:

• You expect it to speed up steady-state decode. It cannot. Decode reads the whole KV history every token; only local HBM sustains that. The Petals measurement is the cleanest external evidence that the network term that hurts decode is RTT, not bandwidth: cutting bandwidth 1 Gbit/s to 100 Mbit/s costs 17.6 ms/step, while adding 95 ms of RTT costs 210.6 ms/step, twelve times more⁵.
• The fetch would cross a WAN as part of a prefill/decode split, for a dense-attention model. Disaggregated inference pins prefill and decode pools to fast-interconnect reach because dense-model KV egress can saturate an inter-site link; Moonshot's profiling-based model estimates 3.8 Tbps of egress for MiniMax-M2.5 (2.1 Tbps for Qwen3-235B) at 32K context on 512 H200s⁶. The rule is conditional on KV volume, not architectural law: the same paper evaluates a cross-datacenter handoff for hybrid-attention and MLA models over a modeled 100 Gb/s link. Its comparable 32K measurements span about 4x (Qwen3.5 versus Qwen3) to 13x (MiMo-V2-Flash versus MiniMax-M2.5) lower per-instance KV throughput; the separate 36x figure is an estimated Ring-2.5-1T memory saving from MLA compression and its 7:1 linear-to-full-attention layer ratio⁶. This is an analytical case study fed by measured profiling data, not a deployed cross-datacenter measurement.
• Prefixes are short. Below rtt x prefill_tok_s tokens (100 tokens at 20 ms RTT and 5,000 tok/s prefill), one round trip already costs more than recomputing; the exact-boundary function in the model below fails closed on this case.
• Tenancy or integrity boundaries would be crossed. A shared store is a shared fate domain; see tenant cache isolation before pooling KV across tenants.

Architecture

flowchart LR
  subgraph Store["Central KV store (DRAM/SSD pool or LMCache server)"]
    S["Prefix and session KV blocks<br/>keyed by token-sequence hash"]
  end
  P["Prefill instance"] -->|"store once<br/>after prefill"| S
  S -->|"fetch once per request<br/>(the pattern that pays)"| D1["Decode instance A<br/>local HBM KV, local attention"]
  S -->|"fetch once on resume<br/>or node churn"| D2["Decode instance B<br/>(different replica or region)"]
  D1 -.->|"never per token:<br/>2.6 GB/token at 8k ctx"| S

The solid edges are the whole architecture: KV enters the store once after prefill and leaves it once per request (or once per session resume). Decode attention always runs against HBM-local blocks. The dashed edge is the variant this page exists to rule out; the moment KV crosses the network per token, the design has silently changed from a cache into a remote memory bus, and the numbers below apply.

The three placements compared:

Placement	Network cost per token	Binding constraint	Verdict
KV local to decode (baseline, incl. pipeline splits)	one activation vector per stage hop (4-8 KiB)	stages x RTT	The production default; what Mesh LLM and Petals do
Central store, once-per-request fetch	zero (fetch amortized over the whole request)	fetch bandwidth vs re-prefill rate	Pays above break-even bandwidth; this page's pattern
Central KV, per-token streaming	context x kv_bytes_per_token	bandwidth	250x slower than the modeled HBM path at 100 Gb/s; do not ship full-history KV per token
Distributed attention over pooled KV	KiB-scale queries plus attention outputs	fabric latency, scheduling, and overlap	Viable on a low-latency datacenter fabric (Infinite-LLM3); not a WAN pattern

The simple n_layers x RTT calculation below is a conservative WAN rejection bound for a sequential RPC implementation. It is not a performance model of Infinite-LLM: DistAttention overlaps remote and local attention work and runs on a low-latency cluster fabric. Megatron-LM's communication analysis still explains why per-layer synchronization is more latency-sensitive than pipeline-parallel stage boundaries⁸.

How it works (validated decision model)

The whole decision reduces to four cost functions and one break-even. This model was executed as written and every assert passed; the printed output follows the listing.

"""Decide when a centralized KV cache helps and when it cannot.

Models the three KV placements for distributed decode plus the one-shot
prefix-fetch case, then asserts the decision boundaries hold.
"""


def kv_bytes_per_token(n_layers: int, n_kv_heads: int, head_dim: int,
                       bytes_per_el: int) -> int:
    """2x covers keys and values (same formula as the NIXL sizing rule)."""
    return 2 * n_layers * n_kv_heads * head_dim * bytes_per_el


def decode_local_s(ctx: int, kvpt: int, hbm_bw: float, stages: int,
                   rtt: float) -> float:
    """KV local to each stage: attention reads history from HBM, the only
    network cost is one activation hop per pipeline stage."""
    return ctx * kvpt / hbm_bw + stages * rtt


def decode_stream_central_s(ctx: int, kvpt: int, net_bw: float,
                            rtt: float) -> float:
    """Variant A: decode node pulls the full KV history over the network
    for every generated token."""
    return ctx * kvpt / net_bw + rtt


def decode_ship_q_s(n_layers: int, rtt: float) -> float:
    """Variant B: queries shipped to the central KV node, one blocking
    round trip per layer (layer l+1 depends on layer l's output).
    Note: no bandwidth parameter exists; payloads are KiB-sized."""
    return n_layers * rtt


def prefix_fetch_s(ctx: int, kvpt: int, net_bw: float, rtt: float,
                   compression: float = 1.0) -> float:
    """One-shot fetch of a cached prefix from a central store."""
    return rtt + ctx * kvpt / (net_bw * compression)


def reprefill_s(ctx: int, prefill_tps: float) -> float:
    """Recompute the prefix locally instead of fetching it."""
    return ctx / prefill_tps


def break_even_bw(kvpt: int, prefill_tps: float,
                  compression: float = 1.0) -> float:
    """Bandwidth above which fetching a cached prefix beats recomputing it
    (RTT ignored; exact in the large-context limit)."""
    return kvpt * prefill_tps / compression


def break_even_bw_exact(ctx: int, kvpt: int, prefill_tps: float, rtt: float,
                        compression: float = 1.0) -> float:
    """RTT-aware break-even; only defined when recompute takes longer than
    one RTT (below that, fetching can never win at any bandwidth)."""
    budget = ctx / prefill_tps - rtt
    assert budget > 0, "recompute is faster than one RTT; never fetch"
    return ctx * kvpt / (budget * compression)


# --- Llama-3.1-70B, FP16 KV, GQA: 80 layers, 8 KV heads, head_dim 128 ---
KVPT = kv_bytes_per_token(80, 8, 128, 2)
assert KVPT == 327_680, KVPT                       # 320 KiB per token

HBM = 3.35e12          # H100 SXM HBM3, B/s
NET_DC = 12.5e9        # 100 Gb/s in B/s
NET_WAN = 0.125e9      # 1 Gb/s in B/s
CTX = 8192

# Variant A: streaming the history per token loses by orders of magnitude.
local = decode_local_s(CTX, KVPT, HBM, stages=4, rtt=0.020)
stream_dc = decode_stream_central_s(CTX, KVPT, NET_DC, rtt=0.001)
stream_wan = decode_stream_central_s(CTX, KVPT, NET_WAN, rtt=0.020)
assert 0.080 < local < 0.082                       # 4 WAN hops dominate
assert stream_dc > 0.21                            # ~215 ms/token at 100 Gb/s
assert stream_wan > 21.0                           # ~21 s/token at 1 Gb/s
assert stream_dc / (CTX * KVPT / HBM) > 250        # vs 0.8 ms HBM read

# Variant B: latency-bound, bandwidth cannot appear in the cost at all.
ship_q = decode_ship_q_s(n_layers=80, rtt=0.020)
pipeline_net = 4 * 0.020
assert ship_q == 1.6                               # 80 RTTs per token
assert ship_q / pipeline_net == 20                 # == n_layers / stages

# One-shot prefix fetch: the case where central KV genuinely wins.
PREFILL_TPS = 5_000
be = break_even_bw(KVPT, PREFILL_TPS)
assert abs(be - 1.6384e9) < 1e3                    # ~13.1 Gb/s uncompressed
be_cachegen = break_even_bw(KVPT, PREFILL_TPS, compression=3.5)
assert be_cachegen < 0.47e9                        # ~3.7 Gb/s compressed

# The boundary is real: fetch wins above it, loses below it.
for ctx in (1_024, 8_192, 131_072):
    be_x = break_even_bw_exact(ctx, KVPT, PREFILL_TPS, rtt=0.02)
    hi, lo = be_x * 1.001, be_x * 0.999
    assert prefix_fetch_s(ctx, KVPT, hi, 0.02) < reprefill_s(ctx, PREFILL_TPS)
    assert prefix_fetch_s(ctx, KVPT, lo, 0.02) > reprefill_s(ctx, PREFILL_TPS)
    assert be_x > be                       # RTT always raises the bar

# Bisection crossover converges to the closed form as context grows.
def crossover(ctx: int) -> float:
    lo_bw, hi_bw = 1e6, 1e12
    for _ in range(200):
        mid = (lo_bw + hi_bw) / 2
        if prefix_fetch_s(ctx, KVPT, mid, 0.02) < reprefill_s(ctx, PREFILL_TPS):
            hi_bw = mid
        else:
            lo_bw = mid
    return hi_bw

assert abs(crossover(1_000_000) - be) / be < 0.001  # RTT amortized away
assert crossover(1_024) > be                        # RTT penalizes short ctx

# Edge cases: empty prefix must never be fetched; MHA multiplies the bill.
assert prefix_fetch_s(0, KVPT, NET_DC, 0.02) > reprefill_s(0, PREFILL_TPS)
assert kv_bytes_per_token(80, 64, 128, 2) == 8 * KVPT   # MHA vs GQA(8)

print(f"KV/token Llama-3.1-70B FP16 : {KVPT/1024:.0f} KiB")
print(f"decode, KV local (4 hops)   : {local*1e3:.1f} ms/token")
print(f"decode, stream @100 Gb/s    : {stream_dc*1e3:.0f} ms/token")
print(f"decode, stream @1 Gb/s      : {stream_wan:.1f} s/token")
print(f"decode, ship-Q @20 ms RTT   : {ship_q:.1f} s/token")
print(f"prefix-fetch break-even     : {be*8/1e9:.1f} Gb/s "
      f"({be_cachegen*8/1e9:.1f} Gb/s at 3.5x compression)")
print("all assertions passed")

Executed output:

KV/token Llama-3.1-70B FP16 : 320 KiB
decode, KV local (4 hops)   : 80.8 ms/token
decode, stream @100 Gb/s    : 216 ms/token
decode, stream @1 Gb/s      : 21.5 s/token
decode, ship-Q @20 ms RTT   : 1.6 s/token
prefix-fetch break-even     : 13.1 Gb/s (3.7 Gb/s at 3.5x compression)
all assertions passed

Reading the results:

• The per-token KV volume comes from the same sizing rule the NIXL page uses: 2 x n_layers x n_kv_heads x head_dim x bytes_per_element. GQA already shrinks it 8x versus full MHA (final assert); MLA and FP8/FP4 KV shrink it further, moving every boundary in this model proportionally.
• The 3.35 TB/s HBM figure is the NVIDIA H100 SXM specification, the same constant the KV speedup page derives from. The 5,000 tok/s prefill rate and the 20 ms RTT are modeling assumptions chosen at 70B-node order of magnitude; the formulas, not the constants, are the deliverable, and substituting your measured prefill rate and RTT is the intended use.
• The exact break-even (break_even_bw_exact) deliberately raises AssertionError when the prefix is so short that recompute beats one RTT. That is a designed fail-closed: below rtt x prefill_tps tokens there is no bandwidth at which fetching wins, and a scheduler built on this model should recompute without consulting the store.
• The empirical cross-check for the "bandwidth is not the binding term" claim is Petals' published two-way sweep: a 10x bandwidth cut costs 17.6 ms/step while 95 ms of added RTT costs 210.6 ms/step⁵. The slope, 210.6/95, recovers about 2.2 serialized round trips per token for a 3-server chain, exactly what decode = compute + hops x rtt predicts. Any proposal that adds round trips per token to buy bandwidth, which is precisely what centralizing decode-time KV does, is trading the cheap term for the expensive one.

How to use it

The reference deployment for a dedicated central store is vLLM plus an LMCache server. The following is the upstream example (examples/disaggregated/lmcache/kv_cache_sharing_lmcache_v1.py, fetched from vllm-project/vllm main on 2026-07-15), which launches one storing instance, one retrieving instance, and the server they share; it is a reference template requiring two GPUs and lmcache installed, not something executed in this KB's environment. The configuration surface is small and worth reading verbatim:

# LMCache-related environment variables (upstream example, quoted verbatim)
port = 8100
os.environ["LMCACHE_CHUNK_SIZE"] = "256"          # tokens per transfer chunk
os.environ["LMCACHE_LOCAL_CPU"] = "False"         # bypass local CPU tier
os.environ["LMCACHE_MAX_LOCAL_CPU_SIZE"] = "5.0"  # GB, if the tier is on
os.environ["LMCACHE_REMOTE_URL"] = f"lm://localhost:{port}"
os.environ["LMCACHE_REMOTE_SERDE"] = "naive"      # raw bytes, no compression

ktc = KVTransferConfig(kv_connector="LMCacheConnectorV1", kv_role="kv_both")
llm = LLM(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    kv_transfer_config=ktc,
    max_model_len=8000,
    gpu_memory_utilization=0.8,
    enforce_eager=True,
)
# server: python -m lmcache.v1.server localhost 8100

Both instances use kv_role="kv_both" (each may store and retrieve); the store-then-retrieve ordering in the example is enforced by the driver script, not the roles. Note that the LMCache documentation now marks this in-process connector mode as deprecated in favor of MP (multi-process) mode, where a standalone lmcache server (ZMQ service, LMCacheMPConnector) holds KV shared by one or more vLLM instances; the current recipe and configuration surface are on the Prefill-as-a-Service page. The NIXL connector covers the disaggregated prefill-to-decode handoff; see disaggregated inference for that recipe and KV cache transfer (NIXL) for the transport itself.

Sizing the store is the same arithmetic as the model above: bytes = tokens_cached x kv_bytes_per_token. A million cached tokens of Llama-3.1-70B FP16 KV is about 312 GiB, which is why Mooncake reaches for the cluster's DRAM and SSD rather than HBM¹.

How to develop with it

A pre-flight fetch gate and a canonical, layout-scoped cache key, executed

The break-even model above tells you where the line is; a client still needs code that checks which side of it a specific fetch sits on before touching the network, and a key scheme that cannot silently serve one model's KV to another. Both are small enough to be worth executing directly rather than describing in prose:

"""Client-side fetch-vs-recompute gate and cache-key scheme for a centralized
KV store. Builds directly on this page's own prefix_fetch_s/reprefill_s
break-even model; executed and asserted independently.
Run: python3 fetch_gate.py
"""
import hashlib
import json


def should_fetch(ctx: int, kvpt: int, net_bw: float, rtt: float,
                 prefill_tps: float, compression: float = 1.0) -> bool:
    """Client-side gate: only issue the fetch if it beats local recompute at
    the measured, not assumed, effective bandwidth. This is the 'fail
    toward recompute' rule as an admission check the client runs before
    touching the network, not after a failed fetch."""
    fetch = rtt + ctx * kvpt / (net_bw * compression)
    recompute = ctx / prefill_tps
    return fetch < recompute


def cache_key(token_ids: list[int], *, model_name: str, model_revision: str,
              tokenizer_revision: str, kv_dtype: str, attention_layout: str,
              world_size: int, kv_rank: int, object_group: int,
              chunk_size: int, cache_salt: str) -> str:
    """Hash a canonical representation of every cache-compatibility field."""
    payload = {
        "schema": 1,
        "token_ids": token_ids,
        "model_name": model_name,
        "model_revision": model_revision,
        "tokenizer_revision": tokenizer_revision,
        "kv_dtype": kv_dtype,
        "attention_layout": attention_layout,
        "world_size": world_size,
        "kv_rank": kv_rank,
        "object_group": object_group,
        "chunk_size": chunk_size,
        "cache_salt": cache_salt,
    }
    encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(encoded).hexdigest()


if __name__ == "__main__":
    KVPT = 327_680           # Llama-3.1-70B FP16 GQA, from the break-even model above
    PREFILL_TPS = 5_000
    NET_DC = 12.5e9          # 100 Gb/s in B/s
    NET_WAN = 0.125e9        # 1 Gb/s in B/s
    NET_MID = 0.625e9        # 5 Gb/s in B/s -- between the compressed and raw break-even
    CTX = 8_192

    cases = [
        ("datacenter fabric, 8K ctx, no compression", CTX, NET_DC, 0.001, 1.0, True),
        ("WAN link (1 Gb/s), 8K ctx, no compression", CTX, NET_WAN, 0.020, 1.0, False),
        ("5 Gb/s link, 8K ctx, no compression", CTX, NET_MID, 0.020, 1.0, False),
        ("5 Gb/s link, 8K ctx, 3.5x compression: recovers", CTX, NET_MID, 0.020, 3.5, True),
    ]
    print(f"{'case':<48} {'fetch_s':>9} {'recompute_s':>12} {'should_fetch':>13}")
    for label, ctx, bw, rtt, comp, expected in cases:
        gate = should_fetch(ctx, KVPT, bw, rtt, PREFILL_TPS, comp)
        fetch_s = rtt + ctx * KVPT / (bw * comp)
        recompute_s = ctx / PREFILL_TPS
        print(f"{label:<48} {fetch_s:>9.4f} {recompute_s:>12.4f} {str(gate):>13}")
        assert gate == expected, (label, gate, expected)

    # Cache keys cover every compatibility and tenant-isolation dimension.
    toks = [1, 2, 3, 4, 5]
    base = {
        "model_name": "llama-3.1-70b",
        "model_revision": "rev-a",
        "tokenizer_revision": "tok-a",
        "kv_dtype": "fp16",
        "attention_layout": "gqa-8kv-128d-80l",
        "world_size": 2,
        "kv_rank": 0,
        "object_group": 0,
        "chunk_size": 256,
        "cache_salt": "tenant-7",
    }
    k1 = cache_key(toks, **base)
    assert k1 == cache_key(toks, **base), "identical inputs must be deterministic"
    changes = {
        "model_revision": "rev-b",
        "tokenizer_revision": "tok-b",
        "kv_dtype": "fp8",
        "attention_layout": "mla-512d-61l",
        "world_size": 4,
        "kv_rank": 1,
        "object_group": 2,
        "chunk_size": 512,
        "cache_salt": "tenant-8",
    }
    for field, changed_value in changes.items():
        changed = dict(base)
        changed[field] = changed_value
        assert k1 != cache_key(toks, **changed), f"{field} was not key-scoped"

    # Canonical JSON avoids delimiter ambiguity in model and revision names.
    ambiguous_a = cache_key(toks, **{**base, "model_name": "a", "model_revision": "b:c"})
    ambiguous_b = cache_key(toks, **{**base, "model_name": "a:b", "model_revision": "c"})

    print("\nALL ASSERTIONS PASSED")

Executed output:

case                                               fetch_s  recompute_s  should_fetch
datacenter fabric, 8K ctx, no compression           0.2157       1.6384          True
WAN link (1 Gb/s), 8K ctx, no compression          21.4948       1.6384         False
5 Gb/s link, 8K ctx, no compression                 4.3150       1.6384         False
5 Gb/s link, 8K ctx, 3.5x compression: recovers     1.2471       1.6384          True

ALL ASSERTIONS PASSED

The 5 Gb/s row sits between the uncompressed and CacheGen-compressed network-only boundaries in the model (13.1 Gb/s and 3.7 Gb/s), so compression changes the result under those assumptions. A production gate should use measured end-to-end fetch latency, including storage reads, queueing, serialization, and codec time, rather than infer the decision from link rate alone.

should_fetch and vLLM's kv_load_failure_policy operate at different stages. The client gate avoids issuing a predictably slow fetch. The NIXL connector's current default is fail when an admitted load fails; recompute preserves availability but can inject a full prefill into the request and cause latency jitter. Choose the failure policy from the service's error-rate and tail-latency objectives, then alert on both failed loads and recomputations.⁷

• Do not let speculative decoding rollback and cache writes race. If decode nodes write accepted-token KV back to a shared store while a drafter runs, a rejected draft must roll the store back too, the same position-dependent-state rule speculative decoding already imposes locally.

How to maintain it

• Watch hit rate and fetch latency as one metric pair. The store pays only when hit_rate x (reprefill_s - fetch_s) exceeds its operating cost. A high-hit-rate store whose fetch latency has drifted above re-prefill time (fabric degradation, TCP fallback) is actively making the fleet slower while every dashboard shows green hits.
• Alert on transfer failures, not just misses. The NIXL connector exports failed-transfer and expired-request counters; a rising rate is the silent-degradation signature of an RDMA path that fell back to TCP (KV cache transfer).
• Eviction should track prefix stability, not recency alone. System prompts and shared corpus documents are stable and high-fanout; per-user conversation tails are neither. Weighted eviction that protects high-fanout prefixes preserves most of the value at a fraction of the capacity; see KV cache management for the block-hash plumbing this rides on.
• Invalidate on every model or dtype rollout. A weight update makes the entire store stale at once. Treat store flush as a step in the model-rollout runbook, and version keys so a partial flush cannot mix generations.

How to run it in production

• Placement first: same fabric as the decode pool. The store belongs within RDMA reach of its consumers, same rail or rack where possible, exactly the placement rule disaggregated inference sets for prefill/decode pools. A 100 Gb/s link with microsecond-scale fabric latency clears this example's network-only boundary, but storage and serialization latency can still make short or cold-prefix fetches slower than recomputation.
• Across regions or a WAN, restrict it to TTFT-shaped uses and measure them. Session resume, churn recovery in decentralized pools, and cold-start of a new replica are once-per-session candidates. Compression lowers transferred bytes but does not remove storage, codec, queueing, or RTT costs; admit a fetch only from measured end-to-end latency. Steady-state decode still runs on node-local KV.
• Split prefill and decode across non-colocated sites only when profiling supports it. Dense-attention KV egress generally makes a WAN handoff unattractive. For hybrid-attention or MLA models, Moonshot's PrfaaS paper evaluates a 1T-model case study with a threshold near 19.4K uncached tokens and reports model-derived improvements of 54% in throughput and 64% in P90 TTFT, with about 13 Gb/s average egress on a modeled 100 Gb/s link⁶. These results come from an analytical model populated by measured profiles, not an end-to-end deployed two-datacenter experiment. Re-profile the model, hardware, workload, and transfer stack before adopting the topology.
• Capacity-plan with the formula, then measure. kv_bytes_per_token for your exact model (GQA/MLA and KV dtype change it by up to an order of magnitude) times expected cached tokens, plus headroom for the eviction policy to work with. The KV cache OOM runbook applies unchanged when the "GPU" running out is a DRAM pool.
• Load-test the miss storm. The worst production day is a store restart: every request simultaneously becomes a miss and a full re-prefill. Admission control (QoS and admission control) must be sized for zero-hit-rate throughput, or the store's failure takes the fleet's TTFT SLO with it.

Failure modes

• The store quietly becomes a decode dependency. Someone routes per-token attention reads (a paged-KV tier fault, a misconfigured offload threshold) through the remote store and per-token latency jumps by orders of magnitude. The dashed edge in the architecture diagram is a bug signature, not an option: any per-token network read of KV history is a sev, and the 216 ms/token figure above is what it looks like on a 100 Gb/s fabric.
• Fetch slower than recompute. Below break-even bandwidth (congestion, TCP fallback, compression disabled) every "cache hit" is a slowdown. Guard with a latency budget: if fetch estimate exceeds ctx / prefill_tps, recompute and record the event.
• Stale or cross-model KV served after a rollout. Confident, fluent, wrong output with no error anywhere. Only key versioning and rollout-time invalidation prevent it.
• Hotspot inversion. Prefix-aware routing concentrates hot prefixes on few instances and degrades balance; this is the exact failure BanaServe's global store exists to remove⁴. If you keep cache-aware routing alongside a central store, you are paying for the problem and its solution simultaneously.
• Shared-store blast radius across tenants. Poisoned or probed cache entries cross tenant boundaries in a pooled store; tenant cache isolation covers the partitioning and the timing-side-channel argument.
• Expecting decode speedup and measuring none. Not a malfunction; the architecture working as designed. The store improves TTFT and goodput via reuse. The per-token lever over slow links remains speculative decoding, which commits several tokens per round trip and attacks the term that actually binds.

References

• Mooncake: R. Qin et al., "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving," arXiv 2407.00079 (FAST'25). https://arxiv.org/abs/2407.00079
• CacheGen: Y. Liu et al., "CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving," arXiv 2310.07240 (SIGCOMM'24). https://arxiv.org/abs/2310.07240
• Infinite-LLM: B. Lin et al., "Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache," arXiv 2401.02669. https://arxiv.org/abs/2401.02669
• BanaServe: "BanaServe: Dynamic Resource Rebalancing for Disaggregated LLM Serving," arXiv 2510.13223. https://arxiv.org/abs/2510.13223
• PrfaaS: R. Qin et al., "Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter," arXiv 2604.15039 (preprint). https://arxiv.org/abs/2604.15039
• Petals: A. Borzunov et al., "Distributed Inference and Fine-tuning of Large Language Models Over The Internet," arXiv 2312.08361 (Table 3: bandwidth vs RTT sweep). https://arxiv.org/abs/2312.08361
• Megatron-LM: D. Narayanan et al., "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM," arXiv 2104.04473 (section 3.2, pipeline vs tensor parallel communication volumes). https://arxiv.org/abs/2104.04473
• vLLM remote KV sharing example (LMCache server): https://github.com/vllm-project/vllm/blob/main/examples/disaggregated/lmcache/kv_cache_sharing_lmcache_v1.py
• LMCache: https://github.com/LMCache/LMCache
• NVIDIA H100 specifications (3.35 TB/s HBM3, SXM): https://www.nvidia.com/en-us/data-center/h100/

Related: Disaggregated Inference · KV Cache Transfer (NIXL) · KV Cache Management · KV Cache Fundamentals · How Much Faster Is Inference with KV Cache · Non-Colocated Inference: Which Pattern? · Cross-WAN Model-Parallel Inference · Mesh LLM & Skippy Stage Splits · P2P Transport for Decentralized Inference · Region-Replica Routing · Speculative Decoding · Disaggregation Rate Matching · Tenant Cache Isolation · Inference Parallelism Strategies · Glossary

---

1. arXiv 2407.00079, abstract: Mooncake "leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache"; "up to a 525% increase in throughput in certain simulated scenarios"; under Kimi's real workloads it enables the system to "handle 75% more requests." Verified against the arXiv abstract on 2026-07-15. ↩↩↩

2. arXiv 2310.07240, abstract: CacheGen achieves a 3.5-4.3x reduction in KV cache size and a 3.2-3.7x reduction in total context-loading delay, adapting compression level to available bandwidth. Verified against the arXiv abstract on 2026-07-15. ↩↩

3. arXiv 2401.02669, abstract: Infinite-LLM "disaggregates attention layers from an LLM's inference process," manages KV cache via "a pooled GPU memory strategy across a cluster," evaluated on a 32-A100 cluster with 1.35-3.4x throughput improvement over state-of-the-art at the time. Verified against the arXiv abstract on 2026-07-15. ↩↩

4. arXiv 2510.13223, abstract: layer-level weight migration, attention-level KV cache migration, and Global KV Cache Store sharing let routers schedule "unconstrained by cache placement"; 1.2x-3.9x higher throughput versus vLLM and 1.1x-2.8x versus DistServe. ↩↩↩

5. Derived from arXiv 2312.08361, Table 3 (3xA100, BLOOM-176B; the table reports steps/s, not ms/step): reducing bandwidth from 1 Gbit/s to 100 Mbit/s at fixed RTT (1.71 to 1.66 steps/s) adds 17.6 ms/step; adding 95 ms RTT at fixed bandwidth (1.66 to 1.23 steps/s) adds 210.6 ms/step. The 210.6/95 slope implies ~2.2 serialized round trips per token for the 3-server chain measured. ↩↩

6. arXiv 2604.15039v2 (preprint, Moonshot AI and Tsinghua; same first author as Mooncake), "Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter." The paper's 32K measurements show about 13x lower per-instance KV throughput for MiMo-V2-Flash than MiniMax-M2.5 (4.66 versus 59.93 Gbps) and about 4x for Qwen3.5 than Qwen3 (8.25 versus 33.35 Gbps). Its separate roughly 36x Ring-2.5-1T figure is an estimated memory saving: about 4.5x from MLA compression multiplied by about 8x from the 7:1 linear-to-full-attention layer ratio. The PrfaaS-PD case study reports 3.24 versus 2.11 req/s (+54%), 64% lower P90 TTFT for long-context requests, a roughly 19.4K-token threshold, and about 13 Gbps average egress on a 100 Gbps link. All case-study throughput and bandwidth results are derived by feeding measured profiling data into the paper's analytical model; they are not measurements of a deployed cross-datacenter system. Verified against arXiv v2 on 2026-07-16; vendor-evaluated and not independently reproduced. ↩↩↩

7. vLLM docs/features/nixl_connector_usage.md and vllm/config/kv_transfer.py at commit 9182e8697133e95f6e4ed236628a344b12e1c702: kv_load_failure_policy defaults to fail; the documentation warns that recompute runs prefill work on a decode-optimized instance, adds jitter, and increases other requests' tail latency. https://github.com/vllm-project/vllm/blob/9182e8697133e95f6e4ed236628a344b12e1c702/docs/features/nixl_connector_usage.md#failure-handling ↩

8. arXiv 2104.04473, section 3.2: pipeline parallelism communicates bsh per stage boundary per microbatch while tensor parallelism communicates 8bsh(t-1)/t per layer per device with blocking collectives, which is the layers-vs-stages round-trip asymmetry the query-shipping variant inherits. ↩