Markdown

Centralized KV cache placement: when a shared store pays and when it cannot¶

Scope: where the KV cache should live in distributed and decentralized LLM inference. Covers the shared (centralized) KV store pattern behind Mooncake, LMCache, and BanaServe, the two dead-end variants (per-token KV streaming and per-token attention offload), and an executed break-even model that tells you which side of the line a given deployment sits on. Complements disaggregated inference (the intra-datacenter pattern that uses a KV handoff) and cross-WAN model-parallel inference (the decentralized pattern that never moves KV at all).

Verdict up front: a centralized KV store pays exactly when the network touch is once per request, and never when it is once per token. The break-even is a one-line formula (net_bw > kv_bytes_per_token x prefill_tok_s / compression, roughly 13 Gb/s for Llama-3.1-70B at FP16, executed below). Per-token variants lose by 2 to 4 orders of magnitude regardless of how good the central node's bandwidth is, because decode is latency-bound, not bandwidth-bound.

What it is¶

A centralized KV cache is a store that holds KV blocks outside the GPU that will consume them: a dedicated cache server (LMCache server mode), a pool built from the cluster's idle DRAM and SSD (Mooncake¹), or a global store that any prefill or decode instance can read (BanaServe⁴). Serving instances fetch KV blocks from it instead of recomputing them with a prefill pass.

The design space has three interaction patterns, and they are not variations of one idea; they live on different timescales and have opposite economics:

Once-per-request prefix fetch. A request arrives whose prefix (system prompt, shared document, earlier turns of the conversation) already has KV in the store. The serving instance pulls those blocks once, then decodes with a fully local cache. This is Mooncake, LMCache, the vLLM disaggregated prefill handoff over NIXL, and CacheGen's compressed streaming². This pattern works, and the executed model below gives its exact break-even.
Per-token KV streaming. Decode runs on a node that does not hold the KV; every generated token pulls the attention history over the network. No shipping system does this, for the reason quantified below: at 8k context on a 70B model it needs 2.6 GB of reads per token.
Per-token attention offload (ship the query to the KV). The KV stays central and each layer's query is sent to it; attention is computed where the KV lives. Infinite-LLM's DistAttention³ is the real system in this direction, and it is instructive that it runs inside one 32-GPU A100 cluster on a datacenter fabric: the cost is one blocking round trip per layer per token, which only microsecond-RTT networks can absorb.

The one-sentence summary of this page: pattern 1 is an architecture, patterns 2 and 3 are RTT multipliers wearing an architecture's clothes.

Why use it¶

Prefix deduplication across replicas. Fleet-wide, most cache-worthy bytes are shared prefixes (system prompts, RAG documents, multi-turn history). A central store lets any replica serve a conversation without having previously seen it, so the router can balance purely on load. BanaServe makes this decoupling explicit: its Global KV Cache Store lets routing ignore cache placement and still reports 1.2x to 3.9x higher throughput than vLLM and 1.1x to 2.8x over DistServe⁴.
Capacity beyond HBM. Mooncake builds its pool from "the underutilized CPU, DRAM, and SSD resources of the GPU cluster", reporting up to a 525% throughput increase in simulated long-context scenarios and 75% more requests handled under Kimi's real workload¹. The win is reuse that HBM alone could not hold, not faster decode.
Churn survival in decentralized serving. In a cross-WAN pipeline, a healed stage currently pays a full re-prefill of the accumulated history. A central session store turns that into one fetch, which is the TTFT-shaped benefit a high-bandwidth central node genuinely can deliver to decentralized decode nodes.
Slow-network reuse becomes viable with compression. CacheGen compresses KV 3.5x to 4.3x and streams it, cutting total context-loading delay 3.2x to 3.7x versus fetching raw KV or re-prefilling, with adaptive quality under varying bandwidth². Compression moves the break-even bandwidth down by the same factor, which is what makes the pattern reach beyond the datacenter.

When to use it (and when not)¶

Use a centralized KV store when all of these hold:

The workload has real prefix reuse across instances or across time (multi-turn chat, shared system prompts, RAG over a common corpus). A store nobody hits twice is pure overhead.
The fetch path clears the break-even: effective bandwidth above kv_bytes_per_token x prefill_tok_s / compression (executed below; about 13 Gb/s uncompressed for Llama-3.1-70B FP16, about 3.7 Gb/s with CacheGen-class compression). Below that line, recomputing the prefix locally is faster than fetching it.
The store sits on the same fast fabric as the decode pool, or the use is explicitly TTFT-shaped (session resume, churn recovery) where a WAN fetch still beats a WAN re-prefill.

Do not use it when:

You expect it to speed up steady-state decode. It cannot. Decode reads the whole KV history every token; only local HBM sustains that. The Petals measurement is the cleanest external evidence that the network term that hurts decode is RTT, not bandwidth: cutting bandwidth 1 Gbit/s to 100 Mbit/s costs 17.6 ms/step, while adding 95 ms of RTT costs 210.6 ms/step, twelve times more⁵.
The fetch would cross a WAN as part of a prefill/decode split, for a dense-attention model. Disaggregated inference pins prefill and decode pools to fast-interconnect reach because dense-model KV egress saturates any inter-site link; Moonshot measures 3.8 Tbps of egress for a dense MiniMax-M2.5 deployment (2.1 Tbps for dense Qwen3-235B) at 32K context on 512 H200s⁶. The rule is conditional on KV volume, not architectural law: for hybrid-attention/MLA models that cut per-token KV 13x to 36x, the same team (the Mooncake authors) shows a deliberate cross-datacenter prefill-to-decode handoff working over roughly 100 Gb/s commodity Ethernet (PrfaaS⁶). Run your model's numbers through the break-even before applying either version of the rule.
Prefixes are short. Below rtt x prefill_tok_s tokens (100 tokens at 20 ms RTT and 5,000 tok/s prefill), one round trip already costs more than recomputing; the exact-boundary function in the model below fails closed on this case.
Tenancy or integrity boundaries would be crossed. A shared store is a shared fate domain; see tenant cache isolation before pooling KV across tenants.

Architecture¶

flowchart LR
  subgraph Store["Central KV store (DRAM/SSD pool or LMCache server)"]
    S["Prefix and session KV blocks<br/>keyed by token-sequence hash"]
  end
  P["Prefill instance"] -->|"store once<br/>after prefill"| S
  S -->|"fetch once per request<br/>(the pattern that pays)"| D1["Decode instance A<br/>local HBM KV, local attention"]
  S -->|"fetch once on resume<br/>or node churn"| D2["Decode instance B<br/>(different replica or region)"]
  D1 -.->|"never per token:<br/>2.6 GB/token at 8k ctx"| S

The solid edges are the whole architecture: KV enters the store once after prefill and leaves it once per request (or once per session resume). Decode attention always runs against HBM-local blocks. The dashed edge is the variant this page exists to rule out; the moment KV crosses the network per token, the design has silently changed from a cache into a remote memory bus, and the numbers below apply.

The three placements compared:

Placement	Network cost per token	Binding constraint	Verdict
KV local to decode (baseline, incl. pipeline splits)	one activation vector per stage hop (4-8 KiB)	`stages x RTT`	The production default; what Mesh LLM and Petals do
Central store, once-per-request fetch	zero (fetch amortized over the whole request)	fetch bandwidth vs re-prefill rate	Pays above break-even bandwidth; this page's pattern
Central KV, per-token streaming	`context x kv_bytes_per_token`	bandwidth, catastrophically	250x worse than HBM even at 100 Gb/s; never ship
Central KV, per-token query shipping	KiB-sized, but `n_layers` blocking RTTs	latency; bandwidth does not appear in the cost	Datacenter-fabric only (Infinite-LLM³); 20x worse than a 4-stage pipeline at any RTT

The query-shipping row is the same mathematics that makes tensor parallelism unshippable over slow links: Megatron-LM's communication analysis (section 3.2) has pipeline parallelism moving bsh per stage boundary while tensor parallelism moves per-layer volumes with blocking synchronization, so the round-trip count scales with layers, not stages⁷.

How it works (validated decision model)¶

The whole decision reduces to four cost functions and one break-even. This model was executed as written and every assert passed; the printed output follows the listing.

"""Decide when a centralized KV cache helps and when it cannot.

Models the three KV placements for distributed decode plus the one-shot
prefix-fetch case, then asserts the decision boundaries hold.
"""


def kv_bytes_per_token(n_layers: int, n_kv_heads: int, head_dim: int,
                       bytes_per_el: int) -> int:
    """2x covers keys and values (same formula as the NIXL sizing rule)."""
    return 2 * n_layers * n_kv_heads * head_dim * bytes_per_el


def decode_local_s(ctx: int, kvpt: int, hbm_bw: float, stages: int,
                   rtt: float) -> float:
    """KV local to each stage: attention reads history from HBM, the only
    network cost is one activation hop per pipeline stage."""
    return ctx * kvpt / hbm_bw + stages * rtt


def decode_stream_central_s(ctx: int, kvpt: int, net_bw: float,
                            rtt: float) -> float:
    """Variant A: decode node pulls the full KV history over the network
    for every generated token."""
    return ctx * kvpt / net_bw + rtt


def decode_ship_q_s(n_layers: int, rtt: float) -> float:
    """Variant B: queries shipped to the central KV node, one blocking
    round trip per layer (layer l+1 depends on layer l's output).
    Note: no bandwidth parameter exists; payloads are KiB-sized."""
    return n_layers * rtt


def prefix_fetch_s(ctx: int, kvpt: int, net_bw: float, rtt: float,
                   compression: float = 1.0) -> float:
    """One-shot fetch of a cached prefix from a central store."""
    return rtt + ctx * kvpt / (net_bw * compression)


def reprefill_s(ctx: int, prefill_tps: float) -> float:
    """Recompute the prefix locally instead of fetching it."""
    return ctx / prefill_tps


def break_even_bw(kvpt: int, prefill_tps: float,
                  compression: float = 1.0) -> float:
    """Bandwidth above which fetching a cached prefix beats recomputing it
    (RTT ignored; exact in the large-context limit)."""
    return kvpt * prefill_tps / compression


def break_even_bw_exact(ctx: int, kvpt: int, prefill_tps: float, rtt: float,
                        compression: float = 1.0) -> float:
    """RTT-aware break-even; only defined when recompute takes longer than
    one RTT (below that, fetching can never win at any bandwidth)."""
    budget = ctx / prefill_tps - rtt
    assert budget > 0, "recompute is faster than one RTT; never fetch"
    return ctx * kvpt / (budget * compression)


# --- Llama-3.1-70B, FP16 KV, GQA: 80 layers, 8 KV heads, head_dim 128 ---
KVPT = kv_bytes_per_token(80, 8, 128, 2)
assert KVPT == 327_680, KVPT                       # 320 KiB per token

HBM = 3.35e12          # H100 SXM HBM3, B/s
NET_DC = 12.5e9        # 100 Gb/s in B/s
NET_WAN = 0.125e9      # 1 Gb/s in B/s
CTX = 8192

# Variant A: streaming the history per token loses by orders of magnitude.
local = decode_local_s(CTX, KVPT, HBM, stages=4, rtt=0.020)
stream_dc = decode_stream_central_s(CTX, KVPT, NET_DC, rtt=0.001)
stream_wan = decode_stream_central_s(CTX, KVPT, NET_WAN, rtt=0.020)
assert 0.080 < local < 0.082                       # 4 WAN hops dominate
assert stream_dc > 0.21                            # ~215 ms/token at 100 Gb/s
assert stream_wan > 21.0                           # ~21 s/token at 1 Gb/s
assert stream_dc / (CTX * KVPT / HBM) > 250        # vs 0.8 ms HBM read

# Variant B: latency-bound, bandwidth cannot appear in the cost at all.
ship_q = decode_ship_q_s(n_layers=80, rtt=0.020)
pipeline_net = 4 * 0.020
assert ship_q == 1.6                               # 80 RTTs per token
assert ship_q / pipeline_net == 20                 # == n_layers / stages

# One-shot prefix fetch: the case where central KV genuinely wins.
PREFILL_TPS = 5_000
be = break_even_bw(KVPT, PREFILL_TPS)
assert abs(be - 1.6384e9) < 1e3                    # ~13.1 Gb/s uncompressed
be_cachegen = break_even_bw(KVPT, PREFILL_TPS, compression=3.5)
assert be_cachegen < 0.47e9                        # ~3.7 Gb/s compressed

# The boundary is real: fetch wins above it, loses below it.
for ctx in (1_024, 8_192, 131_072):
    be_x = break_even_bw_exact(ctx, KVPT, PREFILL_TPS, rtt=0.02)
    hi, lo = be_x * 1.001, be_x * 0.999
    assert prefix_fetch_s(ctx, KVPT, hi, 0.02) < reprefill_s(ctx, PREFILL_TPS)
    assert prefix_fetch_s(ctx, KVPT, lo, 0.02) > reprefill_s(ctx, PREFILL_TPS)
    assert be_x > be                       # RTT always raises the bar

# Bisection crossover converges to the closed form as context grows.
def crossover(ctx: int) -> float:
    lo_bw, hi_bw = 1e6, 1e12
    for _ in range(200):
        mid = (lo_bw + hi_bw) / 2
        if prefix_fetch_s(ctx, KVPT, mid, 0.02) < reprefill_s(ctx, PREFILL_TPS):
            hi_bw = mid
        else:
            lo_bw = mid
    return hi_bw

assert abs(crossover(1_000_000) - be) / be < 0.001  # RTT amortized away
assert crossover(1_024) > be                        # RTT penalizes short ctx

# Edge cases: empty prefix must never be fetched; MHA multiplies the bill.
assert prefix_fetch_s(0, KVPT, NET_DC, 0.02) > reprefill_s(0, PREFILL_TPS)
assert kv_bytes_per_token(80, 64, 128, 2) == 8 * KVPT   # MHA vs GQA(8)

print(f"KV/token Llama-3.1-70B FP16 : {KVPT/1024:.0f} KiB")
print(f"decode, KV local (4 hops)   : {local*1e3:.1f} ms/token")
print(f"decode, stream @100 Gb/s    : {stream_dc*1e3:.0f} ms/token")
print(f"decode, stream @1 Gb/s      : {stream_wan:.1f} s/token")
print(f"decode, ship-Q @20 ms RTT   : {ship_q:.1f} s/token")
print(f"prefix-fetch break-even     : {be*8/1e9:.1f} Gb/s "
      f"({be_cachegen*8/1e9:.1f} Gb/s at 3.5x compression)")
print("all assertions passed")

Executed output:

KV/token Llama-3.1-70B FP16 : 320 KiB
decode, KV local (4 hops)   : 80.8 ms/token
decode, stream @100 Gb/s    : 216 ms/token
decode, stream @1 Gb/s      : 21.5 s/token
decode, ship-Q @20 ms RTT   : 1.6 s/token
prefix-fetch break-even     : 13.1 Gb/s (3.7 Gb/s at 3.5x compression)
all assertions passed

Reading the results:

The per-token KV volume comes from the same sizing rule the NIXL page uses: 2 x n_layers x n_kv_heads x head_dim x bytes_per_element. GQA already shrinks it 8x versus full MHA (final assert); MLA and FP8/FP4 KV shrink it further, moving every boundary in this model proportionally.
The 3.35 TB/s HBM figure is the NVIDIA H100 SXM specification, the same constant the KV speedup page derives from. The 5,000 tok/s prefill rate and the 20 ms RTT are modeling assumptions chosen at 70B-node order of magnitude; the formulas, not the constants, are the deliverable, and substituting your measured prefill rate and RTT is the intended use.
The exact break-even (break_even_bw_exact) deliberately raises AssertionError when the prefix is so short that recompute beats one RTT. That is a designed fail-closed: below rtt x prefill_tps tokens there is no bandwidth at which fetching wins, and a scheduler built on this model should recompute without consulting the store.
The empirical cross-check for the "bandwidth is not the binding term" claim is Petals' published two-way sweep: a 10x bandwidth cut costs 17.6 ms/step while 95 ms of added RTT costs 210.6 ms/step⁵. The slope, 210.6/95, recovers about 2.2 serialized round trips per token for a 3-server chain, exactly what decode = compute + hops x rtt predicts. Any proposal that adds round trips per token to buy bandwidth, which is precisely what centralizing decode-time KV does, is trading the cheap term for the expensive one.

How to use it¶

The reference deployment for a dedicated central store is vLLM plus an LMCache server. The following is the upstream example (examples/disaggregated/lmcache/kv_cache_sharing_lmcache_v1.py, fetched from vllm-project/vllm main on 2026-07-15), which launches one storing instance, one retrieving instance, and the server they share; it is a reference template requiring two GPUs and lmcache installed, not something executed in this KB's environment. The configuration surface is small and worth reading verbatim:

# LMCache-related environment variables (upstream example, quoted verbatim)
port = 8100
os.environ["LMCACHE_CHUNK_SIZE"] = "256"          # tokens per transfer chunk
os.environ["LMCACHE_LOCAL_CPU"] = "False"         # bypass local CPU tier
os.environ["LMCACHE_MAX_LOCAL_CPU_SIZE"] = "5.0"  # GB, if the tier is on
os.environ["LMCACHE_REMOTE_URL"] = f"lm://localhost:{port}"
os.environ["LMCACHE_REMOTE_SERDE"] = "naive"      # raw bytes, no compression

ktc = KVTransferConfig(kv_connector="LMCacheConnectorV1", kv_role="kv_both")
llm = LLM(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    kv_transfer_config=ktc,
    max_model_len=8000,
    gpu_memory_utilization=0.8,
    enforce_eager=True,
)
# server: python -m lmcache.v1.server localhost 8100

Both instances use kv_role="kv_both" (each may store and retrieve); the store-then-retrieve ordering in the example is enforced by the driver script, not the roles. Note that the LMCache documentation now marks this in-process connector mode as deprecated in favor of MP (multi-process) mode, where a standalone lmcache server (ZMQ service, LMCacheMPConnector) holds KV shared by one or more vLLM instances; the current recipe and configuration surface are on the Prefill-as-a-Service page. The NIXL connector covers the disaggregated prefill-to-decode handoff; see disaggregated inference for that recipe and KV cache transfer (NIXL) for the transport itself.

Sizing the store is the same arithmetic as the model above: bytes = tokens_cached x kv_bytes_per_token. A million cached tokens of Llama-3.1-70B FP16 KV is about 312 GiB, which is why Mooncake reaches for the cluster's DRAM and SSD rather than HBM¹.

How to develop with it¶

Chunk size is an amortization knob, not a tuning afterthought. The upstream example pins LMCACHE_CHUNK_SIZE=256 against vLLM's default 16-token blocks; per-transfer fixed costs need the larger granularity to disappear into the payload. If you change it, re-measure fetch latency at your real block sizes.
Pick the serializer by which side of the break-even you sit on. naive (raw bytes) is right on an RDMA fabric where bandwidth is far above break-even. On constrained links, CacheGen-class compression buys back 3.5x to 4.3x of the bandwidth requirement at measured, near-negligible quality cost²; the model's compression parameter is exactly this factor.
Key by full token-sequence hash and treat the entry as model-specific. KV is only valid for the exact model weights, dtype, and attention layout that produced it; a store shared across a heterogeneous fleet must carry model and revision in the key or it will serve garbage that decodes confidently.
Fail toward recompute in client code. The vLLM NIXL connector exposes kv_load_failure_policy with fail and recompute; the recompute policy converts a missing or corrupt fetch into jitter instead of a failed request, which matches the fail-closed shape of break_even_bw_exact above.
Do not let speculative decoding rollback and cache writes race. If decode nodes write accepted-token KV back to a shared store while a drafter runs, a rejected draft must roll the store back too, the same position-dependent-state rule speculative decoding already imposes locally.

How to maintain it¶

Watch hit rate and fetch latency as one metric pair. The store pays only when hit_rate x (reprefill_s - fetch_s) exceeds its operating cost. A high-hit-rate store whose fetch latency has drifted above re-prefill time (fabric degradation, TCP fallback) is actively making the fleet slower while every dashboard shows green hits.
Alert on transfer failures, not just misses. The NIXL connector exports failed-transfer and expired-request counters; a rising rate is the silent-degradation signature of an RDMA path that fell back to TCP (KV cache transfer).
Eviction should track prefix stability, not recency alone. System prompts and shared corpus documents are stable and high-fanout; per-user conversation tails are neither. Weighted eviction that protects high-fanout prefixes preserves most of the value at a fraction of the capacity; see KV cache management for the block-hash plumbing this rides on.
Invalidate on every model or dtype rollout. A weight update makes the entire store stale at once. Treat store flush as a step in the model-rollout runbook, and version keys so a partial flush cannot mix generations.

How to run it in production¶

Placement first: same fabric as the decode pool. The store belongs within RDMA reach of its consumers, same rail or rack where possible, exactly the placement rule disaggregated inference sets for prefill/decode pools. At 100 Gb/s and microsecond RTTs the fetch is far above the 13 Gb/s break-even and the pattern is pure win for any reused prefix.
Across regions or a WAN, restrict it to TTFT-shaped uses. Session resume, churn recovery in decentralized pools, and cold-start of a new replica are once-per-session fetches where even a 3 to 4 Gb/s effective link (with compression) beats re-prefill. Steady-state decode still runs on node-local KV; nothing about the store changes the latency-bound decode economics.
Split prefill and decode across non-colocated sites only when the break-even arithmetic says so. For dense-attention models this reintroduces a WAN KV handoff on the request critical path, the documented anti-pattern in the non-colocated pattern chooser, and the rule stays "never." For hybrid-attention/MLA models the inequality can genuinely flip: Moonshot's PrfaaS routes long-context prefill (above a computed threshold near 19.4K uncached tokens) to a remote compute-dense H200 cluster and ships the finished KV once over inter-datacenter Ethernet, reporting 54% higher throughput and 64% lower P90 TTFT than a homogeneous baseline at 13 Gb/s average egress on a 100 Gb/s link⁶. The extra benefit is hardware arbitrage, compute-dense prefill silicon in one site and bandwidth-optimized decode silicon in another. This is a vendor-evaluated preprint; validate the KV-egress arithmetic for your own model before copying the topology.
Capacity-plan with the formula, then measure. kv_bytes_per_token for your exact model (GQA/MLA and KV dtype change it by up to an order of magnitude) times expected cached tokens, plus headroom for the eviction policy to work with. The KV cache OOM runbook applies unchanged when the "GPU" running out is a DRAM pool.
Load-test the miss storm. The worst production day is a store restart: every request simultaneously becomes a miss and a full re-prefill. Admission control (QoS and admission control) must be sized for zero-hit-rate throughput, or the store's failure takes the fleet's TTFT SLO with it.

Failure modes¶

The store quietly becomes a decode dependency. Someone routes per-token attention reads (a paged-KV tier fault, a misconfigured offload threshold) through the remote store and per-token latency jumps by orders of magnitude. The dashed edge in the architecture diagram is a bug signature, not an option: any per-token network read of KV history is a sev, and the 216 ms/token figure above is what it looks like on a 100 Gb/s fabric.
Fetch slower than recompute. Below break-even bandwidth (congestion, TCP fallback, compression disabled) every "cache hit" is a slowdown. Guard with a latency budget: if fetch estimate exceeds ctx / prefill_tps, recompute and record the event.
Stale or cross-model KV served after a rollout. Confident, fluent, wrong output with no error anywhere. Only key versioning and rollout-time invalidation prevent it.
Hotspot inversion. Prefix-aware routing concentrates hot prefixes on few instances and degrades balance; this is the exact failure BanaServe's global store exists to remove⁴. If you keep cache-aware routing alongside a central store, you are paying for the problem and its solution simultaneously.
Shared-store blast radius across tenants. Poisoned or probed cache entries cross tenant boundaries in a pooled store; tenant cache isolation covers the partitioning and the timing-side-channel argument.
Expecting decode speedup and measuring none. Not a malfunction; the architecture working as designed. The store improves TTFT and goodput via reuse. The per-token lever over slow links remains speculative decoding, which commits several tokens per round trip and attacks the term that actually binds.

References¶

Mooncake: R. Qin et al., "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving," arXiv 2407.00079 (FAST'25). https://arxiv.org/abs/2407.00079
CacheGen: Y. Liu et al., "CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving," arXiv 2310.07240 (SIGCOMM'24). https://arxiv.org/abs/2310.07240
Infinite-LLM: B. Lin et al., "Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache," arXiv 2401.02669. https://arxiv.org/abs/2401.02669
BanaServe: "BanaServe: Dynamic Resource Rebalancing for Disaggregated LLM Serving," arXiv 2510.13223. https://arxiv.org/abs/2510.13223
PrfaaS: R. Qin et al., "Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter," arXiv 2604.15039 (preprint). https://arxiv.org/abs/2604.15039
Petals: A. Borzunov et al., "Distributed Inference and Fine-tuning of Large Language Models Over The Internet," arXiv 2312.08361 (Table 3: bandwidth vs RTT sweep). https://arxiv.org/abs/2312.08361
Megatron-LM: D. Narayanan et al., "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM," arXiv 2104.04473 (section 3.2, pipeline vs tensor parallel communication volumes). https://arxiv.org/abs/2104.04473
vLLM remote KV sharing example (LMCache server): https://github.com/vllm-project/vllm/blob/main/examples/disaggregated/lmcache/kv_cache_sharing_lmcache_v1.py
LMCache: https://github.com/LMCache/LMCache
NVIDIA H100 specifications (3.35 TB/s HBM3, SXM): https://www.nvidia.com/en-us/data-center/h100/

arXiv 2407.00079, abstract: Mooncake "leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache"; "up to a 525% increase in throughput in certain simulated scenarios"; under Kimi's real workloads it enables the system to "handle 75% more requests." Verified against the arXiv abstract on 2026-07-15. ↩↩↩
arXiv 2310.07240, abstract: CacheGen achieves a 3.5-4.3x reduction in KV cache size and a 3.2-3.7x reduction in total context-loading delay, adapting compression level to available bandwidth. Verified against the arXiv abstract on 2026-07-15. ↩↩↩
arXiv 2401.02669, abstract: Infinite-LLM "disaggregates attention layers from an LLM's inference process," manages KV cache via "a pooled GPU memory strategy across a cluster," evaluated on a 32-A100 cluster with 1.35-3.4x throughput improvement over state-of-the-art at the time. Verified against the arXiv abstract on 2026-07-15. ↩↩
arXiv 2510.13223, abstract: layer-level weight migration, attention-level KV cache migration, and Global KV Cache Store sharing let routers schedule "unconstrained by cache placement"; 1.2x-3.9x higher throughput versus vLLM and 1.1x-2.8x versus DistServe. ↩↩↩
Derived from arXiv 2312.08361, Table 3 (3xA100, BLOOM-176B; the table reports steps/s, not ms/step): reducing bandwidth from 1 Gbit/s to 100 Mbit/s at fixed RTT (1.71 to 1.66 steps/s) adds 17.6 ms/step; adding 95 ms RTT at fixed bandwidth (1.66 to 1.23 steps/s) adds 210.6 ms/step. The 210.6/95 slope implies ~2.2 serialized round trips per token for the 3-server chain measured. ↩↩
arXiv 2604.15039 (preprint, Moonshot AI and Tsinghua; same first author as Mooncake), "Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter." For dense attention, "prefill emits too much state too quickly, so the network becomes the hard coupling between phases"; a dense deployment at 32K context on 512 H200s needs 3.8 Tbps of KV egress for MiniMax-M2.5 (2.1 Tbps for Qwen3-235B). Hybrid-attention/MLA models cut KV volume 13x (MiMo-V2-Flash vs MiniMax-M2.5: 4.66 vs 59.93 Gbps at 32K) to roughly 36x (Ring-2.5-1T: ~4.5x from MLA compression times ~8x from a 7:1 linear-to-full-attention layer ratio). The PrfaaS-PD case study on a 1T hybrid model reports 3.24 vs 2.11 req/s (+54%) against a homogeneous 96-H20 baseline, 64% lower P90 TTFT for long-context requests, a ~19.4K-token optimal routing threshold, and 13 Gbps average cross-datacenter egress on a 100 Gbps link, with decode always local to the delivered KV and zero per-token cross-site traffic. Verified against the arXiv HTML on 2026-07-15; vendor-evaluated, not independently reproduced. ↩↩↩
arXiv 2104.04473, section 3.2: pipeline parallelism communicates bsh per stage boundary per microbatch while tensor parallelism communicates 8bsh(t-1)/t per layer per device with blocking collectives, which is the layers-vs-stages round-trip asymmetry the query-shipping variant inherits. ↩