Markdown

Prefill-as-a-Service: cross-datacenter prefill, and how to build the KV plumbing with LMCache¶

Scope: the Prefill-as-a-Service (PrfaaS) serving paradigm from Moonshot AI and Tsinghua (arXiv 2604.15039): routing long uncached prefills to standalone compute-dense clusters and transferring the resulting KV cache over commodity Ethernet to local prefill/decode clusters for decode. Covers the throughput model that decides the routing threshold (reproduced and executed below from the paper's own published inputs), why it only works for hybrid-attention models, and the LMCache configuration surface (docs.lmcache.ai) that provides the open-source KV transfer and storage layer such a deployment runs on. Extends disaggregated inference beyond one fabric and applies the break-even discipline of centralized KV cache placement to a concrete architecture.

Verdict up front: PrfaaS is the once-per-request KV pattern taken to its logical end, prefill anywhere compute is cheap, decode where bandwidth is. It is worth deploying only when three gates all pass: the model is hybrid-attention/MLA (per-instance KV throughput in single-digit Gbps, not the ~60 Gbps of dense GQA), the workload is long-context with real prefix reuse, and a scheduler enforces a length threshold so short requests never touch the inter-datacenter link. The paper's headline (+54% throughput, 64% lower P90 TTFT) is a model-derived case study on vendor hardware, and the equal-cost gain is a more modest ~15%; the reproducible part is the arithmetic, which this page executes and confirms.

What it is¶

PrfaaS-PD is a two-tier serving topology. Local PD clusters run conventional prefill/decode disaggregation on bandwidth-optimized accelerators (H20-class) over intra-cluster RDMA and can serve any request end to end. PrfaaS clusters are standalone pools of compute-dense accelerators (H200-class, or prefill-specialized parts like Rubin CPX) that do exactly one thing: run long-context prefill for requests whose uncached length exceeds a routing threshold t, then push the finished KV cache over an inter-datacenter link (VPC peering or dedicated lines, ~100 Gb/s class) to a local PD cluster, which decodes. Decode never runs remote from its KV, and nothing crosses the inter-datacenter link per token; the transfer is once per request, overlapped with prefill via layer-wise pipelining.¹

The enabling condition is architectural, not systems-level. Only full-attention layers produce KV that grows with sequence length; linear-attention and sliding-window layers carry fixed-size recurrent state. Hybrid models interleave the two (Kimi Linear at 3:1 KDA:MLA, MiMo-V2-Flash at 5:1 SWA:GQA, Qwen3.5-397B at 3:1 GDN:GQA, Ring-2.5-1T at 7:1 Lightning:MLA), which collapses the per-instance "KV throughput" Phi_kv(l) = S_kv(l) / T_prefill(l), the rate at which one prefill instance emits KV bytes, from ~60 Gb/s (dense MiniMax-M2.5 at 32K) to 2.6-8.3 Gb/s across the hybrid models the paper benchmarks.² That 13x-class reduction is what moves the prefill-to-decode boundary from RDMA fabrics onto commodity Ethernet; centralized KV cache placement covers the same inequality in break-even form.

LMCache is the open-source layer closest to this architecture's storage and transfer subsystem: a standalone KV cache service for vLLM (and other engines) with tiered storage (CPU DRAM, NVMe via GPUDirect Storage, then S3/Redis/Valkey/Bigtable/Mooncake/NIXL backends), disaggregated-prefill transfer over NVLink/RDMA/TCP, CacheGen compression for constrained links, and CacheBlend for non-prefix reuse.⁴ What LMCache does not ship is PrfaaS's brain: the global KV manager, the length-threshold router, and the dual-timescale scheduler remain things you build (see the production section).

Why use it¶

Heterogeneous hardware without a shared fabric. Compute-dense and bandwidth-dense accelerators rarely live in the same RDMA domain, and forcing them into one fixes the prefill:decode hardware ratio forever. PrfaaS lets each phase scale independently on the silicon that suits it, which is where inference hardware is already heading (Rubin CPX for prefill; LPU-class parts for decode).¹
Throughput and TTFT, at the case-study numbers. Against a homogeneous 96-H20 baseline, the PrfaaS-PD configuration (32 H200 + 64 H20) reaches 3.24 vs 2.11 req/s (+54%) with mean TTFT halved (2.22 s vs 4.44 s) and P90 TTFT down 64% (3.51 s vs 9.73 s), because long requests stop queueing behind short ones for local prefill capacity. At equal hardware cost the paper puts the gain near 15%.³
Scheduling is most of the win over naive offload. Sending all prefill to the remote cluster and all decode local (no threshold, no balancing) yields only 1.16x; selective offloading plus the threshold recovers the remaining 32%.³ The mechanism, not the hardware, is the product.
The bandwidth bill is small when the gates pass. At the optimal threshold, 49.6% of requests offload, the offloaded mean uncached length is ~44K tokens, and average egress is ~13 Gb/s, 13% of one 100 Gb/s link.³ For scale: a 10,000-GPU prefill datacenter of Ring-2.5-1T-class models aggregates to ~1.8 Tb/s, within modern datacenter interconnect capacity, whereas one 512-GPU cluster of dense MiniMax-M2.5 alone would demand 3.8 Tb/s.²

When to use it (and when not)¶

Use it when all three gates pass:

Model gate: hybrid-attention or MLA architecture with measured Phi_kv in the single-digit Gb/s range at your operating lengths. Profile it exactly as the paper does (KV size and prefill latency at 1K/8K/32K/128K); dense GQA models fail this gate by an order of magnitude and stay inside one fabric (disaggregated inference).
Workload gate: a long-context, prefix-heavy request mix (agentic traffic; the case study uses a truncated log-normal with ~27K-token mean). If most requests are short, the threshold router sends almost nothing remote and the PrfaaS cluster idles.
Scheduler gate: you can route on uncached length (which requires global prefix-cache metadata), monitor egress congestion, and re-balance. Without this you get the naive-heterogeneous outcome (1.16x for heterogeneous hardware money).

Do not use it when:

The model is dense-attention. The 2.1-3.8 Tb/s egress arithmetic is prohibitive; this is exactly the dense-model "never split P/D across sites" rule that cross-WAN inference and centralized KV cache placement preserve.
You expect per-token benefits. Decode economics are untouched; decode runs local to delivered KV, and steady-state tokens/s is governed by the decode cluster alone (the latency-bound argument).
Requests are short or cache-hit-dominated end to end. An incremental prefill below the threshold never justifies a cross-datacenter trip; the local PD path already serves it optimally.
You cannot see prefix-cache placement globally. Routing on total rather than uncached length systematically over-offloads and burns the link on bytes the local cluster already had.

Architecture¶

flowchart LR
  C["Client"] --> R["Global scheduler:<br/>route on uncached length vs threshold t,<br/>cache affinity, egress congestion"]
  R -->|"uncached len <= t"| PDP["Local PD cluster (H20-class)<br/>PD-P prefill nodes"]
  R -->|"uncached len > t"| PF["PrfaaS cluster (H200-class)<br/>long-context prefill only"]
  PF -->|"KV cache, once per request,<br/>layer-wise pipelined, multi-conn TCP<br/>(~100 Gb/s inter-DC Ethernet)"| PDD["Local PD cluster<br/>PD-D decode nodes"]
  PDP -->|"KV via intra-cluster RDMA"| PDD
  PDD -->|"tokens"| C
  M["Global KV cache manager<br/>hybrid prefix cache pool"] -.-> R

Two storage details from the paper matter for anyone reimplementing this. First, hybrid models need a hybrid prefix cache pool: full-attention KV is block-level and supports partial prefix matching, but the recurrent state of linear/SWA layers is request-level, fixed-size, and reusable only on an exact length match, so the two live in separate KV groups over one shared block pool (built on vLLM's hybrid KV cache manager).¹ Second, blocks are split into prefix-cache blocks (must be fully populated before reuse) and transfer-cache blocks (tail KV of an in-flight request, discarded once the prefill-to-decode transfer completes), so transfer traffic never pollutes the reuse pool.¹

How it works (validated throughput model)¶

The scheduling decisions all fall out of one small model (the paper's section 3.4): stage throughputs Theta_prfaas = min(compute, egress) (eq. 3), Theta_pd-p (eq. 4), Theta_pd-d (eq. 5), the pipeline bound Lambda_max = min(Theta_prfaas/p, Theta_pd-p/(1-p), Theta_pd-d) (eq. 6), and the threshold optimality condition Theta_prfaas/p = Theta_pd-p/(1-p) (eq. 7). The following script reproduces the paper's case study from its own published inputs (Table 5 profiling data plus the stated workload distribution) and was executed as written; every assert passed. Output follows the listing.

"""Reproduce the PrfaaS-PD case study (arXiv 2604.15039, section 4) from its
own published inputs: Table 5 profiling data, the truncated log-normal
workload, and the throughput model of section 3.4 (equations 1-8).
"""
import math
from bisect import bisect_right

MU, SIGMA = 9.90, 1.00                    # ln-space workload parameters
LO, HI = 128.0, 128_000.0                 # truncation bounds (tokens)
GBIT = 1e9 / 8                            # bytes per Gbit

# Table 5: (tokens, KVCache MiB, prefill seconds on one 8xH200 instance)
PROFILE = [(1_000, 190.8, 0.44), (8_000, 308.9, 0.72),
           (32_000, 701.3, 1.84), (128_000, 2316.3, 7.40)]


def phi(x: float) -> float:
    """Standard normal CDF."""
    return 0.5 * (1.0 + math.erf(x / math.sqrt(2.0)))


def _z(t: float) -> float:
    return (math.log(t) - MU) / SIGMA


def p_longer(t: float) -> float:
    """P(L > t) under the truncated log-normal."""
    z_lo, z_hi = _z(LO), _z(HI)
    return (phi(z_hi) - phi(_z(t))) / (phi(z_hi) - phi(z_lo))


def mean_between(a: float, b: float) -> float:
    """E[L | a < L <= b] under the truncated log-normal (closed form)."""
    m = math.exp(MU + SIGMA**2 / 2)
    num = phi(_z(b) - SIGMA) - phi(_z(a) - SIGMA)
    den = phi(_z(b)) - phi(_z(a))
    return m * num / den


def interp(l: float, col: int) -> float:
    """Piecewise-linear interpolation of Table 5 column (1=MiB, 2=s)."""
    xs = [r[0] for r in PROFILE]
    i = min(max(bisect_right(xs, l) - 1, 0), len(PROFILE) - 2)
    (x0, *r0), (x1, *r1) = PROFILE[i], PROFILE[i + 1]
    w = (l - x0) / (x1 - x0)
    return r0[col - 1] + w * (r1[col - 1] - r0[col - 1])


def kv_mib(l: float) -> float:
    return interp(l, 1)


def t_prefill_h200(l: float) -> float:
    return interp(l, 2)


# --- Workload facts the paper states (section 4.1 / 4.3.1) ---
mean_len = mean_between(LO, HI)
assert abs(mean_len - 27_000) / 27_000 < 0.02        # "mean ~27K tokens"
T_STAR = 19_400
p_star = p_longer(T_STAR)
assert abs(p_star - 0.496) < 0.005                   # "49.6% routed to PrfaaS"
l_long = mean_between(T_STAR, HI)
assert abs(l_long - 44_000) / 44_000 < 0.03          # "E[L|L>t] ~ 44K"
l_short = mean_between(LO, T_STAR)

# --- Stage throughputs at the published operating point (Table 6) ---
N_PRFAAS, N_P, N_D = 4, 3, 5          # 8-GPU instances: 32 H200 / 64 H20
B_OUT = 100 * GBIT                    # inter-cluster link, bytes/s

theta_prfaas = min(N_PRFAAS / t_prefill_h200(l_long),
                   B_OUT / (kv_mib(l_long) * 2**20))          # eq. 3
assert abs(theta_prfaas - 1.61) / 1.61 < 0.05                 # paper: 1.61
compute_bound = N_PRFAAS / t_prefill_h200(l_long) < B_OUT / (
    kv_mib(l_long) * 2**20)
assert compute_bound                   # paper: "currently compute-bound"

egress_gbps = theta_prfaas * kv_mib(l_long) * 2**20 / GBIT    # section 4.3.1
assert abs(egress_gbps - 13.0) < 1.5                          # paper: ~13 Gbps

# Decode stage (eq. 5): BS_max is recoverable from the paper's own numbers.
LOUT, SLO_TOKS = 1024, 40.0
decode_s = LOUT / SLO_TOKS                                    # 25.6 s/request
bs_max = 3.91 * decode_s / N_D
assert abs(bs_max - 20.0) < 0.1        # Table 6 decode rows imply BS_max = 20
theta_pd_d = N_D * bs_max / decode_s
theta_pd_d_homog = 3 * bs_max / decode_s
assert abs(theta_pd_d_homog - 2.35) < 0.02   # homogeneous Nd=3 row cross-check

# End-to-end throughput (eq. 6), using the paper's own Theta_pd-p = 1.64
# (H20 prefill was profiled on hardware we cannot re-run).
THETA_PD_P = 1.64
lam_max = min(theta_prfaas / p_star, THETA_PD_P / (1 - p_star), theta_pd_d)
assert abs(lam_max - 3.24) / 3.24 < 0.05                      # paper: 3.24
assert abs(lam_max / 2.11 - 1.54) < 0.08                      # paper: 1.54x

# Balance condition (eq. 7) holds at the published optimum within 5%.
lhs, rhs = theta_prfaas / p_star, THETA_PD_P / (1 - p_star)
assert abs(lhs - rhs) / rhs < 0.05

# --- Independent re-derivation of the threshold (fig. 5b / eq. 7) ---
# Fit one H20-vs-H200 compute ratio from the homogeneous baseline
# (Theta_pd-p = 2.11 req/s with 9 instances at l = mean_len), then bisect
# eq. 7. A single constant is a simplification; the paper profiles full
# curves, so we accept a tolerance band around t* = 19.4K.
k_h20 = 9 / (2.11 * t_prefill_h200(mean_len))


def balance_gap(t: float) -> float:
    p = p_longer(t)
    prfaas = min(N_PRFAAS / t_prefill_h200(mean_between(t, HI)),
                 B_OUT / (kv_mib(mean_between(t, HI)) * 2**20))
    pd_p = N_P / (k_h20 * t_prefill_h200(mean_between(LO, t)))
    return prfaas / p - pd_p / (1 - p)


lo_t, hi_t = 2_000.0, 100_000.0
assert balance_gap(lo_t) < 0 < balance_gap(hi_t)
for _ in range(80):
    mid = (lo_t + hi_t) / 2
    lo_t, hi_t = (mid, hi_t) if balance_gap(mid) < 0 else (lo_t, mid)
t_solved = (lo_t + hi_t) / 2
assert 15_000 < t_solved < 23_000      # paper's grid search: t* = 19.4K

# Dense-model contrast (Table 3 / section 2.3): why this needs hybrid KV.
PHI_DENSE_32K, PHI_HYBRID_32K = 59.93, 4.66           # Gbps, MiniMax vs MiMo
assert PHI_DENSE_32K / PHI_HYBRID_32K > 12            # "13x reduction"
dense_512gpu_tbps = (512 / 8) * PHI_DENSE_32K / 1000
assert dense_512gpu_tbps > 3.5         # paper: 3.8 Tbps, no WAN carries this

print(f"workload mean               : {mean_len:,.0f} tokens")
print(f"P(L > 19.4K)                : {p_star:.1%}   (paper 49.6%)")
print(f"E[L | L > t] / E[L | L <= t]: {l_long:,.0f} / {l_short:,.0f} tokens")
print(f"Theta_prfaas (eq. 3)        : {theta_prfaas:.2f} req/s (paper 1.61)")
print(f"PrfaaS egress               : {egress_gbps:.1f} Gbps  (paper ~13)")
print(f"Lambda_max (eq. 6)          : {lam_max:.2f} req/s (paper 3.24)")
print(f"re-derived threshold t*     : {t_solved/1000:.1f}K  (paper 19.4K)")
print(f"dense egress, 512 H200s     : {dense_512gpu_tbps:.1f} Tbps (paper 3.8)")
print("all assertions passed")

Executed output:

workload mean               : 27,313 tokens
P(L > 19.4K)                : 49.5%   (paper 49.6%)
E[L | L > t] / E[L | L <= t]: 44,756 / 10,224 tokens
Theta_prfaas (eq. 3)        : 1.55 req/s (paper 1.61)
PrfaaS egress               : 11.9 Gbps  (paper ~13)
Lambda_max (eq. 6)          : 3.13 req/s (paper 3.24)
re-derived threshold t*     : 17.8K  (paper 19.4K)
dense egress, 512 H200s     : 3.8 Tbps (paper 3.8)
all assertions passed

Reading the results honestly:

The workload statistics, the egress bandwidth, the compute-vs-bandwidth binding, the decode-side batch constant (BS_max = 20 falls straight out of their Table 6 rows), and the end-to-end throughput all reproduce from published inputs to within 5-10%. The residual gaps (1.55 vs 1.61 req/s, 11.9 vs ~13 Gb/s) come from our piecewise-linear interpolation of four Table 5 points where the paper uses its full profiled curves.
The threshold re-derivation (17.8K vs 19.4K) additionally fits a single H20:H200 compute ratio from the homogeneous baseline, which flattens a length-dependent ratio into a constant; the crossing exists, is unique, and lands within 8% of the paper's grid-search optimum. Treat the mechanism as confirmed and the exact threshold as something you must re-derive from your own profiling.
One thing this reproduction deliberately exposes: everything in the paper's section 4, including the +54%, is the analytical model evaluated on measured profiling data, not a wall-clock measurement of a deployed two-datacenter system. The paper says so; repeat it whenever quoting the headline.³
The dense-model contrast is the load-bearing negative result: 3.8 Tb/s of egress for one 512-GPU dense cluster is why none of this applies to dense-attention fleets.

How to use it (LMCache as the KV transfer and storage layer)¶

LMCache's current architecture is MP (multi-process) mode: one lmcache server per node, a standalone ZMQ service with a FastAPI HTTP frontend, shared by the vLLM pods on that node (process isolation, no GIL contention with inference, cache survives engine restarts). The older in-process mode, including the LMCacheConnectorV1 flow shown in vLLM's own kv_cache_sharing_lmcache_v1.py example quoted in centralized KV cache placement, is documented as deprecated in favor of MP mode.⁵

The disaggregated-prefill recipe from the LMCache docs (reference template, quoted from docs.lmcache.ai on 2026-07-15; requires GPUs, lmcache, and NIXL, not executed in this KB's environment):

# One LMCache server per side (prefill shown; decoder mirrors on 6556/8091)
lmcache server --port 6555 --http-port 8090 \
  --l1-size-gb 100 --eviction-policy LRU --chunk-size 256 \
  --instance-id prefiller

# Prefill vLLM: NIXL moves KV to the decoder; LMCacheMP offloads/reuses
VLLM_NIXL_SIDE_CHANNEL_HOST=<PREFILL_IP> VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
UCX_NET_DEVICES=all NCCL_CUMEM_ENABLE=1 \
vllm serve <model> --port 8001 --tensor-parallel-size 1 \
  --kv-transfer-config '{"kv_connector":"MultiConnector",
    "kv_role":"kv_producer","kv_connector_extra_config":{"connectors":[
      {"kv_connector":"NixlConnector","kv_role":"kv_producer",
       "kv_load_failure_policy":"fail"},
      {"kv_connector":"LMCacheMPConnector","kv_role":"kv_both",
       "kv_connector_extra_config":{"lmcache.mp.host":"tcp://localhost",
                                    "lmcache.mp.port":6555}}]}}'

# Decode vLLM: same shape with kv_role kv_consumer, its own ports/server

# Router: sends prefill then decode, threading the NIXL handshake
vllm-router --policy round_robin --vllm-pd-disaggregation \
  --prefill http://<PREFILL_IP>:8001 --decode http://<DECODE_IP>:8002 \
  --host 0.0.0.0 --port 30000

The configuration surface that matters for a PrfaaS-shaped deployment is the MP-mode lmcache server CLI, not the older YAML/env-var surface (that surface belongs to the deprecated in-process connector mode and no longer matches what an MP deployment actually reads):⁷

Flag	Default	Role in this architecture
`--chunk-size`	256	Tokens per transfer/storage chunk; the per-transfer-cost amortizer
`--l1-size-gb`	required, no default	Sizes the pinned-DRAM L1 tier, or the GDS slab file when `--gds-l1-path` is set
`--gds-l1-path`	unset (opt-in)	Switches the L1 medium from pinned DRAM to an NVMe slab accessed via GPUDirect Storage DMA
`--l2-adapter <JSON>` (repeatable)	unset	L2/remote backend; each JSON object's `type` field selects an adapter (`nixl_store`, `fs`, `mooncake_store`, `aerospike`, `s3`, `resp`, and others); repeat the flag to cascade several
`--engine-type`	`default`	`default` = standard prefix caching; `blend`/`blend_legacy` select CacheBlend for non-prefix reuse (CacheGen compression is not yet implemented for MP mode)

Mapping to the paper's three subsystems: the LMCache servers plus L1/L2 tiers implement the per-cluster hybrid prefix cache pool; NIXL (or the LMCache remote backend over TCP) implements the KV transfer; the global KV cache manager and the length-threshold scheduler have no off-the-shelf equivalent, and the vllm-router above is a plain round-robin PD router, not a PrfaaS router. Budget engineering time accordingly.

How to develop with it¶

Route on uncached length, which means the router must query cache state. The paper's rule under scarce bandwidth: prefill locally if l_total - l_pd <= t, else offload. Under abundant bandwidth, take the best prefix across clusters (l_prefix = max(l_prfaas, l_pd)) and transfer cache across clusters when the winner is remote.¹ LMCache MP mode exposes LOOKUP over its ZMQ protocol and an HTTP frontend, which is the hook a custom router uses to compute uncached length before placement.
Re-derive the threshold from your own profiling, not the paper's. The executed model above is the template: profile S_kv(l) and T_prefill(l) at four lengths on both hardware pools, plug in your measured request-length distribution, bisect equation 7. The 19.4K figure is one workload on one model on one hardware pair.
The CacheGen compression tradeoff still applies, but not as an MP-mode flag today. Centralized KV cache placement covers when the ~3.5-4.3x volume reduction CacheGen buys is worth its codec cost; as of this writing that compression path is only implemented for LMCache's deprecated in-process mode, not MP mode, so an MP-mode deployment on a constrained link has to weigh compression elsewhere in the stack (or re-check whether MP mode has since picked it up).
Respect hybrid-model cache semantics. Recurrent state is request-level and exact-match-only; full-attention KV is block-level and prefix-matchable. vLLM's hybrid KV cache manager (which the paper builds on) handles the split, and LMCache's hybrid-models support adds hard alignment requirements for Mamba/GDN hybrids: the LMCache --chunk-size, vLLM's --max-num-batched-tokens, and the unified block size must align, with --mamba-cache-mode align --enable-prefix-caching mandatory for GDN models, and the docs warn that generation is not bit-exact between a cached and a fresh run.⁶ Any custom eviction or dedup logic you add must never treat a recurrent-state entry as partially reusable.¹
Test the miss path explicitly. The NixlConnector's kv_load_failure_policy in the recipe above is set to fail; production wants recompute semantics somewhere in the stack so a lost transfer degrades to a local re-prefill rather than a failed request, per the fail-toward-recompute rule in centralized KV cache placement.

How to maintain it¶

Watch three signals per the paper's scheduler: egress-link utilization, PrfaaS queue depth, and per-stage throughput balance. Short-term congestion triggers threshold re-search; long-term drift (traffic mix, cache hit rate) triggers converting PD-cluster nodes between prefill and decode roles to restore equations 7 and 8.¹ LMCache ships Prometheus metrics and OTel tracing for the cache side;⁵ the balance monitor is yours.
Re-profile on every model revision. Phi_kv is an architecture property; a new checkpoint with a different hybrid ratio, KV dtype, or attention layout moves every operating point. Store the four-point profile with the model artifact.
Version cache keys across the fleet, both clusters. A cross-datacenter KV entry produced by last week's weights is the same silent-corruption hazard as in any shared store; rollouts must invalidate both the local pools and anything in flight.
Capacity-plan the pools with the same arithmetic. S_kv(l) times expected cached tokens per cluster, plus the transfer-cache working set (in-flight requests times their tail KV), sized so transfer blocks never evict hot prefix blocks.

How to run it in production¶

Enforce the three gates at admission, not in a design doc. A config flag that lets dense-model traffic onto the PrfaaS path recreates the 60 Gb/s-per-instance egress problem instantly. Gate offload eligibility on model identity and measured Phi_kv.
Smooth the link, do not just size it. The paper's transport stack is layer-wise prefill pipelining (KV of layer n transfers while layer n+1 computes), multi-connection TCP to fill the pipe, and congestion monitoring wired into the scheduler so routing backs off before queues build.¹ A single-stream TCP transfer at 100 Gb/s over a real RTT will not fill the link on its own.
Keep decode SLO isolation. The decode pool's BS_max is SLO-governed (the case study: 40 tok/s per request, giving BS_max = 20 on their hardware); PrfaaS increases arrival at the decode pool, so admission control must cap it, or TTFT wins convert to TPOT losses.
Treat the inter-DC link as hostile. KV crosses an organizational boundary; VPC peering or dedicated lines still warrant encryption in transit and tenant isolation in the shared pools. Cache entries are model activations; leaking them is leaking the context.
Fail toward local. Link down, remote queue full, or transfer timeout must all degrade to local prefill (higher TTFT, correct output). The PrfaaS cluster is an accelerator, never a dependency; if losing it takes down serving, the topology is wired wrong.

Failure modes¶

Naive offload. All prefill remote, no threshold: 1.16x instead of 1.54x in the paper's own comparison, plus a congested link. The threshold is the architecture.³
Routing on total instead of uncached length. Agentic traffic is dominated by incremental prefills over cached prefixes; ignoring cache state over-offloads and re-transfers bytes the local pool already held.
Dense model on the hybrid path. One 8-GPU dense instance at 32K emits ~60 Gb/s of KV; a handful saturates the link and stalls every in-flight transfer behind it.²
Headline misquote. Quoting +54% without "model-derived, heterogeneous-hardware, ~15% at equal cost" overstates the case; the honest pitch is TTFT and elasticity, with throughput contingent on hardware pricing.³
Recurrent-state mishandling. Treating request-level linear-attention state as block-matchable prefix cache serves wrong-context state on a partial hit; the hybrid pool split exists precisely to prevent this.¹
Transfer blocks evicting prefix blocks. Under burst, in-flight transfer-cache can push out hot shared prefixes, trading a one-time transfer win for a fleet-wide hit-rate collapse; partition the pools as the paper does.¹

References¶

PrfaaS: R. Qin, W. He, Y. Wang, Z. Li, X. Xu, Y. Wu, W. Zheng, M. Zhang, "Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter," arXiv 2604.15039 (preprint, Moonshot AI and Tsinghua). https://arxiv.org/abs/2604.15039
LMCache documentation (MP mode, disaggregated prefill, configuration reference): https://docs.lmcache.ai/
LMCache repository: https://github.com/LMCache/LMCache
CacheGen (the cachegen serde): Y. Liu et al., arXiv 2310.07240 (SIGCOMM'24). https://arxiv.org/abs/2310.07240
CacheBlend (the blending feature): J. Yao et al., arXiv 2405.16444 (EuroSys'25). https://arxiv.org/abs/2405.16444
Mooncake (the lineage system and an LMCache L2 backend): R. Qin et al., arXiv 2407.00079 (FAST'25). https://arxiv.org/abs/2407.00079
Kimi Linear (the case-study model architecture): arXiv 2510.26692. https://arxiv.org/abs/2510.26692
"Hybrid Models as First-Class Citizens in vLLM" (PyTorch blog): https://pytorch.org/blog/hybrid-models-as-first-class-citizens-in-vllm/ · vLLM hybrid KV cache manager design doc: https://docs.vllm.ai/en/stable/design/hybrid_kv_cache_manager/
LMCache hybrid attention models guide: https://docs.lmcache.ai/mp/hybrid_models.html

arXiv 2604.15039, sections 3.1-3.4: PrfaaS/local-PD cluster split with intra-cluster RDMA and inter-cluster VPC peering or dedicated lines; length-threshold routing on incremental (uncached) prefill length; hybrid prefix cache pool with separate KV groups over a shared block pool, prefix-cache vs transfer-cache block classes, built on vLLM's hybrid KVCache manager; layer-wise prefill pipelining, multi-connection TCP, congestion monitoring; bandwidth-scarce vs bandwidth-abundant routing rules; dual-timescale scheduling restoring equations 7 and 8. Read in full on 2026-07-15. ↩↩↩↩↩↩↩↩↩↩
arXiv 2604.15039, section 2 (Tables 1-3): hybrid ratios (Kimi Linear 3:1 KDA:MLA, MiMo-V2-Flash 5:1 SWA:GQA, Qwen3.5-397B 3:1 GDN:GQA, Ring-2.5-1T 7:1 Lightning:MLA); KV throughput at 32K on 8xH200 with SGLang v0.5.9: MiniMax-M2.5 59.93 Gbps and Qwen3-235B 33.35 Gbps (dense) vs Kimi Linear 3.87, MiMo-V2-Flash 4.66, Qwen3.5-397B 8.25, Ring-2.5-1T 2.59 Gbps (hybrid); 512-H200 dense egress 3.8 Tbps (MiniMax-M2.5) / 2.1 Tbps (Qwen3); Ring-2.5-1T ~170 Gbps at 32K average, under 100 Gbps when routing only 128K-class requests; ~1.8 Tbps for a 10,000-GPU deployment. ↩↩↩
arXiv 2604.15039, section 4: 32 H200 (4 instances) + 64 H20 (8 instances) vs 96 H20; internal 1T Kimi Linear-style model, truncated log-normal input lengths (mu 9.90, sigma 1.00, [128, 128K], mean ~27K), 1024-token outputs, 40 tok/s SLO. Optimal t = 19.4K, Np/Nd = 3/5; 49.6% of requests offloaded at E[L|L>t] ~ 44K; ~13 Gbps average egress (13% of the 100 Gbps link); Lambda_max 3.24 vs 2.11 req/s (+54%, or ~15% at equal cost per section 4.4); mean/P90 TTFT 2.22/3.51 s vs 4.44/9.73 s; naive heterogeneous PD (all prefill remote, no scheduling) reaches only 1.16x. The paper states all throughput and bandwidth results "are derived by feeding the measured profiling data into the throughput model"; they are model-derived from real profiling, not an end-to-end deployed-system measurement. Vendor-evaluated preprint, not independently reproduced. ↩↩↩↩↩↩
docs.lmcache.ai (fetched 2026-07-15): "a KV cache management layer for LLM inference"; storage backends CPU RAM, local SSD, Redis/Valkey, S3, Bigtable, Mooncake, NIXL, Aerospike; CacheGen compression; CacheBlend non-prefix reuse; P2P sharing; PD disaggregation over NVLink, RDMA, or TCP. ↩
docs.lmcache.ai MP-mode pages (fetched 2026-07-15): lmcache server as a standalone ZMQ service with FastAPI HTTP frontend, one per node serving multiple vLLM pods; process isolation, shared L1, independent CPU-memory scaling; L1 CPU DRAM or NVMe via GPUDirect Storage, L2 S3/Bigtable/Redis/filesystem/NIXL; disaggregated prefill via NIXL with MultiConnector combining NixlConnector and LMCacheMPConnector (lmcache.mp.host/lmcache.mp.port extra-config keys, VLLM_NIXL_SIDE_CHANNEL_HOST/PORT for the handshake); Prometheus metrics and OTel tracing; the in-process mode is marked deprecated in favor of MP mode. ↩↩
docs.lmcache.ai/mp/hybrid_models.html (fetched 2026-07-15): validated hybrid architectures include Gemma 3/4, gpt-oss (SWA+full), Qwen3.5/3.6 (Mamba/GDN+full), DeepSeek-V4-Flash (sparse-MLA), GLM 5.1/5.2, MiniMax-M3; for Mamba/linear hybrids the LMCache server --chunk-size N and vLLM --max-num-batched-tokens (in [N, 2N)) must derive from the unified block size, with --mamba-cache-mode align --enable-prefix-caching mandatory for GDN; linear-attention layers keep a recurrent state cache that LMCache treats as opaque pages; "generation is not bit-exact between a cached and a fresh run"; only text KV is validated, and DeepSeek-V4-style compressed/indexer caches are not yet handled by the multiprocess connector. ↩
docs.lmcache.ai MP-mode Configuration Reference, lmcache/v1/multiprocess/config.py and lmcache/v1/distributed/config.py (fetched 2026-07-15): --chunk-size (default 256, tokens); --l1-size-gb (required, sizes the pinned-DRAM L1 or, with --gds-l1-path set, the GDS slab file); --gds-l1-path (unset by default, opt-in NVMe-via-GPUDirect-Storage L1); --l2-adapter <JSON> (repeatable, unset by default, type field selects nixl_store/fs/mooncake_store/aerospike/s3/resp/others, multiple flags cascade in order); --engine-type (default, blend, or blend_legacy; CacheGen compression is not yet implemented for MP mode). The older chunk_size/local_cpu/remote_url/remote_serde/enable_blending-style YAML keys and LMCACHE_-prefixed environment variables belong to the deprecated in-process connector mode and do not apply to an MP-mode deployment. ↩