Skip to content
Markdown

Prefill-as-a-Service: cross-datacenter prefill, and how to build the KV plumbing with LMCache

Scope: the Prefill-as-a-Service (PrfaaS) serving paradigm from Moonshot AI and Tsinghua (arXiv 2604.15039): routing long uncached prefills to standalone compute-dense clusters and transferring the resulting KV cache over commodity Ethernet to local prefill/decode clusters for decode. Covers the throughput model that decides the routing threshold (reproduced and executed below from the paper's own published inputs), why it only works for hybrid-attention models, and the LMCache configuration surface that provides the open-source KV transfer and storage layer such a deployment runs on, verified below against the installed package, not only its docs. Extends disaggregated inference beyond one fabric and applies the break-even discipline of centralized KV cache placement to a concrete architecture. Classification: an architecture and economics study of a proposed paradigm (PrfaaS itself has no released implementation; its global KV cache manager and length-threshold scheduler are described in the paper but not shipped as code anywhere) paired with a real, installable developer guide to the one piece of the stack that does ship, LMCache's KV transfer and storage layer; the HOW sections split accordingly.

Verdict up front: PrfaaS moves long prefills to a remote compute-dense pool, transfers the resulting KV once, and keeps decode local to that KV. Consider it only when three gates pass: the model's measured KV output rate fits the inter-datacenter budget, the workload contains enough long uncached or incremental prefill, and a scheduler applies a profiled length threshold. Prefix reuse changes the uncached-length distribution and can improve placement, but it is not an architectural prerequisite. The paper's +54% throughput and 64% lower P90 TTFT are outputs of an analytical case study populated by measured vendor-hardware profiles; the equal-cost gain is about 15%.

What it is

PrfaaS-PD is a two-tier serving topology. Local PD clusters run conventional prefill/decode disaggregation on bandwidth-optimized accelerators (H20-class) over intra-cluster RDMA and can serve any request end to end. PrfaaS clusters are standalone pools of compute-dense accelerators (H200-class, or prefill-specialized parts like Rubin CPX) that do exactly one thing: run long-context prefill for requests whose uncached length exceeds a routing threshold t, then push the finished KV cache over an inter-datacenter link (VPC peering or dedicated lines, ~100 Gb/s class) to a local PD cluster, which decodes. Decode never runs remote from its KV, and nothing crosses the inter-datacenter link per token; the transfer is once per request, overlapped with prefill via layer-wise pipelining.1

The enabling condition is the model's measured KV output rate. Only full-attention layers produce KV that grows with sequence length; linear-attention and sliding-window layers carry fixed-size recurrent state. Hybrid models interleave the two (Kimi Linear at 3:1 KDA:MLA, MiMo-V2-Flash at 5:1 SWA:GQA, Qwen3.5-397B at 3:1 GDN:GQA, Ring-2.5-1T at 7:1 Lightning:MLA), reducing per-instance KV throughput Phi_kv(l) = S_kv(l) / T_prefill(l) to 2.6-8.3 Gb/s in the paper's measured hybrid-model profiles.2 At 32K, comparable pairs show about 4x lower throughput for Qwen3.5 than Qwen3 and about 13x for MiMo-V2-Flash than MiniMax-M2.5. Those reductions can move a one-shot handoff from an RDMA-only domain to a provisioned Ethernet link; they do not establish a universal ratio.

LMCache is the open-source layer closest to this architecture's storage and transfer subsystem: a standalone KV cache service for vLLM (and other engines) with tiered storage (CPU DRAM, NVMe via GPUDirect Storage, then S3/Redis/Valkey/Bigtable/Mooncake/NIXL backends), disaggregated-prefill transfer over NVLink/RDMA/TCP, CacheGen compression for constrained links, and CacheBlend for non-prefix reuse.4 LMCache does not provide PrfaaS's global KV manager, threshold router, or dual-timescale scheduler; those control-plane components must be implemented separately.

Why use it

  • Heterogeneous hardware without a shared fabric. Compute-dense and bandwidth-dense accelerators rarely live in the same RDMA domain, and forcing them into one fixes the prefill:decode hardware ratio forever. PrfaaS lets each phase scale independently on the silicon that suits it, which is where inference hardware is already heading (Rubin CPX for prefill; LPU-class parts for decode).1
  • Throughput and TTFT, at the case-study numbers. Against a homogeneous 96-H20 baseline, the PrfaaS-PD configuration (32 H200 + 64 H20) reaches 3.24 vs 2.11 req/s (+54%) with mean TTFT halved (2.22 s vs 4.44 s) and P90 TTFT down 64% (3.51 s vs 9.73 s), because long requests stop queueing behind short ones for local prefill capacity. At equal hardware cost the paper puts the gain near 15%.3
  • Scheduling accounts for much of the modeled gain over naive offload. Sending all prefill to the remote cluster and all decode local yields 1.16x in the paper's model; selective offload and phase balancing raise the modeled result to 1.54x.3
  • The bandwidth bill is small when the gates pass. At the optimal threshold, 49.6% of requests offload, the offloaded mean uncached length is ~44K tokens, and average egress is ~13 Gb/s, 13% of one 100 Gb/s link.3 For scale: a 10,000-GPU prefill datacenter of Ring-2.5-1T-class models aggregates to ~1.8 Tb/s, within modern datacenter interconnect capacity, whereas one 512-GPU cluster of dense MiniMax-M2.5 alone would demand 3.8 Tb/s.2

When to use it (and when not)

Use it when all three gates pass:

  • Model gate: hybrid-attention or MLA architecture with measured Phi_kv in the single-digit Gb/s range at your operating lengths. Profile it exactly as the paper does (KV size and prefill latency at 1K/8K/32K/128K); dense GQA models fail this gate by an order of magnitude and stay inside one fabric (disaggregated inference).
  • Workload gate: enough requests have long uncached or incremental prefill (the case study uses a truncated log-normal distribution with roughly 27K-token mean). Prefix reuse changes this distribution but is optional; if most uncached lengths are short, the remote pool idles.
  • Scheduler gate: the router receives a trustworthy uncached length from the serving/cache control plane, monitors egress congestion, and rebalances capacity. Without reusable cache, uncached length equals total prompt length. With reuse, routing requires tenant-scoped global prefix metadata.

Do not use it when:

  • The model is dense-attention. The 2.1-3.8 Tb/s egress arithmetic is prohibitive; this is exactly the dense-model "never split P/D across sites" rule that cross-WAN inference and centralized KV cache placement preserve.
  • You expect per-token benefits. Decode economics are untouched; decode runs local to delivered KV, and steady-state tokens/s is governed by the decode cluster alone (the latency-bound argument).
  • Requests are short or cache-hit-dominated end to end. An incremental prefill below the threshold never justifies a cross-datacenter trip; the local PD path already serves it optimally.
  • Reuse exists but cache placement is not visible globally. Routing on total rather than uncached length over-offloads requests whose reusable prefix is already local. Disable cache-aware offload or use a control-plane API that reports the longest fully reusable, tenant-scoped prefix.

Architecture

flowchart LR
  C["Client"] --> R["Global scheduler:<br/>route on uncached length vs threshold t,<br/>cache affinity, egress congestion"]
  R -->|"uncached len <= t"| PDP["Local PD cluster (H20-class)<br/>PD-P prefill nodes"]
  R -->|"uncached len > t"| PF["PrfaaS cluster (H200-class)<br/>long-context prefill only"]
  PF -->|"KV cache, once per request,<br/>layer-wise pipelined, multi-conn TCP<br/>(~100 Gb/s inter-DC Ethernet)"| PDD["Local PD cluster<br/>PD-D decode nodes"]
  PDP -->|"KV via intra-cluster RDMA"| PDD
  PDD -->|"tokens"| C
  M["Global KV cache manager<br/>hybrid prefix cache pool"] -.-> R

Two storage details from the paper matter for anyone reimplementing this. First, hybrid models need a hybrid prefix cache pool: full-attention KV is block-level and supports partial prefix matching, but the recurrent state of linear/SWA layers is request-level, fixed-size, and reusable only on an exact length match, so the two live in separate KV groups over one shared block pool (built on vLLM's hybrid KV cache manager).1 Second, blocks are split into prefix-cache blocks (must be fully populated before reuse) and transfer-cache blocks (tail KV of an in-flight request, discarded once the prefill-to-decode transfer completes), so transfer traffic never pollutes the reuse pool.1

How it works (validated throughput model)

The scheduling decisions all fall out of one small model (the paper's section 3.4): stage throughputs Theta_prfaas = min(compute, egress) (eq. 3), Theta_pd-p (eq. 4), Theta_pd-d (eq. 5), the pipeline bound Lambda_max = min(Theta_prfaas/p, Theta_pd-p/(1-p), Theta_pd-d) (eq. 6), and the threshold optimality condition Theta_prfaas/p = Theta_pd-p/(1-p) (eq. 7). The following script reproduces the paper's case study from its own published inputs (Table 5 profiling data plus the stated workload distribution) and was executed as written; every assert passed. Output follows the listing.

"""Reproduce the PrfaaS-PD case study (arXiv 2604.15039, section 4) from its
own published inputs: Table 5 profiling data, the truncated log-normal
workload, and the throughput model of section 3.4 (equations 1-8).
"""
import math
from bisect import bisect_right

MU, SIGMA = 9.90, 1.00                    # ln-space workload parameters
LO, HI = 128.0, 128_000.0                 # truncation bounds (tokens)
GBIT = 1e9 / 8                            # bytes per Gbit

# Table 5: (tokens, KVCache MiB, prefill seconds on one 8xH200 instance)
PROFILE = [(1_000, 190.8, 0.44), (8_000, 308.9, 0.72),
           (32_000, 701.3, 1.84), (128_000, 2316.3, 7.40)]


def phi(x: float) -> float:
    """Standard normal CDF."""
    return 0.5 * (1.0 + math.erf(x / math.sqrt(2.0)))


def _z(t: float) -> float:
    return (math.log(t) - MU) / SIGMA


def p_longer(t: float) -> float:
    """P(L > t) under the truncated log-normal."""
    z_lo, z_hi = _z(LO), _z(HI)
    return (phi(z_hi) - phi(_z(t))) / (phi(z_hi) - phi(z_lo))


def mean_between(a: float, b: float) -> float:
    """E[L | a < L <= b] under the truncated log-normal (closed form)."""
    m = math.exp(MU + SIGMA**2 / 2)
    num = phi(_z(b) - SIGMA) - phi(_z(a) - SIGMA)
    den = phi(_z(b)) - phi(_z(a))
    return m * num / den


def interp(l: float, col: int) -> float:
    """Piecewise-linear interpolation of Table 5 column (1=MiB, 2=s)."""
    xs = [r[0] for r in PROFILE]
    i = min(max(bisect_right(xs, l) - 1, 0), len(PROFILE) - 2)
    (x0, *r0), (x1, *r1) = PROFILE[i], PROFILE[i + 1]
    w = (l - x0) / (x1 - x0)
    return r0[col - 1] + w * (r1[col - 1] - r0[col - 1])


def kv_mib(l: float) -> float:
    return interp(l, 1)


def t_prefill_h200(l: float) -> float:
    return interp(l, 2)


# --- Workload facts the paper states (section 4.1 / 4.3.1) ---
mean_len = mean_between(LO, HI)
assert abs(mean_len - 27_000) / 27_000 < 0.02        # "mean ~27K tokens"
T_STAR = 19_400
p_star = p_longer(T_STAR)
assert abs(p_star - 0.496) < 0.005                   # "49.6% routed to PrfaaS"
l_long = mean_between(T_STAR, HI)
assert abs(l_long - 44_000) / 44_000 < 0.03          # "E[L|L>t] ~ 44K"
l_short = mean_between(LO, T_STAR)

# --- Stage throughputs at the published operating point (Table 6) ---
N_PRFAAS, N_P, N_D = 4, 3, 5          # 8-GPU instances: 32 H200 / 64 H20
B_OUT = 100 * GBIT                    # inter-cluster link, bytes/s

theta_prfaas = min(N_PRFAAS / t_prefill_h200(l_long),
                   B_OUT / (kv_mib(l_long) * 2**20))          # eq. 3
assert abs(theta_prfaas - 1.61) / 1.61 < 0.05                 # paper: 1.61
compute_bound = N_PRFAAS / t_prefill_h200(l_long) < B_OUT / (
    kv_mib(l_long) * 2**20)
assert compute_bound                   # paper: "currently compute-bound"

egress_gbps = theta_prfaas * kv_mib(l_long) * 2**20 / GBIT    # section 4.3.1
assert abs(egress_gbps - 13.0) < 1.5                          # paper: ~13 Gbps

# Decode stage (eq. 5): BS_max is recoverable from the paper's own numbers.
LOUT, SLO_TOKS = 1024, 40.0
decode_s = LOUT / SLO_TOKS                                    # 25.6 s/request
bs_max = 3.91 * decode_s / N_D
assert abs(bs_max - 20.0) < 0.1        # Table 6 decode rows imply BS_max = 20
theta_pd_d = N_D * bs_max / decode_s
theta_pd_d_homog = 3 * bs_max / decode_s
assert abs(theta_pd_d_homog - 2.35) < 0.02   # homogeneous Nd=3 row cross-check

# End-to-end throughput (eq. 6), using the paper's own Theta_pd-p = 1.64
# (H20 prefill was profiled on hardware we cannot re-run).
THETA_PD_P = 1.64
lam_max = min(theta_prfaas / p_star, THETA_PD_P / (1 - p_star), theta_pd_d)
assert abs(lam_max - 3.24) / 3.24 < 0.05                      # paper: 3.24
assert abs(lam_max / 2.11 - 1.54) < 0.08                      # paper: 1.54x

# Balance condition (eq. 7) holds at the published optimum within 5%.
lhs, rhs = theta_prfaas / p_star, THETA_PD_P / (1 - p_star)
assert abs(lhs - rhs) / rhs < 0.05

# --- Independent re-derivation of the threshold (fig. 5b / eq. 7) ---
# Fit one H20-vs-H200 compute ratio from the homogeneous baseline
# (Theta_pd-p = 2.11 req/s with 9 instances at l = mean_len), then bisect
# eq. 7. A single constant is a simplification; the paper profiles full
# curves, so we accept a tolerance band around t* = 19.4K.
k_h20 = 9 / (2.11 * t_prefill_h200(mean_len))


def balance_gap(t: float) -> float:
    p = p_longer(t)
    prfaas = min(N_PRFAAS / t_prefill_h200(mean_between(t, HI)),
                 B_OUT / (kv_mib(mean_between(t, HI)) * 2**20))
    pd_p = N_P / (k_h20 * t_prefill_h200(mean_between(LO, t)))
    return prfaas / p - pd_p / (1 - p)


lo_t, hi_t = 2_000.0, 100_000.0
assert balance_gap(lo_t) < 0 < balance_gap(hi_t)
for _ in range(80):
    mid = (lo_t + hi_t) / 2
    lo_t, hi_t = (mid, hi_t) if balance_gap(mid) < 0 else (lo_t, mid)
t_solved = (lo_t + hi_t) / 2
assert 15_000 < t_solved < 23_000      # paper's grid search: t* = 19.4K

# Dense-model contrast (Table 3 / section 2.3): why this needs hybrid KV.
PHI_DENSE_32K, PHI_HYBRID_32K = 59.93, 4.66           # Gbps, MiniMax vs MiMo
assert PHI_DENSE_32K / PHI_HYBRID_32K > 12            # "13x reduction"
dense_512gpu_tbps = (512 / 8) * PHI_DENSE_32K / 1000
assert dense_512gpu_tbps > 3.5         # far beyond the case study's 100 Gb/s link

print(f"workload mean               : {mean_len:,.0f} tokens")
print(f"P(L > 19.4K)                : {p_star:.1%}   (paper 49.6%)")
print(f"E[L | L > t] / E[L | L <= t]: {l_long:,.0f} / {l_short:,.0f} tokens")
print(f"Theta_prfaas (eq. 3)        : {theta_prfaas:.2f} req/s (paper 1.61)")
print(f"PrfaaS egress               : {egress_gbps:.1f} Gbps  (paper ~13)")
print(f"Lambda_max (eq. 6)          : {lam_max:.2f} req/s (paper 3.24)")
print(f"re-derived threshold t*     : {t_solved/1000:.1f}K  (paper 19.4K)")
print(f"dense egress, 512 H200s     : {dense_512gpu_tbps:.1f} Tbps (paper 3.8)")
print("all assertions passed")

Executed output:

workload mean               : 27,313 tokens
P(L > 19.4K)                : 49.5%   (paper 49.6%)
E[L | L > t] / E[L | L <= t]: 44,756 / 10,224 tokens
Theta_prfaas (eq. 3)        : 1.55 req/s (paper 1.61)
PrfaaS egress               : 11.9 Gbps  (paper ~13)
Lambda_max (eq. 6)          : 3.13 req/s (paper 3.24)
re-derived threshold t*     : 17.8K  (paper 19.4K)
dense egress, 512 H200s     : 3.8 Tbps (paper 3.8)
all assertions passed

Reading the results honestly:

  • The workload statistics, the egress bandwidth, the compute-vs-bandwidth binding, the decode-side batch constant (BS_max = 20 falls straight out of their Table 6 rows), and the end-to-end throughput all reproduce from published inputs to within 5-10%. The residual gaps (1.55 vs 1.61 req/s, 11.9 vs ~13 Gb/s) come from the piecewise-linear interpolation of four Table 5 points used here where the paper uses its full profiled curves.
  • The threshold re-derivation (17.8K vs 19.4K) additionally fits a single H20:H200 compute ratio from the homogeneous baseline, which flattens a length-dependent ratio into a constant; the crossing exists, is unique, and lands within 8% of the paper's grid-search optimum. Treat the mechanism as confirmed and the exact threshold as something you must re-derive from your own profiling.
  • One thing this reproduction deliberately exposes: everything in the paper's section 4, including the +54%, is the analytical model evaluated on measured profiling data, not a wall-clock measurement of a deployed two-datacenter system. The paper says so; repeat it whenever quoting the headline.3
  • The dense-model contrast is the load-bearing negative result: 3.8 Tb/s of egress for one 512-GPU dense cluster is why none of this applies to dense-attention fleets.

How to use it (LMCache as the KV transfer and storage layer)

LMCache's current architecture is MP (multi-process) mode: one lmcache server per node, a standalone ZMQ service with a FastAPI HTTP frontend, shared by the vLLM pods on that node (process isolation, no GIL contention with inference, cache survives engine restarts). The older in-process mode, including the LMCacheConnectorV1 flow shown in vLLM's own kv_cache_sharing_lmcache_v1.py example quoted in centralized KV cache placement, is documented as deprecated in favor of MP mode.5

Independently verified here: pip install lmcache installs lmcache==0.5.1 cleanly on CPU-only Python 3.12 (no GPU needed for the CLI itself; torch correctly falls back to StubCPUDevice when no NVIDIA driver is present). lmcache --help and lmcache server --help were run for real against that install. A real, reproducible defect surfaced immediately: lmcache --help raises ModuleNotFoundError: No module named 'openai' even for subcommands that have nothing to do with OpenAI, because the CLI eagerly imports every subcommand module (including bench/engine_bench, which imports openai.OpenAI) before parsing args; pip install openai (not a declared dependency of the lmcache package) works around it. With that workaround, lmcache server --help printed its complete, real flag set, reproduced in the table below and corrected against it, not against documentation. lmcache/v1/multiprocess/http_apis/cache_api.py and .../cache_control/prefetch_service.py were also read from the installed package (not fetched from GitHub) to re-check the found_keys claim in "Obtaining uncached length safely" below; it holds in the installed v0.5.1 code exactly as stated.

LMCache publishes tagged releases, including v0.5.1, but the MP MultiConnector guide used here was verified at commit 76744ce2514f3444e9a6fb06d67b23739fa93b51. That guide explicitly requires the real-blocks fix in vLLM PR 46865; without it, LMCache offload under MultiConnector silently does not trigger. It also requires nixl>=1.3.0 through the lmcache[nixl] extra.8 Use an LMCache/vLLM pair containing those changes and pin both artifacts; do not assume the v0.5.1 package implements this newer template.

The following is a version-locked reference template from that LMCache source snapshot. It requires two GPU hosts, a compatible vLLM build, LMCache from the pinned source, and NIXL; it was not executed in this KB environment:

PREFILL_IP="${PREFILL_IP:?set to the prefill host address}"
DECODE_IP="${DECODE_IP:?set to the decode host address}"
MODEL="${MODEL:?set to the pinned model path or repository revision}"

# One LMCache server per side (prefill shown; decoder mirrors on 6556/8091)
lmcache server --port 6555 --http-port 8090 \
  --l1-size-gb 100 --eviction-policy LRU --chunk-size 256 \
  --instance-id prefiller

# Prefill vLLM: NIXL moves KV to the decoder; LMCacheMP offloads/reuses
VLLM_NIXL_SIDE_CHANNEL_HOST="$PREFILL_IP" VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
UCX_NET_DEVICES=all NCCL_CUMEM_ENABLE=1 \
vllm serve "$MODEL" --port 8001 --tensor-parallel-size 1 \
  --kv-transfer-config '{"kv_connector":"MultiConnector",
    "kv_role":"kv_producer","kv_connector_extra_config":{"connectors":[
      {"kv_connector":"NixlConnector","kv_role":"kv_producer",
       "kv_load_failure_policy":"fail"},
      {"kv_connector":"LMCacheMPConnector","kv_role":"kv_both",
       "kv_connector_extra_config":{"lmcache.mp.host":"tcp://localhost",
                                    "lmcache.mp.port":6555}}]}}'

# Decode-side LMCache server and vLLM consumer
lmcache server --port 6556 --http-port 8091 \
  --l1-size-gb 100 --eviction-policy LRU --chunk-size 256 \
  --instance-id decoder

VLLM_NIXL_SIDE_CHANNEL_HOST="$DECODE_IP" VLLM_NIXL_SIDE_CHANNEL_PORT=5558 \
UCX_NET_DEVICES=all NCCL_CUMEM_ENABLE=1 \
vllm serve "$MODEL" --port 8002 --tensor-parallel-size 1 \
  --kv-transfer-config '{"kv_connector":"MultiConnector",
    "kv_role":"kv_consumer","kv_connector_extra_config":{"connectors":[
      {"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"},
      {"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"tcp://localhost","lmcache.mp.port":6556}}]}}'

# Router: sends prefill then decode, threading the NIXL handshake
vllm-router --policy round_robin --vllm-pd-disaggregation \
  --prefill "http://${PREFILL_IP}:8001" --decode "http://${DECODE_IP}:8002" \
  --host 0.0.0.0 --port 30000

The configuration surface that matters for a PrfaaS-shaped deployment is the MP-mode lmcache server CLI, not the older YAML/env-var surface (that surface belongs to the deprecated in-process connector mode and no longer matches what an MP deployment actually reads). The table below is transcribed from the real, executed lmcache server --help output on the installed lmcache==0.5.1 package, correcting one inaccuracy an earlier revision of this page carried from documentation alone: NIXL is not an --l2-adapter type. It is configured separately, under --p2p-transfer-engine (default nixl), alongside --p2p-advertise-url/--p2p-listen-url. The real, complete --l2-adapter type values as of this version are aerospike, dax, fault_inject, fs, fs_native, hfbucket, mock, mooncake_store, native_plugin, p2p, plugin, raw_block, resp, s3; nixl_store does not appear in this list and is not a valid adapter type at this version.7

Flag Default Role in this architecture
--chunk-size 256 Tokens per transfer/storage chunk; the per-transfer-cost amortizer
--hash-algorithm blake3 Hash algorithm for token-based cache keys (builtin, sha256_cbor, blake3)
--l1-size-gb required, no default Sizes the pinned-DRAM L1 tier, or the GDS slab file when --gds-l1-path is set
--gds-l1-path unset (opt-in) Switches the L1 medium from pinned DRAM to an NVMe slab accessed via GPUDirect Storage DMA (cuFile on NVIDIA, hipFile on ROCm)
--eviction-policy required, no default (LRU, IsolatedLRU, or noop) IsolatedLRU keeps one LRU list per cache_salt and needs per-tenant quotas configured via the HTTP API, relevant for the multi-tenant isolation this page's production section calls for
--l2-adapter <JSON> (repeatable) unset L2/remote backend; each JSON object's type field selects an adapter (real values above); repeat the flag to cascade several
--p2p-transfer-engine nixl P2P/cross-node KV transfer implementation; this, not an --l2-adapter, is where NIXL is actually configured
--engine-type default default = standard prefix caching; blend selects CacheBlend V3 (the current implementation), blend_legacy the original CacheBlend; CacheGen compression does not appear as a server flag at this version
--coordinator-url unset Joins an MP coordinator for fleet-wide registration, heartbeats, and (with --coordinator-l2-event-reporting) shared L2 usage/eviction accounting across nodes, relevant to the paper's cross-cluster global KV manager gap noted below

Mapping to the paper's three subsystems: the LMCache servers plus L1/L2 tiers implement the per-cluster hybrid prefix cache pool; the --p2p-transfer-engine/NIXL path (or a plain TCP --l2-adapter) implements the KV transfer; the global KV cache manager and the length-threshold scheduler have no off-the-shelf equivalent in this repository (the --coordinator-url mechanism shares L2 eviction/usage state across a fleet but does not implement threshold routing), and the vllm-router above is a plain round-robin PD router, not a PrfaaS router. Budget engineering time accordingly.

How to develop with it

The router's core decision, executed and adversarially tested

The paper's scarce-bandwidth rule is to prefill locally when l_total - l_pd <= t and offload otherwise.1 The router therefore needs an uncached-token count, not an aggregate cache-key count. The executed core below accepts that count from a cache-aware scheduler and tests the threshold boundary, link-failure fallback, and invalid input. The control plane must obtain the longest fully reusable prefix from native scheduler metadata or an API with equivalent semantics.

"""Length-threshold PrfaaS router core, executed and adversarially tested.
Accepts an uncached-token count produced by tenant-aware, longest-prefix
cache metadata. Run: python3 prfaas_router.py
"""
from __future__ import annotations

from typing import Literal

Route = Literal["local", "prfaas"]


def route_decision(uncached_tokens: int, threshold: int, link_healthy: bool) -> Route:
    """Apply the paper's scarce-bandwidth threshold to validated token counts."""
    if uncached_tokens < 0 or threshold < 0:
        raise ValueError("token counts must be non-negative")
    if not link_healthy:
        return "local"
    return "prfaas" if uncached_tokens > threshold else "local"


if __name__ == "__main__":
    THRESHOLD = 19_400  # paper case study; re-profile before deployment

    cases = [
        ("cold 44K-token request, nothing cached", 44_000, True, "prfaas"),
        ("hot agentic turn, prefix fully cached", 0, True, "local"),
        ("exactly at threshold, tie goes local", THRESHOLD, True, "local"),
        ("one token over threshold, offloads", THRESHOLD + 1, True, "prfaas"),
        ("would offload, but link is down: fail toward local", 44_000, False, "local"),
        ("partially cached long request, still over threshold", 33_760, True, "prfaas"),
        ("partially cached long request, cache brings it under threshold", 18_400, True, "local"),
    ]

    print(f"{'case':<58} {'uncached':>9} {'route':>8} {'expected':>9}")
    for label, uncached, healthy, expected in cases:
        route = route_decision(uncached, THRESHOLD, healthy)
        print(f"{label:<58} {uncached:>9,} {route:>8} {expected:>9}")
        assert route == expected, (label, route, expected)

    # Boundary and malformed-input cases.
    assert route_decision(THRESHOLD, THRESHOLD, True) == "local"
    assert route_decision(THRESHOLD + 1, THRESHOLD, True) == "prfaas"

    try:
        route_decision(-1, THRESHOLD, True)
    except ValueError:
        pass
    else:
        raise AssertionError("negative uncached length was accepted")

    print("\nALL ASSERTIONS PASSED")

Executed output:

case                                                        uncached    route  expected
cold 44K-token request, nothing cached                        44,000   prfaas    prfaas
hot agentic turn, prefix fully cached                              0    local     local
exactly at threshold, tie goes local                          19,400    local     local
one token over threshold, offloads                            19,401   prfaas    prfaas
would offload, but link is down: fail toward local            44,000    local     local
partially cached long request, still over threshold           33,760   prfaas    prfaas
partially cached long request, cache brings it under threshold    18,400    local     local

ALL ASSERTIONS PASSED

Obtaining uncached length safely

LMCache MP mode's POST /cache/prefetches and GET /cache/prefetches/{request_id} endpoints are not a routing lookup API. The operation promotes matching objects from L2 into L1, so probing candidates mutates cache state. More importantly, one token chunk expands to one object key per KV rank, and the completed response reports the aggregate number of found object keys. That aggregate does not identify the longest contiguous chunk prefix for which every rank is present. Dividing or multiplying found_keys therefore cannot produce a safe reusable-token count, especially when world_size > 1.9

Obtain l_pd from the serving scheduler's native prefix-cache match or a control-plane API that explicitly returns the longest fully reusable prefix. The lookup must use the same model and tokenizer revisions, attention layout, world size and rank grouping, chunk size, and nonempty tenant cache_salt as the cache entries. If such metadata is unavailable, use total prompt length as a conservative uncached length or disable cache-aware remote routing. Do not derive l_pd from LMCache's aggregate found_keys field.

Under abundant bandwidth, the paper compares the longest reusable prefixes across clusters, l_prefix = max(l_prfaas, l_pd), and may transfer cache when the better prefix is remote.1 That policy requires the same explicit longest-prefix semantics from both clusters; the MP prefetch-status response is insufficient.

  • Re-derive the threshold from your own profiling, not the paper's. The executed model above is the template: profile S_kv(l) and T_prefill(l) at four lengths on both hardware pools, plug in your measured request-length distribution, bisect equation 7. The 19.4K figure is one workload on one model on one hardware pair.
  • The CacheGen compression tradeoff still applies, but not as an MP-mode flag today. Centralized KV cache placement covers when the ~3.5-4.3x volume reduction CacheGen buys is worth its codec cost; as of this writing that compression path is only implemented for LMCache's deprecated in-process mode, not MP mode, so an MP-mode deployment on a constrained link has to weigh compression elsewhere in the stack (or re-check whether MP mode has since picked it up).
  • Respect hybrid-model cache semantics. Recurrent state is request-level and exact-match-only; full-attention KV is block-level and prefix-matchable. vLLM's hybrid KV cache manager (which the paper builds on) handles the split, and LMCache's hybrid-models support adds hard alignment requirements for Mamba/GDN hybrids: the LMCache --chunk-size, vLLM's --max-num-batched-tokens, and the unified block size must align, with --mamba-cache-mode align --enable-prefix-caching mandatory for GDN models, and the docs warn that generation is not bit-exact between a cached and a fresh run.6 Any custom eviction or dedup logic you add must never treat a recurrent-state entry as partially reusable.1
  • Test both NIXL failure policies. The reference recipe uses kv_load_failure_policy=fail, vLLM's default, which returns an error when KV loading fails. recompute can preserve request availability, but vLLM warns that running prefill on a decode-optimized instance adds jitter and increases tail latency for other decode requests. Select the policy from the service's error-rate and tail-latency objectives; neither setting removes the need for link-health admission and bounded transfer timeouts.

How to maintain it

  • Watch three signals per the paper's scheduler: egress-link utilization, PrfaaS queue depth, and per-stage throughput balance. Short-term congestion triggers threshold re-search; long-term drift (traffic mix, cache hit rate) triggers converting PD-cluster nodes between prefill and decode roles to restore equations 7 and 8.1 LMCache ships Prometheus metrics and OTel tracing for the cache side;5 the balance monitor is yours.
  • Re-profile on every model revision. Phi_kv is an architecture property; a new checkpoint with a different hybrid ratio, KV dtype, or attention layout moves every operating point. Store the four-point profile with the model artifact.
  • Version cache keys across the fleet, both clusters. A cross-datacenter KV entry produced by last week's weights is the same silent-corruption hazard as in any shared store; rollouts must invalidate both the local pools and anything in flight.
  • Capacity-plan the pools with the same arithmetic. S_kv(l) times expected cached tokens per cluster, plus the transfer-cache working set (in-flight requests times their tail KV), sized so transfer blocks never evict hot prefix blocks.

How to run it in production

  • Enforce the three gates at admission, not in a design doc. A config flag that lets dense-model traffic onto the PrfaaS path recreates the 60 Gb/s-per-instance egress problem instantly. Gate offload eligibility on model identity and measured Phi_kv.
  • Smooth the link, do not just size it. The paper's transport stack is layer-wise prefill pipelining (KV of layer n transfers while layer n+1 computes), multi-connection TCP to fill the pipe, and congestion monitoring wired into the scheduler so routing backs off before queues build.1 A single-stream TCP transfer at 100 Gb/s over a real RTT will not fill the link on its own.
  • Keep decode SLO isolation. The decode pool's BS_max is SLO-governed (the case study: 40 tok/s per request, giving BS_max = 20 on their hardware); PrfaaS increases arrival at the decode pool, so admission control must cap it, or TTFT wins convert to TPOT losses.
  • Treat the inter-DC link as hostile. KV crosses an organizational boundary; VPC peering or dedicated lines still warrant encryption in transit and tenant isolation in the shared pools. Cache entries are model activations; leaking them is leaking the context.
  • Keep a local serving path. Link-down, remote-queue-full, and pre-routing timeout conditions should select local prefill before remote work is admitted. A transfer that fails after admission follows the configured NIXL policy: return an error with fail, or accept the documented tail-latency risk with recompute. Test both paths under load.

Failure modes

  • Naive offload. All prefill remote, no threshold: 1.16x instead of 1.54x in the paper's own comparison, plus a congested link. The threshold is the architecture.3
  • Routing on total instead of uncached length. Agentic traffic is dominated by incremental prefills over cached prefixes; ignoring cache state over-offloads and re-transfers bytes the local pool already held.
  • Dense model on the hybrid path. One 8-GPU dense instance at 32K emits ~60 Gb/s of KV; a handful saturates the link and stalls every in-flight transfer behind it.2
  • Headline misquote. Quoting +54% without "model-derived, heterogeneous-hardware, ~15% at equal cost" overstates the case; the honest pitch is TTFT and elasticity, with throughput contingent on hardware pricing.3
  • Recurrent-state mishandling. Treating request-level linear-attention state as block-matchable prefix cache serves wrong-context state on a partial hit; the hybrid pool split exists precisely to prevent this.1
  • Transfer blocks evicting prefix blocks. Under burst, in-flight transfer-cache can push out hot shared prefixes, trading a one-time transfer win for a fleet-wide hit-rate collapse; partition the pools as the paper does.1
  • Trusting lmcache --help (or any subcommand's --help) to run cleanly out of the box. At lmcache==0.5.1, the CLI eagerly imports every registered subcommand module before parsing arguments, so a completely unrelated subcommand's missing optional dependency (bench/engine_bench importing openai.OpenAI) breaks top-level --help and server --help alike with ModuleNotFoundError: No module named 'openai'. Confirmed by direct execution, not assumed. pip install openai (undeclared as a dependency) is the workaround; do not assume a bare pip install lmcache gives you a working CLI without it.
  • Assuming --l2-adapter type=nixl_store is a real configuration. It is not, at this version; NIXL is configured under --p2p-transfer-engine/--p2p-advertise-url, a separate flag group from --l2-adapter. A deployment script written against the adapter-type list this page previously carried would fail at argument-parse time.

References

  • PrfaaS: R. Qin, W. He, Y. Wang, Z. Li, X. Xu, Y. Wu, W. Zheng, M. Zhang, "Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter," arXiv 2604.15039 (preprint, Moonshot AI and Tsinghua). https://arxiv.org/abs/2604.15039
  • LMCache documentation (MP mode, disaggregated prefill, configuration reference): https://docs.lmcache.ai/
  • LMCache repository: https://github.com/LMCache/LMCache
  • LMCache MP-mode warm-prefetch HTTP API (POST /cache/prefetches, GET /cache/prefetches/{id}); this is not a longest-prefix routing lookup: https://github.com/LMCache/LMCache/blob/76744ce2514f3444e9a6fb06d67b23739fa93b51/lmcache/v1/multiprocess/http_apis/cache_api.py
  • CacheGen (the cachegen serde): Y. Liu et al., arXiv 2310.07240 (SIGCOMM'24). https://arxiv.org/abs/2310.07240
  • CacheBlend (the blending feature): J. Yao et al., arXiv 2405.16444 (EuroSys'25). https://arxiv.org/abs/2405.16444
  • Mooncake (the lineage system and an LMCache L2 backend): R. Qin et al., arXiv 2407.00079 (FAST'25). https://arxiv.org/abs/2407.00079
  • Kimi Linear (the case-study model architecture): arXiv 2510.26692. https://arxiv.org/abs/2510.26692
  • "Hybrid Models as First-Class Citizens in vLLM" (PyTorch blog): https://pytorch.org/blog/hybrid-models-as-first-class-citizens-in-vllm/ · vLLM hybrid KV cache manager design doc: https://docs.vllm.ai/en/stable/design/hybrid_kv_cache_manager/
  • LMCache hybrid attention models guide: https://docs.lmcache.ai/mp/hybrid_models.html

Related: Centralized KV Cache Placement · Disaggregated Inference · Disaggregation Rate Matching · KV Cache Transfer (NIXL) · KV Cache Management · Datacenter Interconnect (DCI) · Non-Colocated Inference: Which Pattern? · Cross-WAN Model-Parallel Inference · QoS & Admission Control · Tenant Cache Isolation · Speculative Decoding · Glossary

  1. arXiv 2604.15039, sections 3.1-3.4: PrfaaS/local-PD cluster split with intra-cluster RDMA and inter-cluster VPC peering or dedicated lines; length-threshold routing on incremental (uncached) prefill length; hybrid prefix cache pool with separate KV groups over a shared block pool, prefix-cache vs transfer-cache block classes, built on vLLM's hybrid KVCache manager; layer-wise prefill pipelining, multi-connection TCP, congestion monitoring; bandwidth-scarce vs bandwidth-abundant routing rules; dual-timescale scheduling restoring equations 7 and 8. Read in full on 2026-07-15. ↩↩↩↩↩↩↩↩↩↩↩

  2. arXiv 2604.15039, section 2 (Tables 1-3): hybrid ratios (Kimi Linear 3:1 KDA:MLA, MiMo-V2-Flash 5:1 SWA:GQA, Qwen3.5-397B 3:1 GDN:GQA, Ring-2.5-1T 7:1 Lightning:MLA); KV throughput at 32K on 8xH200 with SGLang v0.5.9: MiniMax-M2.5 59.93 Gbps and Qwen3-235B 33.35 Gbps (dense) vs Kimi Linear 3.87, MiMo-V2-Flash 4.66, Qwen3.5-397B 8.25, Ring-2.5-1T 2.59 Gbps (hybrid); 512-H200 dense egress 3.8 Tbps (MiniMax-M2.5) / 2.1 Tbps (Qwen3); Ring-2.5-1T ~170 Gbps at 32K average, under 100 Gbps when routing only 128K-class requests; ~1.8 Tbps for a 10,000-GPU deployment. ↩↩↩

  3. arXiv 2604.15039, section 4: 32 H200 (4 instances) + 64 H20 (8 instances) vs 96 H20; internal 1T Kimi Linear-style model, truncated log-normal input lengths (mu 9.90, sigma 1.00, [128, 128K], mean ~27K), 1024-token outputs, 40 tok/s SLO. Optimal t = 19.4K, Np/Nd = 3/5; 49.6% of requests offloaded at E[L|L>t] ~ 44K; ~13 Gbps average egress (13% of the 100 Gbps link); Lambda_max 3.24 vs 2.11 req/s (+54%, or ~15% at equal cost per section 4.4); mean/P90 TTFT 2.22/3.51 s vs 4.44/9.73 s; naive heterogeneous PD (all prefill remote, no scheduling) reaches only 1.16x. The paper states all throughput and bandwidth results "are derived by feeding the measured profiling data into the throughput model"; they are model-derived from real profiling, not an end-to-end deployed-system measurement. Vendor-evaluated preprint, not independently reproduced. ↩↩↩↩↩↩

  4. docs.lmcache.ai (fetched 2026-07-15): "a KV cache management layer for LLM inference"; storage backends CPU RAM, local SSD, Redis/Valkey, S3, Bigtable, Mooncake, NIXL, Aerospike; CacheGen compression; CacheBlend non-prefix reuse; P2P sharing; PD disaggregation over NVLink, RDMA, or TCP. ↩

  5. docs.lmcache.ai MP-mode pages (fetched 2026-07-15): lmcache server as a standalone ZMQ service with FastAPI HTTP frontend, one per node serving multiple vLLM pods; process isolation, shared L1, independent CPU-memory scaling; L1 CPU DRAM or NVMe via GPUDirect Storage, L2 S3/Bigtable/Redis/filesystem/NIXL; disaggregated prefill via NIXL with MultiConnector combining NixlConnector and LMCacheMPConnector (lmcache.mp.host/lmcache.mp.port extra-config keys, VLLM_NIXL_SIDE_CHANNEL_HOST/PORT for the handshake); Prometheus metrics and OTel tracing; the in-process mode is marked deprecated in favor of MP mode. ↩↩

  6. docs.lmcache.ai/mp/hybrid_models.html (fetched 2026-07-15): validated hybrid architectures include Gemma 3/4, gpt-oss (SWA+full), Qwen3.5/3.6 (Mamba/GDN+full), DeepSeek-V4-Flash (sparse-MLA), GLM 5.1/5.2, MiniMax-M3; for Mamba/linear hybrids the LMCache server --chunk-size N and vLLM --max-num-batched-tokens (in [N, 2N)) must derive from the unified block size, with --mamba-cache-mode align --enable-prefix-caching mandatory for GDN; linear-attention layers keep a recurrent state cache that LMCache treats as opaque pages; "generation is not bit-exact between a cached and a fresh run"; only text KV is validated, and DeepSeek-V4-style compressed/indexer caches are not yet handled by the multiprocess connector. ↩

  7. lmcache server --help executed directly against the installed lmcache==0.5.1 package (2026-07-16), superseding an earlier docs-only citation: --chunk-size (default 256, tokens); --hash-algorithm (default blake3; builtin/sha256_cbor also valid); --l1-size-gb (required, sizes the pinned-DRAM L1 or, with --gds-l1-path set, the GDS slab file); --gds-l1-path (unset by default, opt-in NVMe-via-GPUDirect-Storage L1); --eviction-policy (required: LRU, IsolatedLRU, or noop); --l2-adapter <JSON> (repeatable, unset by default, type field selects one of aerospike/dax/fault_inject/fs/fs_native/hfbucket/mock/mooncake_store/native_plugin/p2p/plugin/raw_block/resp/s3, multiple flags cascade in order; nixl_store is not a valid value at this version); --p2p-transfer-engine (default nixl, separate from --l2-adapter); --engine-type (default, blend, or blend_legacy; no CacheGen flag present at this version); --coordinator-url/--coordinator-l2-event-reporting (fleet-wide L2 usage/eviction sharing). The older chunk_size/local_cpu/remote_url/remote_serde/enable_blending-style YAML keys and LMCACHE_-prefixed environment variables belong to the deprecated in-process connector mode and do not apply to an MP-mode deployment. ↩

  8. LMCache docs/source/mp/disaggregated_prefill.rst at commit 76744ce2514f3444e9a6fb06d67b23739fa93b51, lines 50-61 and 103-145: the guide requires vLLM PR 46865 because offload otherwise silently never triggers under MultiConnector, and requires nixl>=1.3.0. The snapshot points to nightly tags; LMCache also publishes stable releases such as v0.5.1, but this page does not assert that v0.5.1 contains the newer integration. https://github.com/LMCache/LMCache/blob/76744ce2514f3444e9a6fb06d67b23739fa93b51/docs/source/mp/disaggregated_prefill.rst ↩

  9. LMCache ipc_key_to_object_keys, key_resolver.py, prefetch_service.py, and cache_api.py at commit 76744ce2514f3444e9a6fb06d67b23739fa93b51: each token chunk expands to one ObjectKey per kv_rank; found_keys is the population count of found object keys, not the number of complete contiguous chunks. The API does not return which chunk/rank pairs were found, so the aggregate cannot establish the longest reusable prefix. POST /cache/prefetches is also a warm L2-to-L1 promotion, not a side-effect-free read. https://github.com/LMCache/LMCache/blob/76744ce2514f3444e9a6fb06d67b23739fa93b51/lmcache/v1/multiprocess/cache_control/prefetch_service.py ↩