Markdown

Region-replica routing for non-colocated inference¶

Scope: serving one logical LLM inference workload, a vLLM/SGLang/TensorRT-LLM-class engine, across GPUs that are not colocated (spread across regions, clouds, or providers) by running a complete, ordinary, co-located replica at every site and routing each request to the best one. Covers the cross-region proxy's KV-cache-locality, load, and network-aware routing decision; how it differs from the intra-datacenter case in KV cache transfer with NIXL; and an executed model of when fetching a cached prefix cross-region beats local recompute. When no single site can hold a full replica, use cross-WAN model-parallel inference instead. This is the inference-serving counterpart to Recipe: DiLoCo (geo-distributed) for training; the WAN transport substrate is Overlay & mesh networking.

Everything under Inference serving, Inference parallelism strategies, and Disaggregated inference assumes GPUs sit in one fast-fabric domain; those pages say so explicitly (TP groups stay inside an NVLink island, prefill/decode pools stay "within fast-interconnect reach"). This page is what to do when GPUs are spread across regions or providers but a full replica still fits at every site you have: it is production-composable from bricks this KB already documents (engines, disaggregation, routing, overlay networking), unlike the research-stage answer for when no site can hold a full replica, cross-WAN model-parallel inference. Treat the two as different tools for different failure conditions, not a maturity ladder.

flowchart TB
  Q{"Does a full replica fit<br/>on GPUs at one site?"}
  Q -->|"yes, at every site you have"| A["Pattern A: region-replica routing<br/>(production-composable)"]
  Q -->|"no single site can hold a full replica"| B["Pattern B: cross-WAN model-parallel split<br/>(research-stage)"]
  A --> A1["Each region: an ordinary co-located<br/>disaggregated-inference deployment"]
  A1 --> A2["Global proxy: KV-locality + load +<br/>network-aware routing across regions"]
  B --> B1["Layers/experts sharded across<br/>non-colocated nodes, pipeline-parallel over WAN"]
  B1 --> B2["Each node keeps only its own layers' KV;<br/>activations cross the WAN hop, not the cache"]

This page covers Pattern A (the top branch): read on if a full replica fits at every site you have. If no single site can hold a full replica, see cross-WAN model-parallel inference for Pattern B instead.

What it is¶

Each site (region, cloud, or provider) runs a complete, ordinary, co-located deployment following inference serving and disaggregated inference: TP/PP/EP inside the site's own fast fabric, prefill and decode pools on the same rack or NVLink island. A cross-region-aware proxy sits in front of all sites and decides, per request, which region serves it, factoring in KV-cache locality (does this region already hold a warm prefix cache for this session), current queue depth, and inter-region network latency. GORGO (arXiv 2602.11688) names this problem precisely: "LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency."¹ By default the proxy redirects the request to whichever region already holds the warm cache, so nothing but the request and response crosses the WAN; the model, its weights, and the KV cache stay put in the region that computes them.

That default has a real limit: redirecting is cheap per request, but for a multi-turn conversation it taxes every future turn with a cross-region request/response round trip for as long as the session's cache stays put in the region it started in. A proxy that wants to migrate a long-lived session closer to its client (so every subsequent turn is fast, not just this one), or that cannot redirect at all because the client's connection is already pinned to a specific region (a data-residency or compliance requirement, a CDN/edge terminator that already committed the connection), has to pay a one-time cost instead: fetch the cached prefix from wherever it lives, or recompute it locally in the region that will serve the session from now on. That fetch-or-recompute tradeoff, not redirect, is what the executed model below makes quantitative. Treat these as two different actions with two different costs: redirecting moves nothing but the request/response and is the default, cheap answer for a single request; fetching moves the KV cache itself over the WAN and is a session-migration decision, made once, not a per-request routing choice.

This is one of two structurally different ways to serve one inference workload when GPUs are not colocated. This pattern assumes capacity exists at each site to hold a full replica and optimizes which replica serves a request. The other, cross-WAN model-parallel inference, assumes capacity is fundamentally scattered (no site has enough) and optimizes how to assemble a replica out of pieces that live far apart.

Why it matters¶

Economics. The same argument recipe-diloco-geo-distributed.md makes for training applies to serving: usable GPU capacity is not always colocated, and neocloud or volunteer pools are often cheaper than a single centralized cluster with the headroom to hold a full replica everywhere.
User-facing latency. Region-replica routing is also how you put inference near users worldwide; a request answered by a replica in the requester's region has a lower network hop than one served cross-continent, on top of whatever the model itself takes.
The KV cache is the crux, and it is a harder problem here than in training. DiLoCo's cross-worker payload is a pseudo-gradient exchanged every H steps, a fixed, small, infrequent cost. Inference's equivalent state, the KV cache, is large (hundreds of KB per token, see KV cache transfer with NIXL), grows continuously for the life of a request, and the whole point of disaggregation and prefix caching is to keep reusing it cheaply. Moving that state over a WAN link is a fundamentally different cost proposition than moving it over NVLink or IB, which is why this page treats the fetch-versus-recompute decision as load-bearing, not an afterthought.

When to use it (and when not)¶

Prefer this pattern whenever a full replica fits at every site you actually have. It reuses every production-proven piece this KB already documents (engines, disaggregation, SLOs, QoS) unchanged; the only new component is the cross-region proxy.
Reach for cross-WAN model-parallel inference only when no site can host a full replica and consolidating capacity (renting a bigger instance, waiting, or serving a smaller/quantized model instead) is not an option; that pattern carries a real throughput and latency ceiling this one does not.
Never split prefill and decode across non-colocated sites. Disaggregated inference already states the rule for the intra-datacenter case ("keep prefill and decode pools within fast-interconnect reach, same rail/rack where possible"); across a WAN link the KV-cache handoff between an unpaired prefill and decode site is strictly worse than either this pattern (recompute or cache locally) or cross-WAN model-parallel inference (never move the KV cache at all). Disaggregation and cross-site serving solve different problems; do not combine them naively.
Do not put the inference hot path on a WAN overlay meant for control traffic. Overlay & mesh networking is built and validated for a training rendezvous and low-frequency DiLoCo sync, not for a synchronous per-token decode loop; reusing it for cross-region request/response traffic, as this pattern does, is fine. Reusing it for a step-synchronous collective (a naive attempt at cross-WAN model-parallel splitting) will stall exactly as it does for training; see cross-WAN model-parallel inference for why.

Architecture¶

flowchart TB
  C["Client (any region)"] --> GP["Global proxy / GSLB<br/>(KV-locality + load + RTT aware)"]
  GP -->|"warm prefix cached here, or lowest cost"| R1["Region 1: full replica<br/>(disaggregated-inference.md deployment)"]
  GP -->|"session/prefix elsewhere, or R1 saturated"| R2["Region 2: full replica"]
  R1 -.->|"control plane only: health, metrics,<br/>model sync (WireGuard overlay)"| R2

How to use it: routing on KV-cache locality versus network cost¶

Redirecting to the cache-holding region is the default for a single request, and almost always the cheapest option for it: a repeat round trip costs far less than moving hundreds of KB per token of KV cache. The model below instead answers the session-migration question that applies once a proxy decides it wants a long-lived session served from a different region going forward, whether to reduce the per-turn round trip for the rest of the conversation or because the client's connection is pinned there for other reasons (see What it is, above): given a WAN link's bandwidth and RTT, and a model's KV-cache footprint and prefill compute cost, is it cheaper for the new serving region to fetch the cached KV cross-region or to recompute the prefix locally? This is the part of GORGO's "jointly account for KV-cache locality, replica load, and variable network latency" problem that applies to that migration decision, not to routing any single request.¹

# wan_kv_crossover.py -- validated: does fetching a cached KV prefix across a WAN
# region beat recomputing prefill locally, and at what prefix length? Run: python3 wan_kv_crossover.py
from __future__ import annotations


def bytes_per_kv_token(n_layers: int, n_kv_heads: int, head_dim: int, bytes_per_element: int = 2) -> float:
    # bytes_per_token = 2 x n_layers x n_kv_heads x head_dim x bytes_per_element (K and V); see kv-cache-transfer-nixl.md
    return 2 * n_layers * n_kv_heads * head_dim * bytes_per_element


def recompute_seconds(prefix_tokens: int, params: float, tflops: float, mfu: float) -> float:
    # forward-only FLOPs/token ~= 2 x params (Kaplan et al. 2020, C=6ND; forward pass is ~2ND); prefill is compute-bound.
    flops_per_token = 2.0 * params
    return prefix_tokens * flops_per_token / (tflops * mfu)


def fetch_seconds(prefix_tokens: int, bytes_per_token: float, bandwidth_gbps: float, rtt_ms: float) -> float:
    payload_bits = prefix_tokens * bytes_per_token * 8
    return rtt_ms / 1000.0 + payload_bits / (bandwidth_gbps * 1e9)


def crossover_tokens(bytes_per_token: float, bandwidth_gbps: float, rtt_ms: float,
                      params: float, tflops: float, mfu: float) -> float | None:
    """Prefix length above which fetching the cached KV cross-region beats local recompute.
    None means recompute's per-token cost never exceeds transfer's, so fetch never wins."""
    recompute_per_token = 2.0 * params / (tflops * mfu)
    transfer_per_token = bytes_per_token * 8 / (bandwidth_gbps * 1e9)
    if recompute_per_token <= transfer_per_token:
        return None
    return (rtt_ms / 1000.0) / (recompute_per_token - transfer_per_token)


if __name__ == "__main__":
    # Llama-3.1-70B-class dense model: 80 layers, 8 KV heads (GQA), head_dim 128, BF16 KV cache.
    bpt = bytes_per_kv_token(n_layers=80, n_kv_heads=8, head_dim=128)
    assert bpt == 327680.0, bpt  # 320 KiB/token

    params = 70e9
    tflops = 989e12   # H100 SXM dense BF16 (half NVIDIA's published 1,979 TFLOPS sparse figure), see kv-cache-inference-speedup.md
    mfu = 0.40         # illustrative prefill MFU, not hardware-measured here
    rtt_ms = 70.0      # illustrative cross-region RTT

    recompute_per_token_us = 2 * params / (tflops * mfu) * 1e6

    # Regime 1: public-internet-grade cross-region link (5 Gbps effective goodput).
    weak_gbps = 5.0
    weak_transfer_per_token_us = bpt * 8 / (weak_gbps * 1e9) * 1e6
    L_weak = crossover_tokens(bpt, weak_gbps, rtt_ms, params, tflops, mfu)
    assert L_weak is None, L_weak                 # transfer per-token cost exceeds recompute: fetch never wins
    assert weak_transfer_per_token_us > recompute_per_token_us

    # Regime 2: well-provisioned private inter-region backbone (100 Gbps).
    fast_gbps = 100.0
    fast_transfer_per_token_us = bpt * 8 / (fast_gbps * 1e9) * 1e6
    L_fast = crossover_tokens(bpt, fast_gbps, rtt_ms, params, tflops, mfu)
    assert L_fast is not None and L_fast > 0
    assert fast_transfer_per_token_us < recompute_per_token_us

    # Below the crossover, local recompute must be cheaper than the fetch; above it, the reverse.
    short_prefix = int(L_fast * 0.5)
    long_prefix = int(L_fast * 2.0)
    assert recompute_seconds(short_prefix, params, tflops, mfu) < fetch_seconds(short_prefix, bpt, fast_gbps, rtt_ms)
    assert fetch_seconds(long_prefix, bpt, fast_gbps, rtt_ms) < recompute_seconds(long_prefix, params, tflops, mfu)

    # Doubling RTT must push the crossover higher (a fetch has more fixed cost to amortize).
    L_fast_slower_rtt = crossover_tokens(bpt, fast_gbps, rtt_ms * 2, params, tflops, mfu)
    assert L_fast_slower_rtt > L_fast

    print(f"bytes/token                       = {bpt/1024:.1f} KiB")
    print(f"recompute rate                     = {recompute_per_token_us:.2f} us/token")
    print(f"transfer rate @5 Gbps (public WAN)  = {weak_transfer_per_token_us:.2f} us/token -> fetch never wins (crossover: none)")
    print(f"transfer rate @100 Gbps (backbone)  = {fast_transfer_per_token_us:.2f} us/token -> crossover at {L_fast:,.0f} tokens")
    print(f"crossover @100Gbps, 2x RTT ({rtt_ms*2:.0f}ms) = {L_fast_slower_rtt:,.0f} tokens")
    print(f"recompute@1k tokens (any link)      = {recompute_seconds(1000, params, tflops, mfu)*1000:.2f} ms")
    print(f"fetch@1k tokens @100Gbps             = {fetch_seconds(1000, bpt, fast_gbps, rtt_ms)*1000:.2f} ms")
    print(f"recompute@8k tokens (any link)       = {recompute_seconds(8000, params, tflops, mfu)*1000:.2f} ms")
    print(f"fetch@8k tokens @100Gbps             = {fetch_seconds(8000, bpt, fast_gbps, rtt_ms)*1000:.2f} ms")
    print("ALL ASSERTIONS PASSED")

Executed output:

bytes/token                       = 320.0 KiB
recompute rate                     = 353.89 us/token
transfer rate @5 Gbps (public WAN)  = 524.29 us/token -> fetch never wins (crossover: none)
transfer rate @100 Gbps (backbone)  = 26.21 us/token -> crossover at 214 tokens
crossover @100Gbps, 2x RTT (140ms) = 427 tokens
recompute@1k tokens (any link)      = 353.89 ms
fetch@1k tokens @100Gbps             = 96.21 ms
recompute@8k tokens (any link)       = 2831.14 ms
fetch@8k tokens @100Gbps             = 279.72 ms
ALL ASSERTIONS PASSED

The result is not a fixed rule, it is a decision that depends entirely on the actual link: over an ordinary public-internet-grade cross-region path (5 Gbps illustrative), a 70B-class dense model's KV cache is expensive enough per byte that local recompute wins outright, at every prefix length, no crossover exists. Over a well-provisioned private inter-region backbone (100 Gbps illustrative), fetching becomes cheaper once the cached prefix passes a few hundred tokens, and dominates heavily for long prefixes (8k tokens: 280ms fetched versus 2.8s recomputed). This is why a real cross-region router cannot use a static policy; it has to measure the path it actually has (bandwidth, RTT) and the prefix length in front of it, exactly the "tunable parameters" GORGO's cost model exists to set.¹ Swap in your model's real layer count, KV-head count, head dimension, and measured MFU and RTT before trusting a number from this script on your own deployment; the tflops figure is a cited H100 spec, everything else here is illustrative.

How to develop with it: building the cross-region KV-aware router¶

The proxy is llm-request-routing.md's router pattern (a gateway/proxy classifies and dispatches, keeping its own latency far below the generation it gates) generalized from "which model" to "which region." Three signals it needs that a same-datacenter router does not:

KV/prefix locality per replica. Track, per active session or per prefix hash, which region's replica already holds the warm cache (the same idea as SGLang's RadixAttention prefix reuse in inference serving, extended from one engine's prefix tree to a directory across regions). A consistent-hash on a stable session key is the simplest version; a shared prefix-radix index across regions is the fuller one.
Live inter-region network state. RTT and available bandwidth between the proxy and each region drift (congestion, provider changes, time of day); measure them, do not hard-code them. The crossover model above is only as good as these two live inputs.
Per-region queue depth. A region with a warm cache but a saturated decode pool is not necessarily the best redirect target; combine KV locality with the same admission-control signal inference QoS and admission control already tracks locally, now compared across regions, to prefer the least-loaded warm candidate over a saturated one, or a cold region over an overloaded warm one. The section below answers a related but different question, once a proxy has decided to migrate a session's serving region rather than redirect it, does congestion on the fetch path itself (not the compute pool a plain redirect would land on) change whether fetching or recomputing is cheaper.

Combining locality, network cost, and load into one routing score¶

The crossover model above answers "fetch or recompute" for a session migration at a fixed load. It leaves GORGO's third factor, queueing delay, out of the number entirely, exactly the gap the router bullet above flags: fetching and recomputing do not contend for the same resource at the destination region, recomputing consumes that region's own GPU/prefill-compute queue, fetching consumes the cross-region network path's queue (a KV-transfer link or KV-cache store, potentially shared by many concurrent migrations), so either one can be busy independently of the other. The model below adds a queueing-delay term to each side of the same crossover model and asks a sharper question than "which is cheaper in isolation": at what load on the fetch path does recomputing locally win even though fetching would be cheaper over an idle link?

# wan_routing_score.py -- validated: extends wan_kv_crossover.py with an M/M/1 expected-wait
# term for per-region queue depth, operationalizing GORGO's "holistically factors network
# latency, prefill cost, and queueing delay" cost model into one routing decision instead of
# three separate signals. Run: python3 wan_routing_score.py
from __future__ import annotations


def bytes_per_kv_token(n_layers: int, n_kv_heads: int, head_dim: int, bytes_per_element: int = 2) -> float:
    return 2 * n_layers * n_kv_heads * head_dim * bytes_per_element


def recompute_seconds(prefix_tokens: int, params: float, tflops: float, mfu: float) -> float:
    flops_per_token = 2.0 * params
    return prefix_tokens * flops_per_token / (tflops * mfu)


def fetch_seconds(prefix_tokens: int, bytes_per_token: float, bandwidth_gbps: float, rtt_ms: float) -> float:
    payload_bits = prefix_tokens * bytes_per_token * 8
    return rtt_ms / 1000.0 + payload_bits / (bandwidth_gbps * 1e9)


def mm1_wait_seconds(utilization: float, service_rate_per_s: float) -> float:
    """Expected queueing wait (time in queue, not counting own service) for an M/M/1
    queue at utilization rho and service rate mu. Wq = rho / (mu * (1 - rho)), the
    standard result for a memoryless single-server queue."""
    assert 0.0 <= utilization < 1.0, utilization
    if utilization == 0.0:
        return 0.0
    return utilization / (service_rate_per_s * (1.0 - utilization))


def route_total_latency(kv_cost_seconds: float, utilization: float, service_rate_per_s: float) -> float:
    return kv_cost_seconds + mm1_wait_seconds(utilization, service_rate_per_s)


def crossover_utilization(bpt: float, bandwidth_gbps: float, rtt_ms: float,
                           params: float, tflops: float, mfu: float,
                           prefix_tokens: int, idle_region_util: float,
                           service_rate_per_s: float) -> float | None:
    """Utilization of the cross-region fetch path above which recomputing locally, on
    the destination region's own (here, less-loaded) compute queue, wins despite
    fetching's raw cost advantage over an idle link. idle_region_util is the compute
    queue's utilization, deliberately a different queue from the fetch path being swept."""
    fetch_cost = fetch_seconds(prefix_tokens, bpt, bandwidth_gbps, rtt_ms)
    recompute_cost = recompute_seconds(prefix_tokens, params, tflops, mfu)
    idle_wait = mm1_wait_seconds(idle_region_util, service_rate_per_s)
    target = recompute_cost + idle_wait - fetch_cost
    if target <= 0:
        return None
    lo, hi = 0.0, 0.999999
    for _ in range(80):
        mid = 0.5 * (lo + hi)
        if mm1_wait_seconds(mid, service_rate_per_s) >= target:
            hi = mid
        else:
            lo = mid
    return hi


if __name__ == "__main__":
    bpt = bytes_per_kv_token(n_layers=80, n_kv_heads=8, head_dim=128)
    params, tflops, mfu = 70e9, 989e12, 0.40
    fast_gbps, rtt_ms = 100.0, 70.0
    prefix_tokens = 600
    service_rate = 20.0  # requests/s either queue (fetch path or compute pool) drains, illustrative, same rate for both for simplicity

    fetch_cost = fetch_seconds(prefix_tokens, bpt, fast_gbps, rtt_ms)
    recompute_cost = recompute_seconds(prefix_tokens, params, tflops, mfu)
    print(f"fetch_cost={fetch_cost*1000:.2f}ms recompute_cost={recompute_cost*1000:.2f}ms")

    # 1) monotone increasing, diverges as rho -> 1
    waits = [mm1_wait_seconds(r, service_rate) for r in (0.0, 0.5, 0.8, 0.95, 0.99)]
    assert waits == sorted(waits), waits
    assert mm1_wait_seconds(0.0, service_rate) == 0.0
    assert mm1_wait_seconds(0.99, service_rate) > mm1_wait_seconds(0.5, service_rate) * 5

    # 2) equal utilization on both queues: KV-cache footprint/network cost alone decides,
    #    reduces to the existing crossover result (fetch wins for this fast-backbone regime).
    same_util = 0.5
    r1 = route_total_latency(fetch_cost, same_util, service_rate)
    r2 = route_total_latency(recompute_cost, same_util, service_rate)
    assert r1 < r2, (r1, r2)

    # 3) crossover utilization: the fetch path gets congested enough that recomputing on
    #    the destination region's own, less-loaded compute queue wins despite paying full recompute.
    idle_util = 0.1
    xu = crossover_utilization(bpt, fast_gbps, rtt_ms, params, tflops, mfu,
                                prefix_tokens, idle_util, service_rate)
    assert xu is not None and 0.0 < xu < 1.0, xu
    below_util = xu * 0.95
    above_util = xu + (1.0 - xu) * 0.5
    below = route_total_latency(fetch_cost, below_util, service_rate)
    above = route_total_latency(fetch_cost, above_util, service_rate)
    idle_total = recompute_cost + mm1_wait_seconds(idle_util, service_rate)
    assert below < idle_total < above, (below, idle_total, above)
    print(f"crossover utilization (prefix={prefix_tokens}, idle_util={idle_util}) = {xu:.4f}")

    # 4) adversarial: a 1.5x RTT increase (fetch gets relatively worse) must LOWER the crossover
    #    utilization (the fetch path tips over to losing at a lower load).
    xu_slow_rtt = crossover_utilization(bpt, fast_gbps, rtt_ms * 1.5, params, tflops, mfu,
                                         prefix_tokens, idle_util, service_rate)
    assert xu_slow_rtt is not None and xu_slow_rtt < xu, (xu_slow_rtt, xu)
    print(f"crossover utilization at 1.5x RTT ({rtt_ms*1.5:.0f}ms) = {xu_slow_rtt:.4f} (lower, as expected)")

    print("ALL ASSERTIONS PASSED")

Executed output:

fetch_cost=85.73ms recompute_cost=212.34ms
crossover utilization (prefix=600, idle_util=0.1) = 0.7255
crossover utilization at 1.5x RTT (105ms) = 0.6602 (lower, as expected)
ALL ASSERTIONS PASSED

At this illustrative 600-token prefix and 100 Gbps backbone, fetching wins the crossover model above outright at equal load on both queues (fetch costs 85.7ms against a 212.3ms recompute), the same result the simpler model already gives. Adding load changes the answer: once the cross-region fetch path's utilization crosses about 0.73 (with the destination region's own compute queue at a mostly-idle 0.1 utilization and this illustrative 20 req/s service rate), the queueing delay piling up on the fetch path erases its cost advantage, and recomputing locally on the region's own, idle compute queue is faster in expectation. That threshold is not fixed either: a slower cross-region link (1.5x RTT here) drags it down to about 0.66, because a relatively worse fetch has less advantage to spend before queueing on that path erases it. This is the same load-bearing property the simpler crossover model has, stated as an explicit decision rule instead of a qualitative "combine load with locality": re-measure both the network path and the destination's own compute queue, do not bake either into a static policy, and use a utilization threshold like the one above, not KV-cache footprint alone, to decide whether a session migration should fetch or recompute.

How to maintain it¶

Watch the same overlay health signals as the training case: per-peer handshake age, keepalive, and measured RTT/goodput (Overlay & mesh networking), since this pattern's control plane (health, metrics, model sync) rides the same kind of WAN link the training case documents.
Track KV-cache hit rate per region, not just globally. A falling regional hit rate is the leading indicator that the router's locality signal has drifted (traffic mix shifted, a region was added or drained) before TTFT/TPOT SLOs (see SLOs: inference serving) show it.
Re-measure the crossover, do not bake it in. Cloud inter-region bandwidth and RTT are not constants; re-run the fetch-versus-recompute comparison above, and the utilization threshold from the composite score, whenever a region pairing, provider, or traffic pattern changes.

Running it in production¶

	Pattern A: region-replica routing	Pattern B: cross-WAN model-parallel split
Maturity	Production-composable from pieces this KB already documents (engines, disaggregation, routing, overlay networking); the composition itself is not a single vendor-shipped product as of this writing	Research-stage: published, working, open-source systems in the Petals/HexGen/Helix/Parallax lineage, not a supported mode of vLLM/SGLang/TensorRT-LLM/Dynamo
What moves over WAN	By default, requests and responses only, the proxy redirects to wherever the cache lives; a session migrated to reduce a long conversation's per-turn round trip, or pinned by a redirect-blocking constraint, additionally moves the KV cache itself, once, to its new serving region	Per-token activations between pipeline stages; KV cache never leaves the node that owns those layers
Latency ceiling	Roughly the same as any single region's deployment, plus one WAN hop for the request/response	Grows with pipeline depth: each stage adds one more hop to the single round trip every committed token needs (not a separate RTT per stage); ranges from about 1 decoding step/s without WAN-aware speculative decoding to an order of magnitude or more higher with it, at larger model scale, in at least one documented implementation
When it applies	A full replica fits at every site you have	No site can host a full replica; capacity is fundamentally scattered
Where the KV-cache decision lives	The cross-region proxy (a GORGO-style holistic cost model)¹	Not a decision: each node's KV is local to its own layers by construction

Failure modes¶

Splitting prefill and decode across non-colocated sites. Reintroduces a WAN KV-cache handoff exactly where disaggregated inference says to keep it on a fast fabric; slower than either colocating prefill/decode per region (this pattern) or not moving the KV cache at all (cross-WAN model-parallel inference).
A static cross-region routing policy. Bandwidth and RTT between regions drift; a router tuned once will misroute as the path changes. Re-measure, do not hard-code (see How to maintain it, above).
Fetching cached KV cross-region on a weak link, or fetching at all when a plain redirect was available. As the executed model above shows, on an ordinary public-internet-grade path fetching is often strictly worse than local recompute; verify the crossover on your actual link before wiring cross-region KV fetch into a session migration. And check redirect first: for a single request, redirecting to the cache-holding region is almost always cheaper than either fetching or recomputing, since it moves no KV bytes at all; fetch or recompute only when redirect is not the goal (see What it is, above).
Ignoring the fetch path's own queue depth in favor of its raw cost advantage alone. A cross-region link that is cheap when idle is not necessarily the best choice once it is loaded; the composite routing score above shows this concretely, the fetch path can cross a load threshold (about 0.73 utilization in the illustrative case above) past which recomputing on the destination's own, less-loaded compute queue is faster in expectation despite paying a full recompute.
Assuming the composite routing score's utilization threshold is a fixed number. It moves with the network link (a slower path lowers it, as the 1.5x-RTT case above shows) and with the model's own KV footprint and prefill cost; re-derive it for your deployment rather than reusing this page's illustrative 0.73.

Open questions & validation¶

Benchmark the fetch-versus-recompute crossover, and the composite routing score's utilization threshold, on your actual inter-region link (measured bandwidth and RTT, not the illustrative figures above) and your actual model's KV footprint.
No production case study in this KB yet of this pattern's cross-region proxy running against real multi-region vLLM/Dynamo replicas; the individual bricks are documented, the composition is not benchmarked here.
Whether a hybrid (this pattern's regions, each internally also disaggregated per disaggregation rate matching) changes the crossover math above; not modeled here.
Whether per-layer heterogeneous KV-cache compression (a different eviction ratio and K/V bit-width chosen per layer under one global memory budget, as MoE-nD demonstrates for long-context serving) changes the bytes-per-token term in the crossover model above; not modeled here, and no source this page cites combines the two techniques.²
The composite routing score above uses a memoryless M/M/1 queueing model for simplicity, and assumes the fetch path and the destination's compute pool share one illustrative service rate; real admission control (inference QoS and admission control) uses richer batching, preemption, and priority dynamics than a single-server queue captures, and a real KV-transfer path and a real GPU scheduler rarely drain at the same rate. Validate the composite score's fetch-versus-recompute decision against a real deployment's measured queueing behavior on both resources before wiring it into a production migration policy.

References¶

GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving (arXiv 2602.11688): https://arxiv.org/abs/2602.11688
Kaplan et al., Scaling Laws for Neural Language Models (arXiv 2001.08361): https://arxiv.org/abs/2001.08361
MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression (arXiv 2604.17695): https://arxiv.org/abs/2604.17695
NVIDIA H100 specifications (dense BF16 TFLOPS used in the executed model above): https://www.nvidia.com/en-us/data-center/h100/

arXiv 2602.11688, abstract: "Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT... GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters." ↩↩↩↩
MoE-nD, arXiv 2604.17695, abstract: KV cache compression methods "each act on a single axis" (eviction, quantization, low-rank projection, cross-layer sharing) "but apply the same recipe to every layer"; proposes routing each layer to its own (eviction-ratio, K-bits, V-bits) tuple under a global memory budget via an offline-calibrated greedy solver, matching an uncompressed baseline at 14x compression on a LongBench-v1 subset. ↩