Markdown

Decentralized and distributed inference across non-colocated GPUs¶

Scope: serving one logical LLM inference workload, a vLLM/SGLang/TensorRT-LLM-class engine or a decentralized alternative, when the GPUs are not colocated: spread across regions, cloud providers, or a volunteer/decentralized pool with no shared InfiniBand/NVLink fabric between them. Covers the two viable patterns (region-replica routing versus a cross-WAN model-parallel split), how KV-cache handling differs from the intra-datacenter case in KV cache transfer with NIXL, and how to run each in production. This is the inference-serving counterpart to Recipe: DiLoCo (geo-distributed) for training; the WAN transport substrate is Overlay & mesh networking.

Everything under Inference serving, Inference parallelism strategies, and Disaggregated inference assumes GPUs sit in one fast-fabric domain; those pages say so explicitly (TP groups stay inside an NVLink island, prefill/decode pools stay "within fast-interconnect reach"). This page is what to do when that assumption fails. Region-replica routing is production-composable from bricks this KB already documents. Cross-WAN model-parallel splitting (Petals, HexGen, HexGen-2, Helix, Parallax) is research-stage: real, working, published with code, but not something vLLM, SGLang, TensorRT-LLM, or Dynamo ship as a supported deployment mode as of this writing. Treat the two patterns as different tools for different failure conditions, not a maturity ladder.

flowchart TB
  Q{"Does a full replica fit<br/>on GPUs at one site?"}
  Q -->|"yes, at every site you have"| A["Pattern A: region-replica routing<br/>(production-composable)"]
  Q -->|"no single site can hold a full replica"| B["Pattern B: cross-WAN model-parallel split<br/>(research-stage)"]
  A --> A1["Each region: an ordinary co-located<br/>disaggregated-inference deployment"]
  A1 --> A2["Global proxy: KV-locality + load +<br/>network-aware routing across regions"]
  B --> B1["Layers/experts sharded across<br/>non-colocated nodes, pipeline-parallel over WAN"]
  B1 --> B2["Each node keeps only its own layers' KV;<br/>activations cross the WAN hop, not the cache"]

What it is¶

Two structurally different ways to serve one inference workload when GPUs are not colocated:

Pattern A: region-replica routing. Each site (region, cloud, or provider) runs a complete, ordinary, co-located deployment following inference serving and disaggregated inference: TP/PP/EP inside the site's own fast fabric, prefill and decode pools on the same rack or NVLink island. A cross-region-aware proxy sits in front of all sites and decides, per request, which region serves it, factoring in KV-cache locality (does this region already hold a warm prefix cache for this session), current queue depth, and inter-region network latency. GORGO (arXiv 2602.11688) names this problem precisely: "LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency."⁶ Nothing crosses the WAN on the hot path except the request and response; the model, its weights, and its KV cache never leave the region that serves a given request.
Pattern B: cross-WAN model-parallel split. No single site holds a full replica. The model's layers (or, for MoE, its experts) are sharded across nodes that may sit on different continents with only ordinary internet links between them, and inference runs as a pipeline across those nodes for every request. Petals pioneered this for volunteer/community compute (BLOOM-176B inference on consumer GPUs at "$\approx$ 1 step per second"),¹ and HexGen, HexGen-2, Helix, and Parallax generalize it into a scheduling problem: given a heterogeneous set of GPUs and a heterogeneous, WAN-grade network between them, find the layer/tensor-parallel partition and per-request execution path that maximizes throughput or minimizes latency under those constraints.²³⁴⁵

The two patterns solve different problems. Pattern A assumes capacity exists at each site to hold a full replica and optimizes which replica serves a request. Pattern B assumes capacity is fundamentally scattered (no site has enough) and optimizes how to assemble a replica out of pieces that live far apart.

Why it matters¶

Economics. The same argument recipe-diloco-geo-distributed.md makes for training applies to serving: usable GPU capacity is not always colocated, and neocloud or volunteer pools are often cheaper than a single centralized cluster with the headroom to hold a full replica everywhere.
User-facing latency. Region-replica routing is also how you put inference near users worldwide; a request answered by a replica in the requester's region has a lower network hop than one served cross-continent, on top of whatever the model itself takes.
Capacity that cannot be consolidated. Some deployments (community inference of an open model, a decentralized marketplace, GPUs stranded below the memory footprint of the target model at any single site) genuinely cannot assemble a full replica anywhere. Pattern B exists for exactly this case: HexGen's stated goal is to "mitigate the substantial inference costs typically associated with a single centralized datacenter" by deploying "across diverse GPUs interconnected by a fully heterogeneous network."²
The KV cache is the crux, and it is a harder problem here than in training. DiLoCo's cross-worker payload is a pseudo-gradient exchanged every H steps, a fixed, small, infrequent cost. Inference's equivalent state, the KV cache, is large (hundreds of KB per token, see KV cache transfer with NIXL), grows continuously for the life of a request, and the whole point of disaggregation and prefix caching is to keep reusing it cheaply. Moving that state over a WAN link is a fundamentally different cost proposition than moving it over NVLink or IB, which is why this page treats it as the load-bearing section, not an afterthought.

When to use it (and when not)¶

Prefer Pattern A whenever a full replica fits at every site you actually have. It reuses every production-proven piece this KB already documents (engines, disaggregation, SLOs, QoS) unchanged; the only new component is the cross-region proxy.
Reach for Pattern B only when no site can host a full replica and consolidating capacity (renting a bigger instance, waiting, or serving a smaller/quantized model instead) is not an option. Go in expecting a real throughput and latency ceiling: Petals' own reported number, about one decoding step per second on BLOOM-176B, is "enough for many interactive LLM applications" but is nowhere near a single-datacenter deployment of the same model.¹
Never split prefill and decode across non-colocated sites. Disaggregated inference already states the rule for the intra-datacenter case ("keep prefill and decode pools within fast-interconnect reach, same rail/rack where possible"); across a WAN link the KV-cache handoff between an unpaired prefill and decode site is strictly worse than either Pattern A (recompute or cache locally) or Pattern B (never move the KV cache at all, see below). Disaggregation and cross-site serving solve different problems; do not combine them naively.
Do not put the inference hot path on a WAN overlay meant for control traffic. Overlay & mesh networking is built and validated for a training rendezvous and low-frequency DiLoCo sync, not for a synchronous per-token decode loop; reusing it for cross-region request/response traffic (Pattern A) is fine, reusing it for a step-synchronous collective (a naive attempt at Pattern B over gloo/TCP) will stall exactly as it does for training.

Architecture¶

Pattern A: region-replica routing.

flowchart TB
  C["Client (any region)"] --> GP["Global proxy / GSLB<br/>(KV-locality + load + RTT aware)"]
  GP -->|"warm prefix cached here, or lowest cost"| R1["Region 1: full replica<br/>(disaggregated-inference.md deployment)"]
  GP -->|"session/prefix elsewhere, or R1 saturated"| R2["Region 2: full replica"]
  R1 -.->|"control plane only: health, metrics,<br/>model sync (WireGuard overlay)"| R2

Pattern B: cross-WAN model-parallel split.

flowchart LR
  C["Client"] --> N1["Node A (region 1)<br/>layers 1-20, local KV for its layers only"]
  N1 -->|"activations for current token<br/>(WAN hop, gloo/TCP)"| N2["Node B (region 2)<br/>layers 21-40, local KV for its layers only"]
  N2 -->|"activations"| N3["Node C (region 3)<br/>layers 41-60, local KV for its layers only"]
  N3 --> C

In Pattern B the KV cache is never a cross-node transfer at all: each node caches only the layers it owns, for whichever sessions currently route through it, the same locality property inference parallelism strategies already documents for pipeline parallelism ("point-to-point activation handoff between stages"). What crosses the WAN hop is the current token's activation vector, which is orders of magnitude smaller than a KV cache. This is precisely what makes Pattern B tractable at all; it would not be if every hop had to carry the accumulated KV cache instead of one token's activations.

Does MoE make this easier? A real tension, not a clean win¶

Both patterns above mention Mixture-of-Experts almost in passing ("the model's layers, or for MoE, its experts, are sharded across nodes"). That framing understates a real tension: MoE's sparsity helps one part of decentralized serving and actively hurts another, hard enough that the net effect flips with concurrency rather than settling one way.

Where it helps: placement and sparse lookup. An expert is already a discrete, independent parameter block that only interacts with the rest of the model through the router's dispatch and combine step, unlike a dense layer's weight matrix, which has to be sliced (tensor parallelism) and reassembled with an all-reduce every layer. That makes an expert a far more natural unit to hand to a separate, non-colocated node than a TP shard. This is not a new idea: Ryabinin and Gusev's Learning@home proposed exactly this, a training and inference paradigm "designed to handle large amounts of poorly connected participants," predating Petals from the same research lineage.⁷ The structural reason it works at low concurrency: because each token only activates its top-k experts, a single request only has to locate and contact those k experts (for example over a DHT), not traverse every node in sequence the way Petals' dense pipeline-parallel serving must contact every layer-holding node for every request.¹ Growing total capacity by adding more experts does not add hops to any given request, unlike growing a dense model's depth.

Where it hurts: the communication pattern, at real concurrency. Expert parallelism for MoE inference documents the cost that actually dominates production serving: an all-to-all dispatch and combine at every MoE layer of every forward pass, not once per pipeline-stage boundary. And critically, "although one token hits only k experts, in aggregate across all concurrent tokens essentially all experts are active." At any real serving concurrency the sparsity benefit disappears at the batch level: every expert-holding node has to be reached, fast, on every layer, a far tighter synchronization requirement than Pattern B's dense case (one hop per pipeline stage) or DiLoCo's one sync every H steps. It is also exposed to the same straggler effect that page documents: a hot expert stalls the whole layer, since all expert computations must complete before the layer advances. Over genuinely non-colocated nodes, with no RDMA/NVLink and WAN-grade RTT, an every-layer all-to-all is close to the worst pattern to decentralize; the mitigations that make EP practical today (hierarchical two-stage all-to-all, expert collocation, low-latency kernels) all assume a fast fabric to hide the imbalance on, exactly what Pattern B does not have.

The net effect flips with concurrency, and nobody has shipped the WAN-scale answer yet. At low concurrency (one or a few concurrent sessions, the Learning@home/Petals regime), MoE's sparse lookup is a genuine decentralization advantage over a dense model of comparable capacity. At production concurrency, the all-to-all's every-layer synchronization requirement makes MoE harder to decentralize than a dense model, not easier: dense Pattern B degrades gracefully (one WAN hop per pipeline stage), where MoE's naive extension does not (one WAN round trip, potentially to many nodes, per MoE layer, for every concurrent batch). None of HexGen, HexGen-2, Helix, or Parallax, the Pattern B systems cited above, target this specifically: they generalize placement and scheduling across heterogeneous, non-colocated GPUs, but their evaluations are dense-model workloads (OPT, Llama-2), not MoE expert placement under a WAN-grade all-to-all.²³⁴⁵ Treat "decentralized MoE serving at production concurrency" as an open problem this page does not have a validated answer for, not a solved case of Pattern B.

How to use it: routing on KV-cache locality versus network cost¶

Pattern A's proxy has to decide, per request, whether to route to the region that already holds a warm cached prefix (paying network transfer or a repeat round trip) or to route locally and pay a fresh prefill. This is exactly the "jointly account for KV-cache locality, replica load, and variable network latency" problem GORGO names.⁶ The model below makes the tradeoff quantitative: given a WAN link's bandwidth and RTT, and a model's KV-cache footprint and prefill compute cost, is it cheaper to fetch the cached KV cross-region or to recompute the prefix locally?

# wan_kv_crossover.py -- validated: does fetching a cached KV prefix across a WAN
# region beat recomputing prefill locally, and at what prefix length? Run: python3 wan_kv_crossover.py
from __future__ import annotations


def bytes_per_kv_token(n_layers: int, n_kv_heads: int, head_dim: int, bytes_per_element: int = 2) -> float:
    # bytes_per_token = 2 x n_layers x n_kv_heads x head_dim x bytes_per_element (K and V); see kv-cache-transfer-nixl.md
    return 2 * n_layers * n_kv_heads * head_dim * bytes_per_element


def recompute_seconds(prefix_tokens: int, params: float, tflops: float, mfu: float) -> float:
    # forward-only FLOPs/token ~= 2 x params (Kaplan et al. 2020, C=6ND; forward pass is ~2ND); prefill is compute-bound.
    flops_per_token = 2.0 * params
    return prefix_tokens * flops_per_token / (tflops * mfu)


def fetch_seconds(prefix_tokens: int, bytes_per_token: float, bandwidth_gbps: float, rtt_ms: float) -> float:
    payload_bits = prefix_tokens * bytes_per_token * 8
    return rtt_ms / 1000.0 + payload_bits / (bandwidth_gbps * 1e9)


def crossover_tokens(bytes_per_token: float, bandwidth_gbps: float, rtt_ms: float,
                      params: float, tflops: float, mfu: float) -> float | None:
    """Prefix length above which fetching the cached KV cross-region beats local recompute.
    None means recompute's per-token cost never exceeds transfer's, so fetch never wins."""
    recompute_per_token = 2.0 * params / (tflops * mfu)
    transfer_per_token = bytes_per_token * 8 / (bandwidth_gbps * 1e9)
    if recompute_per_token <= transfer_per_token:
        return None
    return (rtt_ms / 1000.0) / (recompute_per_token - transfer_per_token)


if __name__ == "__main__":
    # Llama-3.1-70B-class dense model: 80 layers, 8 KV heads (GQA), head_dim 128, BF16 KV cache.
    bpt = bytes_per_kv_token(n_layers=80, n_kv_heads=8, head_dim=128)
    assert bpt == 327680.0, bpt  # 320 KiB/token

    params = 70e9
    tflops = 989e12   # H100 SXM dense BF16 (half NVIDIA's published 1,979 TFLOPS sparse figure), see kv-cache-inference-speedup.md
    mfu = 0.40         # illustrative prefill MFU, not hardware-measured here
    rtt_ms = 70.0      # illustrative cross-region RTT

    recompute_per_token_us = 2 * params / (tflops * mfu) * 1e6

    # Regime 1: public-internet-grade cross-region link (5 Gbps effective goodput).
    weak_gbps = 5.0
    weak_transfer_per_token_us = bpt * 8 / (weak_gbps * 1e9) * 1e6
    L_weak = crossover_tokens(bpt, weak_gbps, rtt_ms, params, tflops, mfu)
    assert L_weak is None, L_weak                 # transfer per-token cost exceeds recompute: fetch never wins
    assert weak_transfer_per_token_us > recompute_per_token_us

    # Regime 2: well-provisioned private inter-region backbone (100 Gbps).
    fast_gbps = 100.0
    fast_transfer_per_token_us = bpt * 8 / (fast_gbps * 1e9) * 1e6
    L_fast = crossover_tokens(bpt, fast_gbps, rtt_ms, params, tflops, mfu)
    assert L_fast is not None and L_fast > 0
    assert fast_transfer_per_token_us < recompute_per_token_us

    # Below the crossover, local recompute must be cheaper than the fetch; above it, the reverse.
    short_prefix = int(L_fast * 0.5)
    long_prefix = int(L_fast * 2.0)
    assert recompute_seconds(short_prefix, params, tflops, mfu) < fetch_seconds(short_prefix, bpt, fast_gbps, rtt_ms)
    assert fetch_seconds(long_prefix, bpt, fast_gbps, rtt_ms) < recompute_seconds(long_prefix, params, tflops, mfu)

    # Doubling RTT must push the crossover higher (a fetch has more fixed cost to amortize).
    L_fast_slower_rtt = crossover_tokens(bpt, fast_gbps, rtt_ms * 2, params, tflops, mfu)
    assert L_fast_slower_rtt > L_fast

    print(f"bytes/token                       = {bpt/1024:.1f} KiB")
    print(f"recompute rate                     = {recompute_per_token_us:.2f} us/token")
    print(f"transfer rate @5 Gbps (public WAN)  = {weak_transfer_per_token_us:.2f} us/token -> fetch never wins (crossover: none)")
    print(f"transfer rate @100 Gbps (backbone)  = {fast_transfer_per_token_us:.2f} us/token -> crossover at {L_fast:,.0f} tokens")
    print(f"crossover @100Gbps, 2x RTT ({rtt_ms*2:.0f}ms) = {L_fast_slower_rtt:,.0f} tokens")
    print(f"recompute@1k tokens (any link)      = {recompute_seconds(1000, params, tflops, mfu)*1000:.2f} ms")
    print(f"fetch@1k tokens @100Gbps             = {fetch_seconds(1000, bpt, fast_gbps, rtt_ms)*1000:.2f} ms")
    print(f"recompute@8k tokens (any link)       = {recompute_seconds(8000, params, tflops, mfu)*1000:.2f} ms")
    print(f"fetch@8k tokens @100Gbps             = {fetch_seconds(8000, bpt, fast_gbps, rtt_ms)*1000:.2f} ms")
    print("ALL ASSERTIONS PASSED")

Executed output:

bytes/token                       = 320.0 KiB
recompute rate                     = 353.89 us/token
transfer rate @5 Gbps (public WAN)  = 524.29 us/token -> fetch never wins (crossover: none)
transfer rate @100 Gbps (backbone)  = 26.21 us/token -> crossover at 214 tokens
crossover @100Gbps, 2x RTT (140ms) = 427 tokens
recompute@1k tokens (any link)      = 353.89 ms
fetch@1k tokens @100Gbps             = 96.21 ms
recompute@8k tokens (any link)       = 2831.14 ms
fetch@8k tokens @100Gbps             = 279.72 ms
ALL ASSERTIONS PASSED

The result is not a fixed rule, it is a decision that depends entirely on the actual link: over an ordinary public-internet-grade cross-region path (5 Gbps illustrative), a 70B-class dense model's KV cache is expensive enough per byte that local recompute wins outright, at every prefix length, no crossover exists. Over a well-provisioned private inter-region backbone (100 Gbps illustrative), fetching becomes cheaper once the cached prefix passes a few hundred tokens, and dominates heavily for long prefixes (8k tokens: 280ms fetched versus 2.8s recomputed). This is why a real cross-region router cannot use a static policy; it has to measure the path it actually has (bandwidth, RTT) and the prefix length in front of it, exactly the "tunable parameters" GORGO's cost model exists to set.⁶ Swap in your model's real layer count, KV-head count, head dimension, and measured MFU and RTT before trusting a number from this script on your own deployment; the tflops figure is a cited H100 spec, everything else here is illustrative.

How to develop with it: building the cross-region KV-aware router¶

Pattern A's proxy is llm-request-routing.md's router pattern (a gateway/proxy classifies and dispatches, keeping its own latency far below the generation it gates) generalized from "which model" to "which region." Three signals it needs that a same-datacenter router does not:

KV/prefix locality per replica. Track, per active session or per prefix hash, which region's replica already holds the warm cache (the same idea as SGLang's RadixAttention prefix reuse in inference serving, extended from one engine's prefix tree to a directory across regions). A consistent-hash on a stable session key is the simplest version; a shared prefix-radix index across regions is the fuller one.
Live inter-region network state. RTT and available bandwidth between the proxy and each region drift (congestion, provider changes, time of day); measure them, do not hard-code them. The crossover model above is only as good as these two live inputs.
Per-region queue depth. A region with a warm cache but a saturated decode pool is not necessarily the best target; combine KV locality with the same admission-control signal inference QoS and admission control already tracks locally, now compared across regions.

For Pattern B (cross-WAN model-parallel), the router problem is replaced by a placement and pipeline-selection problem: which nodes host which layers or experts (Parallax's "model allocation" phase⁵), and which chain of nodes a given request's pipeline walks at request time (Parallax's "request-time GPU pipeline selection," Helix's max-flow/MILP formulation⁴). Two things carry over from the training case in Overlay & mesh networking: the inter-node transport for activations is gloo/TCP, not NCCL-over-IB, since there is no fast fabric between non-colocated nodes; and a churny node pool (volunteers joining and leaving) needs the same kind of fault-tolerant membership DiLoCo's decentralized path uses, not a fixed WORLD_SIZE.

How to maintain it¶

Watch the same overlay health signals as the training case: per-peer handshake age, keepalive, and measured RTT/goodput (Overlay & mesh networking), since Pattern A's control plane and Pattern B's activation hops both ride the same kind of WAN link.
Track KV-cache hit rate per region, not just globally. A falling regional hit rate is the leading indicator that the router's locality signal has drifted (traffic mix shifted, a region was added or drained) before TTFT/TPOT SLOs (see SLOs: inference serving) show it.
Re-measure the crossover, do not bake it in. Cloud inter-region bandwidth and RTT are not constants; re-run the fetch-versus-recompute comparison above whenever a region pairing, provider, or traffic pattern changes.
For Pattern B, watch per-hop latency accumulation. Each additional WAN hop in the pipeline adds its RTT to time-to-first-token; a pipeline across more nodes is not free the way it is over NVLink.

Running it in production¶

	Pattern A: region-replica routing	Pattern B: cross-WAN model-parallel split
Maturity	Production-composable from pieces this KB already documents (engines, disaggregation, routing, overlay networking); the composition itself is not a single vendor-shipped product as of this writing	Research-stage: Petals, HexGen, HexGen-2, Helix, and Parallax are published, working, open-source systems, not a supported mode of vLLM/SGLang/TensorRT-LLM/Dynamo
What moves over WAN	Requests and responses only; KV cache never leaves the serving region	Per-token activations between pipeline stages; KV cache never leaves the node that owns those layers
Latency ceiling	Roughly the same as any single region's deployment, plus one WAN hop for the request/response	Accumulates one WAN RTT per pipeline stage; Petals reports about 1 decoding step/s on BLOOM-176B on consumer GPUs¹
When it applies	A full replica fits at every site you have	No site can host a full replica; capacity is fundamentally scattered
Where the KV-cache decision lives	The cross-region proxy (GORGO-style holistic cost model)⁶	Not a decision: each node's KV is local to its own layers by construction

Failure modes¶

Splitting prefill and decode across non-colocated sites. Reintroduces a WAN KV-cache handoff exactly where disaggregated inference says to keep it on a fast fabric; slower than either colocating prefill/decode per region (Pattern A) or not moving the KV cache at all (Pattern B).
A static cross-region routing policy. Bandwidth and RTT between regions drift; a router tuned once will misroute as the path changes. Re-measure, do not hard-code (see Maintain, above).
Treating Pattern B as a drop-in vLLM deployment mode. It is a different system (Petals/HexGen/HexGen-2/Helix/Parallax), not a flag on a production engine; going in expecting datacenter-class latency will disappoint.
Putting the decode hot path on the WAN overlay. The overlay in Overlay & mesh networking is validated for control traffic and low-frequency sync, not a step-synchronous loop; this is the same mistake the DiLoCo recipe warns against for training, applied to serving.
Fetching cached KV cross-region on a weak link. As the executed model above shows, on an ordinary public-internet-grade path this is often strictly worse than local recompute; verify the crossover on your actual link before wiring cross-region KV fetch into the hot path.
Ignoring per-region queue depth in favor of KV locality alone. A warm cache in an overloaded region is not the best target; combine locality with load, as GORGO's cost model does.⁶
Assuming an MoE model shards across non-colocated nodes as easily as a dense one. Placement is easier, an expert is already a discrete unit, but the every-layer all-to-all is a tighter synchronization requirement than dense Pattern B's one hop per pipeline stage; see Does MoE make this easier? before assuming a naive HexGen/Helix-style placement decentralizes MoE serving well at production concurrency.

Open questions & validation¶

Benchmark the fetch-versus-recompute crossover on your actual inter-region link (measured bandwidth and RTT, not the illustrative figures above) and your actual model's KV footprint.
No production case study in this KB yet of Pattern A's cross-region proxy running against real multi-region vLLM/Dynamo replicas; the individual bricks are documented, the composition is not benchmarked here.
Pattern B's throughput and latency ceiling versus node count and network heterogeneity, reproduced on your own volunteer/decentralized pool rather than taken from the papers' reported numbers.
Whether a hybrid (Pattern A regions, each internally also disaggregated per disaggregation rate matching) changes the crossover math above; not modeled here.

References¶

Borzunov et al., Petals: Collaborative Inference and Fine-tuning of Large Models (arXiv 2209.01188): https://arxiv.org/abs/2209.01188
Jiang et al., HexGen: Generative Inference of Large Language Model over Heterogeneous Environment (arXiv 2311.11514): https://arxiv.org/abs/2311.11514
HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment (arXiv 2502.07903): https://arxiv.org/abs/2502.07903
Mei et al., Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow (arXiv 2406.01566, ASPLOS 2025): https://arxiv.org/abs/2406.01566
Parallax: Efficient LLM Inference Service over Decentralized Environment (arXiv 2509.26182): https://arxiv.org/abs/2509.26182
GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving (arXiv 2602.11688): https://arxiv.org/abs/2602.11688
BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving (arXiv 2510.13223): https://arxiv.org/abs/2510.13223
No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha (arXiv 2409.17264): https://arxiv.org/abs/2409.17264
Kaplan et al., Scaling Laws for Neural Language Models (arXiv 2001.08361): https://arxiv.org/abs/2001.08361
Ryabinin and Gusev, Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts (arXiv 2002.04013): https://arxiv.org/abs/2002.04013
NVIDIA H100 specifications (dense BF16 TFLOPS used in the executed model above): https://www.nvidia.com/en-us/data-center/h100/

Borzunov et al., arXiv 2209.01188, abstract: "we propose Petals... a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties... running inference of BLOOM-176B on consumer GPUs with $\approx$ 1 step per second, which is enough for many interactive LLM applications." ↩↩↩↩
Jiang et al., arXiv 2311.11514, abstract: "deploying such services in a heterogeneous and cross-datacenter setting to mitigate the substantial inference costs typically associated with a single centralized datacenter... HexGen can choose to achieve up to 2.3 times lower latency deadlines or tolerate up to 4 times more request rates compared with the homogeneous baseline given the same budget," evaluated serving Llama-2 (70B). ↩↩↩
arXiv 2502.07903, abstract: "a scheduling algorithm that formalizes the allocation of disaggregated LLM inference computations and communications over heterogeneous GPUs and network connections as a constraint optimization problem... graph partitioning and max-flow algorithms to co-optimize resource allocation, parallel strategies... and inter-phase key-value (KV) cache communications... up to a 2.0 times and on average a 1.3 times improvement in serving throughput, reduces the average inference latency by 1.5 times... and achieves comparable inference performance with a 30% lower price budget," evaluated on OPT (30B) and Llama-2 (70B). ↩↩
Mei et al., arXiv 2406.01566, abstract: "formulate inference computation of LLMs over heterogeneous GPUs and network connections as a max-flow problem on directed, weighted graphs... a mixed integer linear programming (MILP) algorithm... evaluation on several heterogeneous clusters ranging from 24 to 42 GPU nodes shows that Helix improves serving throughput by up to 3.3x and reduces prompting and decoding latency by up to 66% and 24%, respectively." ↩↩↩
arXiv 2509.26182, abstract: "Parallax decomposes planning into (i) model allocation, which places layers of each replica across diverse GPUs to jointly optimize latency and throughput under memory and link-bandwidth constraints, and (ii) request-time GPU pipeline selection, which stitches layers from different replicas into end-to-end execution chains... evaluated on open-source LLMs deployed over real volunteer nodes." ↩↩↩
arXiv 2602.11688, abstract: "Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT... GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters." ↩↩↩↩↩
Ryabinin and Gusev, arXiv 2002.04013, abstract: "we propose Learning@home: a novel neural network training paradigm designed to handle large amounts of poorly connected participants," the decentralized Mixture-of-Experts lineage that precedes Petals. ↩