Markdown

Decentralized and distributed inference across non-colocated GPUs¶

Scope: this page has split into two deeper pages, one per pattern, now that each has enough of its own material to stand alone. Use the decision below to pick the one you need; this stub stays at the old URL so existing links keep resolving, and carries no other content of its own.

flowchart TB
  Q{"Does a full replica fit<br/>on GPUs at one site?"}
  Q -->|"yes, at every site you have"| A["Pattern A: region-replica routing<br/>(production-composable)"]
  Q -->|"no single site can hold a full replica"| B["Pattern B: cross-WAN model-parallel split<br/>(research-stage)"]
  A --> A1["Each region: an ordinary co-located<br/>disaggregated-inference deployment"]
  A1 --> A2["Global proxy: KV-locality + load +<br/>network-aware routing across regions"]
  B --> B1["Layers/experts sharded across<br/>non-colocated nodes, pipeline-parallel over WAN"]
  B1 --> B2["Each node keeps only its own layers' KV;<br/>activations cross the WAN hop, not the cache"]

If a full replica fits at every site you have, read Region-replica routing for non-colocated inference: each site runs an ordinary co-located deployment, and a cross-region proxy routes requests by KV-cache locality, load, and network cost.
If no single site can hold a full replica, read Cross-WAN model-parallel inference (decentralized serving): the model's layers or experts are sharded across non-colocated nodes and the forward pass runs as a pipeline over a WAN link, covering Petals, HexGen, HexGen-2, Helix, and Parallax.

Two supporting pages sit underneath Pattern B, because it is not one decision but three:

Mesh LLM and Skippy stage splits is the worked per-technology example: an Apache-2.0 implementation that ships the whole stack (QUIC transport, gossip, discovery, layer-package weight distribution, a latency-aware stage planner), plus the honest measurement that splitting a model which already fits on one machine makes decode much slower, because every added stage adds a serialized hop to the decode critical path while the compute it redistributes stays the same.
P2P transport for decentralized inference covers the substrate every Pattern B system needs and most of them delegate to a dependency or leave underspecified: QUIC, NAT traversal, relay fallback, gossip membership, and peer discovery. Overlay and mesh networking covers the WireGuard-overlay alternative and says explicitly not to put a decode hot path on it; that page is the control-plane substrate, this one is the data-plane substrate.

References¶

See the References section on each of the pages above for full citations; this stub introduces no new claims of its own.