Markdown

Cross-WAN model-parallel inference (decentralized serving)¶

Scope: serving one logical LLM inference workload, a vLLM/SGLang/TensorRT-LLM-class engine or a decentralized alternative, by sharding its layers or experts across GPUs that sit in different regions, clouds, or a volunteer/decentralized pool with no shared InfiniBand/NVLink fabric between them, and running the forward pass as a pipeline across a WAN link for every request. Covers Petals, HexGen, HexGen-2, Helix, and Parallax; the WAN-speculative-decoding, coordinator-placement, verification, and fault-tolerance mechanics of a production-grade implementation; and how concurrent request traffic amortizes WAN round-trip latency. This is the research-stage half of non-colocated inference serving; when a full replica fits at every site you have, use region-replica routing instead. This is the inference-serving counterpart to Recipe: DiLoCo (geo-distributed) for training; the WAN transport substrate is Overlay & mesh networking.

Everything under Inference serving, Inference parallelism strategies, and Disaggregated inference assumes GPUs sit in one fast-fabric domain; those pages say so explicitly (TP groups stay inside an NVLink island, prefill/decode pools stay "within fast-interconnect reach"). This page is what to do when that assumption fails and no single site can hold a full replica: implementations in the Petals, HexGen, HexGen-2, Helix, and Parallax lineage are real, working, published with code, but research-stage, not something vLLM, SGLang, TensorRT-LLM, or Dynamo ship as a supported deployment mode as of this writing. If a full replica fits at every site you have, the production-composable answer is region-replica routing instead; the two are different tools for different failure conditions, not a maturity ladder. This pattern's throughput ceiling is not fixed by the architecture; it depends heavily on whether speculative decoding is built into the WAN hop itself, covered in its own section below.

flowchart TB
  Q{"Does a full replica fit<br/>on GPUs at one site?"}
  Q -->|"yes, at every site you have"| A["Pattern A: region-replica routing<br/>(production-composable)"]
  Q -->|"no single site can hold a full replica"| B["Pattern B: cross-WAN model-parallel split<br/>(research-stage)"]
  A --> A1["Each region: an ordinary co-located<br/>disaggregated-inference deployment"]
  A1 --> A2["Global proxy: KV-locality + load +<br/>network-aware routing across regions"]
  B --> B1["Layers/experts sharded across<br/>non-colocated nodes, pipeline-parallel over WAN"]
  B1 --> B2["Each node keeps only its own layers' KV;<br/>activations cross the WAN hop, not the cache"]

This page covers Pattern B (the bottom branch): read on if no single site you have can hold a full replica. If a full replica fits at every site, the production-composable answer is Pattern A, covered in region-replica routing for non-colocated inference.

What it is¶

No single site holds a full replica. The model's layers (or, for MoE, its experts) are sharded across nodes that may sit on different continents with only ordinary internet links between them, and inference runs as a pipeline across those nodes for every request. Petals pioneered this for volunteer/community compute (BLOOM-176B inference on consumer GPUs at "$\approx$ 1 step per second"),¹ and HexGen, HexGen-2, Helix, and Parallax generalize it into a scheduling problem: given a heterogeneous set of GPUs and a heterogeneous, WAN-grade network between them, find the layer/tensor-parallel partition and per-request execution path that maximizes throughput or minimizes latency under those constraints.²³⁴⁵ "Heterogeneous" is not GPU-only in every implementation: Parallax's node backend runs on SGLang or vLLM for GPUs and separately on MLX LM for Apple Silicon, so a pipeline can span NVIDIA/AMD GPUs and unified-memory Macs in the same swarm, the same decode-friendly unified-memory property LLM inference efficiency already notes for single-node serving, now as one node class among several in a decentralized pipeline.⁵ Later implementations in this lineage add speculative decoding and per-stage cryptographic verification to the same pipeline-parallel architecture, reaching well past that early throughput ceiling; the mechanism and its limits are covered in its own section below.⁷

This is one of two structurally different ways to serve one inference workload when GPUs are not colocated. The other, region-replica routing, assumes capacity exists at each site to hold a full replica and optimizes which replica serves a request. This pattern instead assumes capacity is fundamentally scattered (no site has enough) and optimizes how to assemble a replica out of pieces that live far apart.

Why it matters¶

Capacity that cannot be consolidated. Some deployments (community inference of an open model, a decentralized marketplace, GPUs stranded below the memory footprint of the target model at any single site) genuinely cannot assemble a full replica anywhere. This pattern exists for exactly this case: HexGen's stated goal is to "mitigate the substantial inference costs typically associated with a single centralized datacenter" by deploying "across diverse GPUs interconnected by a fully heterogeneous network."²
Economics. The same argument recipe-diloco-geo-distributed.md makes for training applies here: usable GPU capacity is not always colocated, and neocloud or volunteer pools are often cheaper than a single centralized cluster with the headroom to hold a full replica everywhere.
The KV cache is the crux, and it is a harder problem here than in training. DiLoCo's cross-worker payload is a pseudo-gradient exchanged every H steps, a fixed, small, infrequent cost. Inference's equivalent state, the KV cache, is large (hundreds of KB per token, see KV cache transfer with NIXL), grows continuously for the life of a request, and the whole point of disaggregation and prefix caching is to keep reusing it cheaply. This pattern's architecture sidesteps that cost outright by never moving the KV cache over the WAN hop at all, covered next; region-replica routing instead has to make the KV-cache-versus-network-cost tradeoff the centerpiece of its own routing decision.

When to use it (and when not)¶

Reach for this pattern only when no site can host a full replica and consolidating capacity (renting a bigger instance, waiting, or serving a smaller/quantized model instead) is not an option. Go in expecting a real throughput and latency ceiling, and note that ceiling depends heavily on whether speculative decoding is built into the WAN hop: without it, throughput is capped near one committed token per network round trip, the regime Petals' own reported number describes (about one decoding step per second on BLOOM-176B, "enough for many interactive LLM applications" but nowhere near a single-datacenter deployment of the same model);¹ with a WAN-aware speculative-decoding router in the loop, an order of magnitude or more higher single-stream throughput has been demonstrated on larger models, still well below native single-datacenter serving but a materially different ceiling.⁷
Prefer region-replica routing whenever a full replica fits at every site you actually have. It reuses every production-proven piece this KB already documents (engines, disaggregation, SLOs, QoS) unchanged; this pattern is a different, heavier system, not a drop-in mode of a production engine.
Never split prefill and decode across non-colocated sites. Disaggregated inference already states the rule for the intra-datacenter case ("keep prefill and decode pools within fast-interconnect reach, same rail/rack where possible"); across a WAN link the KV-cache handoff between an unpaired prefill and decode site is strictly worse than either region-replica routing (recompute or cache locally) or this pattern (never move the KV cache at all, see below). Disaggregation and cross-site serving solve different problems; do not combine them naively.
Do not put the inference hot path on a WAN overlay meant for control traffic. Overlay & mesh networking is built and validated for a training rendezvous and low-frequency DiLoCo sync, not for a synchronous per-token decode loop; a naive attempt at this pattern over gloo/TCP on that overlay will stall exactly as it does for training. Production implementations in this lineage use a purpose-built transport instead (Parallax's Lattica P2P transport, for example) or the point-to-point gloo/TCP link described in Architecture, below, never the training overlay.

Architecture¶

flowchart LR
  C["Client"] --> N1["Node A (region 1)<br/>layers 1-20, local KV for its layers only"]
  N1 -->|"activations for current token<br/>(WAN hop, gloo/TCP)"| N2["Node B (region 2)<br/>layers 21-40, local KV for its layers only"]
  N2 -->|"activations"| N3["Node C (region 3)<br/>layers 41-60, local KV for its layers only"]
  N3 --> C

The KV cache is never a cross-node transfer at all: each node caches only the layers it owns, for whichever sessions currently route through it, the same locality property inference parallelism strategies already documents for pipeline parallelism ("point-to-point activation handoff between stages"). What crosses the WAN hop is the current token's activation vector, which is orders of magnitude smaller than a KV cache. This is precisely what makes this pattern tractable at all; it would not be if every hop had to carry the accumulated KV cache instead of one token's activations.

Does MoE make this easier? A real tension, not a clean win¶

The architecture above mentions Mixture-of-Experts almost in passing ("the model's layers, or for MoE, its experts, are sharded across nodes"). That framing understates a real tension: MoE's sparsity helps one part of decentralized serving and actively hurts another, hard enough that the net effect flips with concurrency rather than settling one way.

Where it helps: placement and sparse lookup. An expert is already a discrete, independent parameter block that only interacts with the rest of the model through the router's dispatch and combine step, unlike a dense layer's weight matrix, which has to be sliced (tensor parallelism) and reassembled with an all-reduce every layer. That makes an expert a far more natural unit to hand to a separate, non-colocated node than a TP shard. This is not a new idea: Ryabinin and Gusev's Learning@home proposed exactly this, a training and inference paradigm "designed to handle large amounts of poorly connected participants," predating Petals from the same research lineage.⁶ The structural reason it works at low concurrency: because each token only activates its top-k experts, a single request only has to locate and contact those k experts (for example over a DHT), not traverse every node in sequence the way Petals' dense pipeline-parallel serving must contact every layer-holding node for every request.¹ Growing total capacity by adding more experts does not add hops to any given request, unlike growing a dense model's depth.

Where it hurts: the communication pattern, at real concurrency. Expert parallelism for MoE inference documents the cost that actually dominates production serving: an all-to-all dispatch and combine at every MoE layer of every forward pass, not once per pipeline-stage boundary. And critically, "although one token hits only k experts, in aggregate across all concurrent tokens essentially all experts are active." At any real serving concurrency the sparsity benefit disappears at the batch level: every expert-holding node has to be reached, fast, on every layer, a far tighter synchronization requirement than this pattern's dense case (one hop per pipeline stage) or DiLoCo's one sync every H steps. It is also exposed to the same straggler effect that page documents: a hot expert stalls the whole layer, since all expert computations must complete before the layer advances. Over genuinely non-colocated nodes, with no RDMA/NVLink and WAN-grade RTT, an every-layer all-to-all is close to the worst pattern to decentralize; the mitigations that make EP practical today (hierarchical two-stage all-to-all, expert collocation, low-latency kernels) all assume a fast fabric to hide the imbalance on, exactly what this pattern does not have.

The net effect flips with concurrency, and nobody has shipped the WAN-scale answer yet. At low concurrency (one or a few concurrent sessions, the Learning@home/Petals regime), MoE's sparse lookup is a genuine decentralization advantage over a dense model of comparable capacity. At production concurrency, the all-to-all's every-layer synchronization requirement makes MoE harder to decentralize than a dense model, not easier: the dense case degrades gracefully (one WAN hop per pipeline stage), where MoE's naive extension does not (one WAN round trip, potentially to many nodes, per MoE layer, for every concurrent batch). None of HexGen, HexGen-2, Helix, or Parallax, the systems cited above, target this specifically: they generalize placement and scheduling across heterogeneous, non-colocated GPUs, but their evaluations are dense-model workloads (OPT, Llama-2), not MoE expert placement under a WAN-grade all-to-all.²³⁴⁵ Treat "decentralized MoE serving at production concurrency" as an open problem this page does not have a validated answer for, not a solved case of this pattern.

Fast-fabric research is attacking the all-to-all directly, which puts the WAN gap in sharper relief. Two intra-datacenter techniques show how much engineering it takes to tame EP's all-to-all even with NVLink/RDMA available. Semantic Parallelism co-schedules expert placement and request/token routing together instead of treating them as separate concerns: it clusters and collocates experts by their measured co-activation tendency, then proactively rebatches incoming requests onto whichever device already hosts the experts they are likely to activate, cutting all-to-all communication volume in a production SGLang build.⁸ MoEShard instead perfectly load-balances by row- and column-decomposing expert weight matrices across GPUs, avoiding capacity-factor heuristics and token dropping entirely, and reports up to 6.4x faster time-to-first-token on encoder-based MoE inference.⁹ Neither targets a WAN link; both illustrate that eliminating or rebalancing the all-to-all is hard enough to be its own research area even with a fast fabric available. At least one WAN implementation sidesteps the problem rather than solving it: sharding whole layers, gate and every local expert together, so MoE routing and its all-to-all dispatch stay entirely inside one node, and only the pipeline's inter-layer activation crosses the WAN.⁷ That works, but it is a different strategy than expert-parallel-over-WAN, not a demonstration that the harder problem (splitting one MoE layer's experts across non-colocated nodes) is solved.

Speculative decoding over WAN, coordinator placement, and the integrity/confidentiality line¶

This pattern's throughput ceiling is not fixed by the architecture alone. Pipeline parallelism is inter-stage point-to-point send/recv of one activation tensor per micro-batch, the same mechanism whether the stages sit on one fast fabric or scattered across a WAN; what changes is only the per-hop latency. Without speculative decoding, every committed token needs a full traversal of every stage, so throughput is capped near one token per round trip; this is the regime Petals' reported figures describe.¹ Speculative decoding changes what a round trip buys: a drafter proposes several tokens, the target verifies all of them in one pass, and the accept rule min(1, p(x)/q(x)) with residual resampling on rejection, already proven lossless and executed in this KB, guarantees the committed sequence is distributed exactly as the target model regardless of the drafter. Over a fast fabric this is a latency optimization; over a WAN round trip, where every verification pass costs the full trip time, it is what makes this pattern's throughput regime possible at all.

An executed model of when pipelining a WAN speculative pipeline actually pays off¶

This KB's own speculative-decoding.md already derives and validates the expected tokens committed per verification round, E[tokens] = (1 - α^(K+1)) / (1 - α) for per-token draft acceptance α and K draft tokens. That formula answers "how many tokens per round." It does not answer the WAN-specific question: does keeping several rounds' worth of speculative chunks in flight at once (pipelining, to hide the round trip) actually raise throughput, or can it waste work? The model below answers that from first principles, reusing the KB's own accept-count distribution and extending it with the network's round-trip time and a pipelining depth.

The mechanism: a chunk of K draft tokens is dispatched every RTT/D seconds, so D chunks are in flight within one round trip. A dispatch is valid only if it was built on the ring's true, already-confirmed position. When a chunk's real result is a partial accept (fewer than K tokens land, which happens with probability 1 - α^K), every chunk already dispatched on the assumption that this one would fully land was built on a now-wrong continuation and is discarded; a corrected dispatch, built on the true position, must resolve before the pipeline can trust further chunks again. This is a literal, executable version of "a rejection flushes the pipe," not a restatement of the claim.

# wan_pipeline_specdec.py -- validated: a discrete-event simulation of pipelined speculative
# decoding over a WAN round trip, extending this KB's own speculative-decoding.md accept-count
# model (leading Bernoulli(alpha) run truncated at K, +1 bonus/resample) with the network's
# round-trip time and a pipelining depth D. Answers, from first principles and executed here,
# whether pipelining multiple speculative chunks in flight actually raises WAN throughput, and
# when it stops helping. Run: python3 wan_pipeline_specdec.py
from __future__ import annotations

import numpy as np


def expected_tokens_per_step(alpha: float, K: int) -> float:
    # Same closed form already validated in speculative-decoding.md: E[tokens per verify round]
    # for per-token acceptance alpha and K draft tokens (includes the +1 bonus/resample token).
    if alpha >= 1.0:
        return float(K + 1)
    return (1.0 - alpha ** (K + 1)) / (1.0 - alpha)


def draw_leading_accept_count(alpha: float, K: int, rng: np.random.Generator) -> int:
    """One chunk's true outcome: number of leading accepted draft tokens (0..K), the same
    per-token Bernoulli(alpha) model speculative-decoding.md's own Monte Carlo uses."""
    accepts = rng.random(K) < alpha
    m = 0
    for a in accepts:
        if not a:
            break
        m += 1
    return m


def simulate_wan_pipeline(alpha: float, K: int, D: int, rtt_s: float,
                          n_dispatches: int, rng: np.random.Generator) -> tuple[float, float]:
    """Discrete-event simulation of a WAN ring's decode loop with pipelining depth D.

    A chunk of K draft tokens is dispatched every rtt_s/D seconds (D chunks fit in flight
    within one round trip), each resolving rtt_s after its own dispatch. A dispatch is valid
    only if built on the ring's true, already-confirmed position; when a chunk's true result
    is a PARTIAL accept (m < K), every chunk already dispatched on the assumption that this one
    would fully land is discarded until a corrected dispatch, built on the now-known true
    position, resolves in turn.

    Returns (total_committed_tokens, total_wall_clock_seconds).
    """
    assert D >= 1 and K >= 1 and n_dispatches >= D
    dispatch_dt = rtt_s / D
    confirmed = 0
    wasted_until = -1                       # dispatch indices <= this are discarded, not drawn
    for i in range(n_dispatches):
        if i <= wasted_until:
            continue                        # built on a since-invalidated assumption: contributes 0
        m = draw_leading_accept_count(alpha, K, rng)
        confirmed += m + 1                  # the resampled correction or bonus token always commits
        if m < K:                           # partial accept: the assumed full-K continuation was wrong
            wasted_until = i + D - 1        # everything already in flight on that wrong assumption is void
    total_time = rtt_s + (n_dispatches - 1) * dispatch_dt   # wall-clock time of the last arrival
    return float(confirmed), float(total_time)


def crossover_alpha_for(K: int, D: int, rtt_s: float, n_dispatches: int,
                        rng: np.random.Generator, gain_threshold: float = 1.10) -> float:
    """Sweep alpha and return the lowest value where D-way pipelining beats synchronous (D=1)
    decode by more than gain_threshold. rtt_s should NOT affect this value (see the checks below):
    it cancels out of the gain ratio, since both throughputs scale as 1/rtt_s for fixed K, D."""
    alphas = np.linspace(0.5, 0.99, 50)
    gains = []
    for a in alphas:
        c1, t1 = simulate_wan_pipeline(a, K, 1, rtt_s, n_dispatches, rng)
        c4, t4 = simulate_wan_pipeline(a, K, D, rtt_s, n_dispatches, rng)
        gains.append((c4 / t4) / (c1 / t1))
    gains = np.array(gains)
    return alphas[np.argmax(gains > gain_threshold)]


if __name__ == "__main__":
    rng = np.random.default_rng(0)
    K, D = 8, 4         # draft chunk depth, pipelining depth
    rtt_s = 0.38        # illustrative five-stage WAN round trip (seconds); not a hardware measurement
    N = 300_000         # dispatches per run; large enough to average out Monte Carlo noise

    # 1) D=1 (no pipelining) must exactly reproduce this KB's own validated per-round formula:
    #    every dispatch waits for the true previous result, so nothing is ever wasted.
    for alpha in (0.3, 0.6, 0.74, 0.8, 0.9, 0.97):
        committed, total_time = simulate_wan_pipeline(alpha, K, 1, rtt_s, N, rng)
        empirical_tps = committed / N
        closed_form = expected_tokens_per_step(alpha, K)
        assert abs(empirical_tps - closed_form) < 0.05, (alpha, empirical_tps, closed_form)

    # 2) Pipelining never hurts: throughput at D=4 is never meaningfully below D=1, any alpha.
    for alpha in (0.0, 0.3, 0.6, 0.74, 0.8, 0.9, 0.97, 1.0):
        base_c, base_t = simulate_wan_pipeline(alpha, K, 1, rtt_s, N, rng)
        pipe_c, pipe_t = simulate_wan_pipeline(alpha, K, D, rtt_s, N, rng)
        assert (pipe_c / pipe_t) >= (base_c / base_t) * 0.97, (alpha, pipe_c / pipe_t, base_c / base_t)

    # 3) At alpha=0 pipelining buys nothing (every chunk flushes immediately): D=4 throughput
    #    equals D=1 throughput, both reducing to "one bonus token every round trip."
    c1, t1 = simulate_wan_pipeline(0.0, K, 1, rtt_s, N, rng)
    c4, t4 = simulate_wan_pipeline(0.0, K, D, rtt_s, N, rng)
    assert abs((c1 / t1) - (1.0 / rtt_s)) < 0.02
    assert abs((c4 / t4) - (c1 / t1)) < 0.02

    # 4) At alpha=1 (drafter always right) pipelining buys the full D-fold speedup.
    c1, t1 = simulate_wan_pipeline(1.0, K, 1, rtt_s, N, rng)
    c4, t4 = simulate_wan_pipeline(1.0, K, D, rtt_s, N, rng)
    ratio = (c4 / t4) / (c1 / t1)
    assert (D - 0.1) < ratio < D, ratio

    # 5) The crossover: derived here, not asserted from any external source.
    crossover_alpha = crossover_alpha_for(K, D, rtt_s, N, rng)
    assert 0.5 < crossover_alpha < 0.99

    # 6) Adversarial: the crossover must be the SAME regardless of round-trip time. The gain
    #    ratio (pipelined tok/s over synchronous tok/s) cancels rtt_s out algebraically, since
    #    both throughputs scale as 1/rtt_s for fixed K and D; only ABSOLUTE throughput should
    #    move with rtt, never the relative acceptance threshold that decides whether pipelining
    #    is worth it at all.
    crossover_far_rtt = crossover_alpha_for(K, D, rtt_s * 3.0, N, rng)
    assert abs(crossover_alpha - crossover_far_rtt) < 0.02, (crossover_alpha, crossover_far_rtt)

    # 7) Confirm the other half explicitly: ABSOLUTE throughput at a fixed alpha DOES scale
    #    inversely with rtt, by very nearly the rtt ratio itself.
    c_near, t_near = simulate_wan_pipeline(0.9, K, 1, rtt_s, N, rng)
    c_far, t_far = simulate_wan_pipeline(0.9, K, 1, rtt_s * 3.0, N, rng)
    tp_near, tp_far = c_near / t_near, c_far / t_far
    assert abs((tp_near / tp_far) - 3.0) < 0.1, (tp_near, tp_far)

    print("D=1 matches speculative-decoding.md's own closed form at every tested alpha")
    print(f"alpha=1.0: D={D}/D=1 throughput ratio = {ratio:.3f} (full D-fold pipelining gain)")
    print(f"crossover alpha (K={K}, D={D}) at rtt={rtt_s*1000:.0f}ms  = {crossover_alpha:.3f}")
    print(f"crossover alpha (K={K}, D={D}) at rtt={rtt_s*3000:.0f}ms  = {crossover_far_rtt:.3f}   <- same alpha, 3x the round trip")
    print(f"  (alpha^K at the crossover = {crossover_alpha**K:.3f}: a full K-token chunk lands "
          f"{crossover_alpha**K*100:.1f}% of the time there, independent of rtt)")
    print(f"absolute throughput @ alpha=0.9: {tp_near:.3f} tok/s at rtt={rtt_s*1000:.0f}ms vs "
          f"{tp_far:.3f} tok/s at rtt={rtt_s*3000:.0f}ms (ratio {tp_near/tp_far:.2f}, tracks the 3x rtt change)")
    for a in (0.3, 0.6, 0.8, 0.9, 0.97):
        c1, t1 = simulate_wan_pipeline(a, K, 1, rtt_s, N, rng)
        c4, t4 = simulate_wan_pipeline(a, K, D, rtt_s, N, rng)
        print(f"  alpha={a:.3f} (alpha^K={a**K:.3f}): sync={c1/t1:5.2f} tok/s, pipelined(D={D})={c4/t4:6.2f} tok/s, "
              f"gain={((c4/t4)/(c1/t1)):.2f}x")
    print("ALL ASSERTIONS PASSED")

Executed output:

D=1 matches speculative-decoding.md's own closed form at every tested alpha
alpha=1.0: D=4/D=1 throughput ratio = 4.000 (full D-fold pipelining gain)
crossover alpha (K=8, D=4) at rtt=380ms  = 0.780
crossover alpha (K=8, D=4) at rtt=1140ms  = 0.770   <- same alpha, 3x the round trip
  (alpha^K at the crossover = 0.137: a full K-token chunk lands 13.7% of the time there, independent of rtt)
absolute throughput @ alpha=0.9: 16.117 tok/s at rtt=380ms vs 5.375 tok/s at rtt=1140ms (ratio 3.00, tracks the 3x rtt change)
  alpha=0.300 (alpha^K=0.000): sync= 3.76 tok/s, pipelined(D=4)=  3.75 tok/s, gain=1.00x
  alpha=0.600 (alpha^K=0.017): sync= 6.52 tok/s, pipelined(D=4)=  6.59 tok/s, gain=1.01x
  alpha=0.800 (alpha^K=0.168): sync=11.40 tok/s, pipelined(D=4)= 13.05 tok/s, gain=1.14x
  alpha=0.900 (alpha^K=0.430): sync=16.15 tok/s, pipelined(D=4)= 23.75 tok/s, gain=1.47x
  alpha=0.970 (alpha^K=0.784): sync=21.05 tok/s, pipelined(D=4)= 51.06 tok/s, gain=2.43x
ALL ASSERTIONS PASSED

The crossover comes out at 0.780 versus 0.770 across a 3x change in round-trip time, a difference smaller than the alpha grid's own 0.01 resolution and well inside Monte Carlo noise; it is not moving with RTT. Absolute throughput, in the same run, tracks the RTT ratio almost exactly (3.00x for a 3x RTT change). This is the point stated precisely, not approximately: RTT sets the absolute tok/s scale; it does not move the relative acceptance-rate threshold that decides whether pipelining is worth doing at all. That threshold is set entirely by the chunk depth K and the pipelining depth D. Four further things fall directly out of the model, not out of anyone's report: pipelining is free in the specific sense this model measures, committed tokens per wall-clock second never falls below synchronous decoding's; in the limit of a perfect drafter it buys exactly the pipelining depth D in that throughput; at a hopeless drafter it buys nothing there and gracefully degrades to the synchronous case; and for this page's illustrative K=8/D=4 parameterization, the pipeline stops clearing a 10% throughput margin once per-token acceptance drops below roughly 0.78, with a full chunk landing only about 14% of the time at that point. "Free" is a throughput and latency claim, not a zero-marginal-cost one: a discarded chunk still consumed real verification compute, network bytes, and in-flight buffer memory that could have gone to something else, and in a paid decentralized deployment (see Coordinator placement and Verification, below) it may have consumed a node's compute budget too; the model does not credit or debit any of that against the throughput number. Re-run the model with your own chunk size and pipelining depth before trusting a specific crossover value; it is nonetheless the same order of magnitude independently reported elsewhere for WAN speculative pipelines, which is the kind of cross-check that should increase, not replace, confidence in the mechanism.⁷

The design consequence is a per-round router, not one fixed strategy: pipelined chunks where the immediate continuation is templated, verbatim, or repeated (a cheap n-gram drafter's acceptance is high there), and synchronous, tree-structured verification, checked in a single forward pass, on genuinely novel spans where pipelining cannot help. Only a better drafter (higher α) or a shorter round trip (lower RTT) moves the ceiling for the second case; no amount of pipelining depth does.

Coordinator placement is a real optimization problem, not an afterthought¶

A deployment's entry point, whatever drafts tokens and dispatches chunks but holds none of the served model's layers, still participates in every round trip and has to be placed correctly. This is not a hand-wave: it is a solvable combinatorial optimization over a measured latency graph, with the entry point as the tour's depot. One open-source implementation frames it exactly this way and solves it exactly for small swarms:⁷

# excerpted and adapted for illustration from an open-source WAN pipeline-placement
# module; see the reference below for the full implementation (Held-Karp for n<=16,
# nearest-neighbor + 2-opt above; asymmetric per-node uplink cost, coordinator as depot)
def loop_cost(order, L, c_out, c_in):
    """Total per-traversal latency for a node ordering: entry hop + forward hops + return hop.
    c_out[h]/c_in[t] are the coordinator's own entry/exit costs, separate from inter-node hops
    L[i][j] -- so coordinator placement is optimized jointly with node ORDER, not assumed fixed."""
    if not order:
        return 0.0
    cost = c_out[order[0]] + c_in[order[-1]]
    for a, b in zip(order, order[1:]):
        cost += L[a][b]
    return cost

Two properties of this formulation matter beyond the code itself. First, c_out/c_in are exactly the coordinator's own entry and exit latency to whichever node is placed first and last in the ring, distinct from the inter-node hop costs L[i][j]; a placement search that omits them (optimizing only which nodes hold which layers) is solving a different, easier problem than the one that actually determines end-to-end latency. Second, because home/consumer uplinks are asymmetric (fast download, slow upload), a defensible implementation costs a hop by the sender's upload bandwidth, never the receiver's download bandwidth, and treats an unmeasured or absent uplink measurement as bad rather than as zero cost, so a node that cannot report its own uplink cannot land on a load-bearing hop by default. Both are the kind of detail a placement scheme has to get right and easy to miss when placement is treated as an afterthought to "which nodes hold which layers."

Verification is two separate mechanisms: attribution and coverage prove who and how much, not that the computation was correct¶

The untrusted case has two distinct failure modes, and it takes two distinct mechanisms to answer them, not one. A node might skip its assigned computation entirely (never submit anything for its span), or it might fabricate its computation (submit a plausible-looking but wrong output for a genuine input, or run a cheaper model instead of the one it was assigned). Signed receipts answer the first; they do not, by themselves, answer the second, and treating them as if they did is the overclaim to avoid.

Signed receipts prove attribution and coverage, not correctness. Each stage signs a hash-chain over its own (input, output) activation pairs with its node key. One open-source implementation's coverage check is a direct illustration of the mechanism, not a paraphrase of it:⁷

# excerpted and adapted for illustration; see the reference below for the full module
# (ed25519-signed per-stage hash chains over (input, output) activation pairs)
def verify_coverage(receipts, layer_count, expected_by_signer=None):
    """The job-level check run before paying a node: every receipt must (1) carry a valid
    signature, and (2) the attested layer spans must TILE [0, layer_count) with no gap or
    overlap, every layer attested by exactly one signed node. This is what stops a node from
    being paid for a block it didn't hold; it is not a check that the block was computed
    correctly, only that someone identifiable claimed it and no span was silently skipped."""
    spans = []
    for r in receipts:
        verify_receipt(r, None)                              # raises on bad signature, never returns False
        lo, hi = r["layer_start"], r["layer_end"]
        if not (0 <= lo < hi <= layer_count):
            raise ReceiptError(f"receipt block [{lo}:{hi}] outside [0:{layer_count}]")
        spans.append((lo, hi))
    spans.sort()
    cursor = 0
    for lo, hi in spans:
        if lo != cursor:
            raise ReceiptError(f"layer coverage broken at {cursor}: next block starts {lo} (gap or overlap)")
        cursor = hi
    if cursor != layer_count:
        raise ReceiptError(f"layer coverage ends at {cursor}, expected {layer_count}")

Note precisely what this does and does not prove. It proves a node cannot be paid without a signature only it could produce over the exact layer span it was assigned (attribution: the coordinator itself cannot forge a receipt, since it lacks the node's key), and that the assigned spans really do cover the whole model with no gap a skipped node could hide in (coverage). It does not prove the signed (input, output) pair reflects a genuine forward pass through the real model weights: a node can hash and sign a fabricated output exactly as easily as a real one, and verify_coverage has no way to tell the difference, since it never re-executes anything. At least one open-source implementation's own design documentation is explicit about this gap, describing the receipt's hash-chain as, in its own words, an audit trail today, with a cheap cryptographic proof of correct computation dropping into the same slot only "tomorrow," and names its current answer to fabrication as a separate, second mechanism, covered next, not the receipt itself.⁷

A separate challenge mechanism is the actual answer to a node fabricating its output, and it is a probabilistic, economic deterrent, not a cryptographic guarantee. A verifier derives a deterministic input from a seed, feeds it to both the suspect node's block and a trusted local replica of the same block, and compares the two outputs. One open-source implementation's challenge primitive is a direct illustration:⁷

# excerpted and adapted for illustration; see the reference below for the full module
# (a verifier compares a suspect node's block output against a trusted local recompute
# on a deterministic, seeded challenge input)
def compare(suspect_output, trusted_output, cos_thresh=0.99):
    """PASS = the suspect node ran the real block (cosine similarity ~1, a few ULPs of
    floating-point drift); FAIL = fabricated or wrong-model output (cosine far below
    threshold). Cosine, not a hash: out = block(in) is not bit-reproducible across
    different GPUs, kernels, or quantization runtimes, so a bit-exact match would
    false-positive every honest node on heterogeneous hardware."""
    cos = cosine_similarity(suspect_output, trusted_output)
    rel_norm = relative_norm_difference(suspect_output, trusted_output)
    return cos >= cos_thresh and rel_norm < 0.05

This is the mechanism that actually catches a node that "forwards plausible garbage instead of actually running its assigned block," in that implementation's own framing. Three things about it matter as much as the mechanism itself. First, it is a spot check: a verifier can only challenge a sampled subset of a node's traffic (recomputing every block on every token would double that node's compute cost), so a node that cheats on non-challenged input is not caught on that instance, only probabilistically over repeated sampling. Second, the response to a failed challenge in the cited implementation is economic and reputational, a strike against the node's standing, ejection, and withheld pay, not a cryptographic disqualification; a node has an incentive not to cheat because it is likely to be caught and lose future earnings, not because cheating is made impossible. Third, the same source frames this mechanism's own maturity honestly: it names "layer-block spot-check at scale" as still genuine, open research, not a solved problem, and lists a cheap cryptographic proof of correct computation, replacing re-compute-and-compare entirely, as the explicit future direction once one exists. Treat today's answer as "economically deterred, spot-checked, and openly acknowledged as unproven at scale," not as "cryptographically verified."

Neither mechanism says anything about confidentiality: a node must decrypt its input to compute its assigned layer (or to serve as the trusted replica in a challenge), so it necessarily sees every activation passing through it, signature or not, correct or not. At least one open-source implementation's own transport documentation is explicit that this is unavoidable and names the resulting leakage risk, a malicious node seeing a fraction of a user's tokens, as an open, unsolved problem rather than claiming it away.⁷ Treat "attributed," "spot-checked," and "confidential" as three separate claims; the mechanisms above have a real, inspectable answer only for the first two, and only a probabilistic one for the second.

Fine-grained fault tolerance: a spare stage, not a full ring restart, but not a free resume either¶

WAN pipeline fault tolerance could be described only by analogy to DiLoCo's tolerance for worker churn. A stronger, and more precisely scoped, property is achievable: when a middle stage (not the entry point) dies mid-request, only two things need to happen, not a full ring rebuild. One open-source implementation's healing procedure makes the scope explicit:⁷ the pre-warmed spare taking the dead stage's layer block is (re)launched, and the dead stage's predecessor is relaunched pointed at the spare instead (every other surviving stage already dropped its now-broken link on the failure and simply re-handshakes when the new topology comes up, so it never reloads its own weights). That is a real, bounded-cost repair, two stage (re)launches, not N.

It is not a free resume, and should not be described as one. The healed ring's KV cache for the affected stages is gone; the coordinator's own recovery flow re-prefills the prompt plus every token already committed before the failure, on the healed topology, before decoding continues. The user-visible cost is one pause for that re-prefill, then continuous output resuming from exactly where the request left off, not a restart from scratch and not a truly free continuation. That is still a materially better property than restarting the whole request, and the right property to design toward for any deployment expecting real node churn, but "no weight reload" and "no cost" are different claims; only the first is true here.

How many concurrent requests does it take to hide the WAN bubble?¶

The sections above establish two separate ways to spend a WAN round trip productively: speculative pipelining hides it within one request by keeping several draft chunks in flight, and node churn/placement determine the round trip's fixed cost. Neither answers a third, purely operational question this page otherwise only states qualitatively ("each additional pipeline stage adds one more hop's latency to the round trip," "a pipeline across more nodes is not free the way it is over NVLink," in Failure modes and How to maintain it, below): a production deployment serves many concurrent users, not one Petals-style demo session, so does concurrent request traffic itself hide the per-hop latency, the way pipeline-parallel training hides its fill/drain bubble by running enough micro-batches at once?

It does, up to a hard limit set by the stage count, and the model below derives that limit rather than asserting it. Treat a p-stage WAN pipeline as p slots in series: at any instant, each slot can hold at most one request's in-flight token computation, and a slot completing a hop immediately hands its token to the next slot while a new token (from whichever stream is ready) enters the first slot. A single stream's own tokens are strictly sequential (token t+1's embedding lookup needs token t's sampled id, so one stream can never have two tokens in the pipe at once), but different concurrent streams have no such dependency on each other, so C concurrent streams can fill up to C of the p slots simultaneously, exactly the way independent micro-batches fill a training pipeline's stages.

# wan_decode_concurrency.py -- validated: does concurrent request traffic hide a WAN
# pipeline's per-hop round-trip latency the way concurrent micro-batches hide a training
# pipeline's fill/drain bubble (train-pipeline-parallel.md), and how many concurrent
# streams does it take? A synchronous discrete-time simulation of C independent decode
# streams perpetually circulating through a p-stage WAN pipeline, cross-checked against
# Little's Law (L = lambda * W) computed independently from the same run.
# Run: python3 wan_decode_concurrency.py
from __future__ import annotations

from collections import deque


def simulate_wan_decode_pipeline(p: int, rtt_hop_s: float, C: int,
                                  n_ticks: int, warmup_ticks: int) -> tuple[float, float]:
    """C independent decode streams perpetually circulating through a p-stage WAN
    pipeline (one hop = rtt_hop_s). A stream's completed token immediately becomes
    ready to start its next round trip (joins the admission queue for a free slot 0);
    only p tokens can be physically in the pipeline at once. Returns (throughput
    tokens/s, average per-token round-trip latency seconds), measured after warmup."""
    waiting = deque(range(C))
    ready_tick = {i: 0 for i in range(C)}
    pipeline: list[int | None] = [None] * p
    completions = 0
    total_latency_ticks = 0
    for t in range(1, n_ticks + 1):
        exiting = pipeline[p - 1]
        for i in range(p - 1, 0, -1):
            pipeline[i] = pipeline[i - 1]
        pipeline[0] = None
        if exiting is not None:
            if t > warmup_ticks:
                completions += 1
                total_latency_ticks += (t - ready_tick[exiting])
            ready_tick[exiting] = t
            waiting.append(exiting)
        if pipeline[0] is None and waiting:
            pipeline[0] = waiting.popleft()
    measured_ticks = n_ticks - warmup_ticks
    throughput = completions / (measured_ticks * rtt_hop_s)
    avg_latency_s = (total_latency_ticks / completions) * rtt_hop_s if completions else float("nan")
    return throughput, avg_latency_s


if __name__ == "__main__":
    p = 5
    rtt_s = 0.38          # same illustrative five-stage WAN round trip as wan_pipeline_specdec.py
    rtt_hop_s = rtt_s / p
    n_ticks, warmup = 200_000, 50

    # 1) C=1: a single stream is bound by its own full round trip, matching this page's
    #    "roughly one token per network round trip" claim for the no-specdec case.
    tp1, w1 = simulate_wan_decode_pipeline(p, rtt_hop_s, 1, n_ticks, warmup)
    assert abs(tp1 - 1.0 / rtt_s) < 1e-9, tp1
    assert abs(w1 - rtt_s) < 1e-9, w1

    # 2) Throughput scales linearly with C while C <= p (each extra concurrent stream
    #    fills one more of the p pipeline slots, no contention yet).
    results = {}
    for C in (1, 2, 3, 4, 5, 6, 8, 10, 15, 20):
        tp, w = simulate_wan_decode_pipeline(p, rtt_hop_s, C, n_ticks, warmup)
        results[C] = (tp, w)
        if C <= p:
            assert abs(tp - C / rtt_s) < 1e-9, (C, tp)

    # 3) Saturation: throughput never exceeds p/rtt_s (the hop-rate bound), and C > p
    #    buys no further throughput (extra streams only wait longer, never help).
    ceiling = p / rtt_s
    for C, (tp, w) in results.items():
        assert tp <= ceiling + 1e-9, (C, tp, ceiling)
    assert abs(results[p][0] - ceiling) < 1e-9
    assert abs(results[20][0] - ceiling) < 1e-6

    # 4) Little's Law, checked against the simulation's own independently measured
    #    throughput and average latency, not assumed: L = throughput * W must equal
    #    C (the fixed population of this closed system) at every concurrency level.
    for C, (tp, w) in results.items():
        assert abs(C - tp * w) < 1e-6, (C, tp, w, tp * w)

    # 5) Efficiency relative to the p-stage ceiling is exactly C/p for C <= p: each
    #    concurrent stream fills one more of the p slots, linearly, until all p are busy.
    for C in (1, 2, 3, 4, 5):
        tp, _ = results[C]
        efficiency = tp / ceiling
        assert abs(efficiency - C / p) < 1e-9, (C, efficiency)

    print(f"p={p} stages, rtt_s={rtt_s}s (rtt_hop={rtt_hop_s*1000:.1f}ms), ceiling={ceiling:.3f} tok/s")
    for C in (1, 2, 3, 4, 5, 6, 8, 10, 15, 20):
        tp, w = results[C]
        print(f"  C={C:2d}: throughput={tp:6.3f} tok/s ({tp/ceiling*100:5.1f}% of ceiling), "
              f"avg round-trip latency={w*1000:7.1f}ms, Little's Law C~=tp*w: {tp*w:.4f}")
    print("ALL ASSERTIONS PASSED")

Executed output:

p=5 stages, rtt_s=0.38s (rtt_hop=76.0ms), ceiling=13.158 tok/s
  C= 1: throughput= 2.632 tok/s ( 20.0% of ceiling), avg round-trip latency=  380.0ms, Little's Law C~=tp*w: 1.0000
  C= 2: throughput= 5.263 tok/s ( 40.0% of ceiling), avg round-trip latency=  380.0ms, Little's Law C~=tp*w: 2.0000
  C= 3: throughput= 7.895 tok/s ( 60.0% of ceiling), avg round-trip latency=  380.0ms, Little's Law C~=tp*w: 3.0000
  C= 4: throughput=10.526 tok/s ( 80.0% of ceiling), avg round-trip latency=  380.0ms, Little's Law C~=tp*w: 4.0000
  C= 5: throughput=13.158 tok/s (100.0% of ceiling), avg round-trip latency=  380.0ms, Little's Law C~=tp*w: 5.0000
  C= 6: throughput=13.158 tok/s (100.0% of ceiling), avg round-trip latency=  456.0ms, Little's Law C~=tp*w: 6.0000
  C= 8: throughput=13.158 tok/s (100.0% of ceiling), avg round-trip latency=  608.0ms, Little's Law C~=tp*w: 8.0000
  C=10: throughput=13.158 tok/s (100.0% of ceiling), avg round-trip latency=  760.0ms, Little's Law C~=tp*w: 10.0000
  C=15: throughput=13.158 tok/s (100.0% of ceiling), avg round-trip latency= 1140.0ms, Little's Law C~=tp*w: 15.0000
  C=20: throughput=13.158 tok/s (100.0% of ceiling), avg round-trip latency= 1520.0ms, Little's Law C~=tp*w: 20.0000
ALL ASSERTIONS PASSED

Three things fall directly out of the executed run. First, concurrency really does hide the bubble: throughput climbs linearly from 1/RTT at a single stream to the full p/RTT hop-rate ceiling at C = p concurrent streams (5x higher at p=5, matching this page's own five-stage illustrative RTT), for free, with no change to the model, the drafter, or the network. Second, that gain is strictly capped at C = p: pushing concurrency to C = 20 buys nothing further, because only p token computations can physically occupy the pipeline's p slots at once; the extra 15 streams in that run sit in an admission queue, and their measured round-trip latency grows linearly with the excess (1520ms at C=20 versus 380ms at C<=5, a queueing cost with no matching throughput benefit). Third, Little's Law (L = λW, the number of customers in a closed system equals throughput times average time-in-system) holds exactly at every concurrency level in this deterministic simulation, checked against the run's own independently measured throughput and latency rather than assumed; that agreement is a genuine internal-consistency check on the simulation, not a restatement of a textbook identity. This is a distinct derivation from pipeline parallelism's own fill-drain bubble_fraction(p, m) = (p-1)/m, which models a one-shot batch of m micro-batches passing through once; here, request streams circulate perpetually rather than passing through once, so the relevant quantity is simply min(C, p)/p steady-state utilization, not the fill-drain formula, though the underlying intuition, enough concurrent work in flight to keep every stage busy, is the same one that formula captures for training.

The design consequence: provision concurrent-request capacity against the pipeline's own stage count, not against network bandwidth alone. A deployment running consistently below C = p concurrent decode streams is leaving throughput on the table for free (more concurrency costs nothing and helps up to that point); one running far above C = p is buying latency with no throughput return and should either add pipeline depth (more parallel pipelines, see How to develop with it, below) or cap admission rather than let queueing delay grow unbounded. Swap in your own stage count and measured per-hop RTT before trusting a specific C = p threshold; the deterministic, equal-hop-time assumption here is a simplification real jittery WAN links will not exactly match, noted in Open questions, below.

How to develop with it: placement and pipeline-selection scheduling¶

The core engineering problem is a placement and pipeline-selection problem: which nodes host which layers or experts, and which chain of nodes a given request's pipeline walks at request time. Parallax's open-source scheduler implements this as two phases: a layer-allocation phase that decides which nodes join a pipeline and how many layers each holds, and a request-routing phase, built on dynamic programming over measured RTT and per-node latency, that decides which chain of nodes a request actually walks.⁵ Helix's max-flow/MILP formulation solves the same joint placement-and-routing problem with a different algorithm.⁴ Two things carry over from the training case in Overlay & mesh networking: the inter-node transport for activations is gloo/TCP, not NCCL-over-IB, since there is no fast fabric between non-colocated nodes; and a churny node pool (volunteers joining and leaving) needs fault-tolerant membership, not a fixed WORLD_SIZE. Parallax's scheduler answers this concretely: nodes join and leave through a non-blocking event queue, a heartbeat check evicts unresponsive nodes after a configurable timeout, and a coefficient-of-variation threshold on per-layer load triggers an automatic global rebalance, the same kind of live membership handling DiLoCo's decentralized training path needs, applied to inference.⁵

The layer-allocation phase's two algorithms are worth looking at directly rather than describing in the abstract, because the design choices are concrete and reproducible. First, splitting a pipeline's layers across its nodes is a water-filling problem: find a "water level" λ such that each node i gets min(capacity_i, λ · power_i) layers, so faster nodes get proportionally more layers without exceeding what they can hold in memory. Parallax's actual implementation solves for λ by bisection:⁵

# Quoted, lightly trimmed for illustration, from Parallax's src/scheduling/layer_allocation.py,
# BaseLayerAllocator.adjust_pipeline_layers (GradientHQ/parallax, Apache-2.0).
def total_at(lmbd: float) -> float:
    return sum(min(caps[i], lmbd * compute_powers[i]) for i in range(n))

lo, hi = 0.0, max((caps[i] / compute_powers[i]) for i in range(n))
for _ in range(self.water_filling_max_iterations):
    mid = 0.5 * (lo + hi)
    if total_at(mid) >= total_layers:
        hi = mid
    else:
        lo = mid
lam = hi
target = [min(caps[i], lam * compute_powers[i]) for i in range(n)]
# (then floor + largest-remainder integerization, respecting each node's capacity cap)

Second, deciding how many parallel pipelines to form out of a pool of heterogeneous nodes is a dynamic program scoring each candidate pipeline count k by Z(k) = k^2 / s*(k), where s*(k) is the minimum total number of node-stages needed to realize k disjoint, fully-covering pipelines. The allocator's own source comment gives a concrete worked example of why this beats a greedy "just fill one pipeline" policy: for node capacities (40, 40, 20, 20, 10, 10) and a 70-layer model, a single pipeline needs only 2 stages (40 + 30 of a second 40-capacity node), but two pipelines of 3 stages each (40+20+10 twice) score higher under Z(k), so the allocator should prefer 2 pipelines over 1.⁵ The model below reproduces that exact example independently, computing s*(k) by brute force over this small instance to check the claim, alongside the water-filling function reproduced faithfully from the excerpt above:

# parallax_scheduler_check.py -- validated: reproduces and cross-checks the two core
# algorithms behind Parallax's layer-allocation scheduler (GradientHQ/parallax,
# src/scheduling/layer_allocation.py): (1) the water-filling in-place rebalance that
# splits a pipeline's decoder layers across its nodes proportional to compute power,
# subject to per-node capacity caps, and (2) the DP allocator's Z(k) = k^2 / s*(k)
# scoring rule for choosing how many parallel pipelines to form, reproduced on the
# allocator's OWN docstring example. This is an independent re-implementation to
# validate the published algorithm, not a copy of Parallax's source.
# Run: python3 parallax_scheduler_check.py
from __future__ import annotations

from itertools import product
from math import floor


def water_fill_layers(caps: list[int], power: list[float], total_layers: int,
                      iterations: int = 60) -> list[int]:
    """Faithful reproduction of Parallax's adjust_pipeline_layers water-filling core:
    find lambda such that sum_i min(cap_i, lambda * power_i) == total_layers via bisection,
    then integerize with floor + largest-remainder, respecting each node's capacity cap."""
    n = len(caps)
    assert sum(caps) >= total_layers, "pipeline capacity must cover the model"

    def total_at(lmbd: float) -> float:
        return sum(min(caps[i], lmbd * power[i]) for i in range(n))

    lo, hi = 0.0, max(caps[i] / power[i] for i in range(n))
    for _ in range(iterations):
        mid = 0.5 * (lo + hi)
        if total_at(mid) >= total_layers:
            hi = mid
        else:
            lo = mid
    lam = hi
    target = [min(caps[i], lam * power[i]) for i in range(n)]
    counts = [min(caps[i], int(floor(target[i]))) for i in range(n)]
    remaining = total_layers - sum(counts)
    if remaining > 0:
        order = sorted(range(n), key=lambda i: -(target[i] - counts[i]))
        for i in order:
            if remaining == 0:
                break
            room = caps[i] - counts[i]
            if room > 0:
                counts[i] += 1
                remaining -= 1
    return counts


def s_star_bruteforce(caps: list[int], total_layers: int, k_target: int) -> float:
    """Exact minimum total stages to realize k_target disjoint pipelines, each summing
    to >= total_layers, by brute-force search over ways to partition/select nodes.
    Tractable only for small node counts; this is the DP's target, computed exactly
    here for cross-validation, not the DP itself."""
    n = len(caps)
    best = float("inf")
    for assignment in product(range(k_target + 1), repeat=n):
        sums = [0] * k_target
        used = 0
        for node_i, pipe in enumerate(assignment):
            if pipe > 0:
                sums[pipe - 1] += caps[node_i]
                used += 1
        if all(s >= total_layers for s in sums):
            best = min(best, used)
    return best


if __name__ == "__main__":
    # 1) Water-filling: conservation, capacity respect, and proportional-to-power split.
    caps = [24, 16, 8]
    power = [100.0, 50.0, 25.0]   # node 0 has 4x node 2's compute, 2x node 1's
    L = 30
    counts = water_fill_layers(caps, power, L)
    assert sum(counts) == L, counts
    assert all(0 <= counts[i] <= caps[i] for i in range(3)), counts
    assert counts[0] >= counts[1] >= counts[2], counts

    # 2) Adversarial: a node whose capacity is the binding constraint gets EXACTLY its
    #    cap, not more; water level must respect it rather than overfilling.
    caps2 = [5, 100, 100]
    power2 = [100.0, 100.0, 100.0]
    counts2 = water_fill_layers(caps2, power2, total_layers=90)
    assert counts2[0] == 5, counts2
    assert abs(counts2[1] - counts2[2]) <= 1
    assert sum(counts2) == 90

    # 3) Edge: single node, must take exactly the required layers (never more than its cap).
    assert water_fill_layers([50], [10.0], 30) == [30]

    # 4) Monotonicity: raising a node's compute power (others fixed) never DECREASES its
    #    water-filled share, holding total layers and every cap fixed.
    base = water_fill_layers([20, 20, 20], [10.0, 10.0, 10.0], 40)
    boosted = water_fill_layers([20, 20, 20], [20.0, 10.0, 10.0], 40)
    assert boosted[0] >= base[0], (base, boosted)
    assert sum(boosted) == 40

    print(f"water-filling: caps={caps}, power={power}, L={L} -> layers={counts} (sum={sum(counts)})")
    print(f"capacity-bound case: caps={caps2}, L=90 -> layers={counts2} (node 0 saturates at cap={caps2[0]})")

    # 5) Reproduce the allocator's OWN docstring example: capacities (40, 40, 20, 20, 10, 10),
    #    total_layers=70, "should yield (40,20,10)+(40,20,10) instead of a single 2-stage
    #    pipeline (40+30)". Verify k=2 beats k=1 under Z(k) = k^2 / s*(k), computed exactly
    #    by brute force over this small (6-node) instance.
    docstring_caps = [40, 40, 20, 20, 10, 10]
    L_doc = 70
    s1 = s_star_bruteforce(docstring_caps, L_doc, k_target=1)
    s2 = s_star_bruteforce(docstring_caps, L_doc, k_target=2)
    assert s1 == 2, s1
    assert s2 == 6, s2
    z1 = (1 ** 2) / s1
    z2 = (2 ** 2) / s2
    assert z2 > z1, (z1, z2)
    s3 = s_star_bruteforce(docstring_caps, L_doc, k_target=3)
    assert s3 == float("inf"), s3

    print(f"docstring example: caps={docstring_caps}, L={L_doc}")
    print(f"  s*(1)={s1} -> Z(1)={z1:.3f}   (a single 2-stage pipeline: 40 + 30 of the second 40)")
    print(f"  s*(2)={s2} -> Z(2)={z2:.3f}   (two 3-stage pipelines: (40+20+10) x 2)")
    print(f"  s*(3)={s3}                (infeasible: total capacity 120 < 3 x 70 = 210)")
    print(f"  Z(2) > Z(1): the allocator correctly prefers 2 pipelines over 1, matching its own docstring claim")
    print("ALL ASSERTIONS PASSED")

Executed output:

water-filling: caps=[24, 16, 8], power=[100.0, 50.0, 25.0], L=30 -> layers=[17, 9, 4] (sum=30)
capacity-bound case: caps=[5, 100, 100], L=90 -> layers=[5, 43, 42] (node 0 saturates at cap=5)
docstring example: caps=[40, 40, 20, 20, 10, 10], L=70
  s*(1)=2 -> Z(1)=0.500   (a single 2-stage pipeline: 40 + 30 of the second 40)
  s*(2)=6 -> Z(2)=0.667   (two 3-stage pipelines: (40+20+10) x 2)
  s*(3)=inf                (infeasible: total capacity 120 < 3 x 70 = 210)
  Z(2) > Z(1): the allocator correctly prefers 2 pipelines over 1, matching its own docstring claim
ALL ASSERTIONS PASSED

This confirms, independently, both that the water-filling function behaves the way a capacity-aware proportional split should (conservation, capacity-binding, monotonicity in compute power) and that the DP allocator's Z(k) = k^2 / s*(k) scoring rule genuinely produces the design behavior its own source comment claims: given the choice, it prefers more, shorter pipelines over one long one when doing so uses the heterogeneous node pool more evenly. Forming multiple parallel pipelines this way, rather than one deep pipeline, is also the direct answer to the concurrency question above for a large node pool: each pipeline gets its own p and can independently host up to p concurrent streams before saturating, so k pipelines of depth p_k each raise the whole deployment's concurrency ceiling to sum(p_k), not a single pipeline's p. The request-routing phase's dynamic program (RTT as edge cost, per-node latency as vertex cost) and its "turning point" detection, truncating a node's served shard when the optimal path stops using all of it, are documented in the same source but not re-derived here; treat that part as read and cited, not independently validated.

How to maintain it¶

Watch the same WAN-link health signals as the training case: per-peer handshake age, keepalive, and measured RTT/goodput (Overlay & mesh networking), even though this pattern's own hot-path activation transport (gloo/TCP, or Parallax's Lattica P2P transport) is deliberately not the training overlay itself; the link characteristics that overlay is validated against still apply to whatever transport actually carries the activation hops.
Watch node churn and load-imbalance signals directly, not just RTT. Parallax's scheduler already tracks two of them: a heartbeat-timeout eviction for unresponsive nodes, and a coefficient-of-variation threshold on per-layer load that triggers an automatic global rebalance.⁵ A pool rebalancing constantly is a leading indicator the node pool is too volatile for its current pipeline-count/depth choice, before a user-visible latency SLO catches it.
Watch per-hop latency accumulation, and whether concurrency is actually amortizing it. Each additional pipeline stage adds one more hop's latency to the single round trip every committed token needs, not a separate RTT per stage; a pipeline across more nodes is not free the way it is over NVLink. Use the concurrency model above to check whether current concurrent-stream volume is near, at, or well past a given pipeline's stage count p; running consistently below p means throughput is being left on the table for free, and running far above it means queueing latency is growing for no further throughput gain.

Running it in production¶

	Pattern A: region-replica routing	Pattern B: cross-WAN model-parallel split
Maturity	Production-composable from pieces this KB already documents (engines, disaggregation, routing, overlay networking); the composition itself is not a single vendor-shipped product as of this writing	Research-stage: published, working, open-source systems in the Petals/HexGen/Helix/Parallax lineage, not a supported mode of vLLM/SGLang/TensorRT-LLM/Dynamo
What moves over WAN	By default, requests and responses only, the proxy redirects to wherever the cache lives; a session migrated to reduce a long conversation's per-turn round trip, or pinned by a redirect-blocking constraint, additionally moves the KV cache itself, once, to its new serving region	Per-token activations between pipeline stages; KV cache never leaves the node that owns those layers
Latency ceiling	Roughly the same as any single region's deployment, plus one WAN hop for the request/response	Grows with pipeline depth: each stage adds one more hop to the single round trip every committed token needs (not a separate RTT per stage); ranges from about 1 decoding step/s without WAN-aware speculative decoding¹ to an order of magnitude or more higher with it, at larger model scale, in at least one documented implementation⁷
When it applies	A full replica fits at every site you have	No site can host a full replica; capacity is fundamentally scattered
Where the KV-cache decision lives	The cross-region proxy (a GORGO-style holistic cost model, see region-replica routing)	Not a decision: each node's KV is local to its own layers by construction

Failure modes¶

Splitting prefill and decode across non-colocated sites. Reintroduces a WAN KV-cache handoff exactly where disaggregated inference says to keep it on a fast fabric; slower than either colocating prefill/decode per region (region-replica routing) or not moving the KV cache at all (this pattern).
Treating this pattern as a drop-in vLLM deployment mode. It is a different system (Petals/HexGen/HexGen-2/Helix/Parallax), not a flag on a production engine; going in expecting datacenter-class latency will disappoint.
Putting the decode hot path on the WAN overlay. The overlay in Overlay & mesh networking is validated for control traffic and low-frequency sync, not a step-synchronous loop; this is the same mistake the DiLoCo recipe warns against for training, applied to serving.
Assuming an MoE model shards across non-colocated nodes as easily as a dense one. Placement is easier, an expert is already a discrete unit, but the every-layer all-to-all is a tighter synchronization requirement than this pattern's dense case (one hop per pipeline stage); see Does MoE make this easier? before assuming a naive HexGen/Helix-style placement decentralizes MoE serving well at production concurrency.
Assuming the throughput ceiling is fixed at roughly one token per round trip, or that concurrency alone fixes it. That number describes a pipeline without speculative decoding built into the WAN hop. Adding speculative decoding, with a working per-round router, asynchronous pipelining, and a compiled draft model, has demonstrably reached an order of magnitude or more higher throughput on larger models. Concurrent request traffic raises throughput too, but only up to the pipeline's own stage count p (see the concurrency model above); past that point, more concurrent requests add queueing latency with no further throughput, so do not provision concurrency as a substitute for either speculative decoding or more/deeper pipelines.
Treating a signed receipt's coverage check as proof the computation was correct. A signed hash-chain over (input, output) activations proves attribution (only the assigned node's key could have produced it) and coverage (no layer was silently skipped); it does not prove the output was really computed by running the assigned layers, since a node can sign a fabricated output as easily as a genuine one. Catching fabrication needs the separate challenge mechanism above, and even that is a probabilistic spot check, not a guarantee.
Treating a signed compute receipt, or a passed challenge, as a confidentiality guarantee. A node that faithfully computes its layer, or that serves as the trusted replica in a challenge, has still seen every activation that passed through it; neither mechanism says anything about whether the data stayed private. Do not route sensitive prompts through untrusted stages on the assumption that "receipts are on" or "challenges are on" solves this.

Open questions & validation¶

Pattern B's throughput and latency ceiling versus node count and network heterogeneity, reproduced on your own volunteer/decentralized pool rather than taken from the papers' reported numbers.
Whether Semantic Parallelism's or MoEShard's all-to-all mitigations (fast-fabric only, as cited above) can be adapted to a WAN link at all, or whether sharding whole layers (sidestepping expert-parallel placement over WAN entirely, as noted above) is the only viable strategy until a WAN-native expert-placement scheme is published.
The specific throughput, latency, and verification-overhead figures behind the acceptance-gated law and optimization sequence above come from a single, non-peer-reviewed, self-published source and have not been independently reproduced in this KB; treat the mechanism (the α^K law, the direction and rough magnitude of each optimization step) as the load-bearing claim, and the exact numbers as illustrative until reproduced independently.
The concurrency model above assumes deterministic, equal per-hop RTT and a single shared pipeline. Real WAN links have jittery, asymmetric RTT, and a large deployment forms multiple parallel pipelines rather than one deep one (see How to develop with it, above). Whether jitter shifts the C = p saturation point, and how the model composes across several parallel pipelines each with its own p and RTT distribution, is not modeled here.
The layer-block challenge described above is a spot check, not a per-token guarantee: what sampling rate is needed to bound a rational node's expected gain from cheating below its expected loss from an eventual strike, given the cited implementation's own economic response, is not modeled anywhere in this KB, and the source itself names spot-checking at scale as open research rather than a solved problem.

References¶

Borzunov et al., Petals: Collaborative Inference and Fine-tuning of Large Models (arXiv 2209.01188): https://arxiv.org/abs/2209.01188
Jiang et al., HexGen: Generative Inference of Large Language Model over Heterogeneous Environment (arXiv 2311.11514): https://arxiv.org/abs/2311.11514
HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment (arXiv 2502.07903): https://arxiv.org/abs/2502.07903
Mei et al., Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow (arXiv 2406.01566, ASPLOS 2025): https://arxiv.org/abs/2406.01566
Parallax: Efficient LLM Inference Service over Decentralized Environment (arXiv 2509.26182): https://arxiv.org/abs/2509.26182, implementation: https://github.com/GradientHQ/parallax
Ryabinin and Gusev, Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts (arXiv 2002.04013): https://arxiv.org/abs/2002.04013
A self-published, non-peer-reviewed technical report and its accompanying open-source implementation, documenting WAN-aware speculative decoding, coordinator placement, lossless sampling, mid-stage fault tolerance, and per-stage attribution/coverage receipts plus a separate compute-fabrication challenge for pipeline-parallel serving; not independently reproduced in this KB, archived at: https://doi.org/10.5281/zenodo.21178430 and https://github.com/leyten/shard, with the specific modules excerpted or cited above at shard/topology.py (placement search), shard/receipt.py (signed attribution and coverage verification), shard/challenge.py (trusted-recompute fabrication challenge), phase0/wire.py (authenticated, pickle-free transport), phase0/specsample.py (lossless speculative sampling), and phase0/heal.py (mid-request stage healing)
Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling (arXiv 2503.04398): https://arxiv.org/abs/2503.04398
MoEShard: Accelerating MoE Model Inference with Expert Sharding (arXiv 2503.08467): https://arxiv.org/abs/2503.08467

Borzunov et al., arXiv 2209.01188, abstract: "we propose Petals... a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties... running inference of BLOOM-176B on consumer GPUs with $\approx$ 1 step per second, which is enough for many interactive LLM applications." ↩↩↩↩↩
Jiang et al., arXiv 2311.11514, abstract: "deploying such services in a heterogeneous and cross-datacenter setting to mitigate the substantial inference costs typically associated with a single centralized datacenter... HexGen can choose to achieve up to 2.3 times lower latency deadlines or tolerate up to 4 times more request rates compared with the homogeneous baseline given the same budget," evaluated serving Llama-2 (70B). ↩↩↩
arXiv 2502.07903, abstract: "a scheduling algorithm that formalizes the allocation of disaggregated LLM inference computations and communications over heterogeneous GPUs and network connections as a constraint optimization problem... graph partitioning and max-flow algorithms to co-optimize resource allocation, parallel strategies... and inter-phase key-value (KV) cache communications... up to a 2.0 times and on average a 1.3 times improvement in serving throughput, reduces the average inference latency by 1.5 times... and achieves comparable inference performance with a 30% lower price budget," evaluated on OPT (30B) and Llama-2 (70B). ↩↩
Mei et al., arXiv 2406.01566, abstract: "formulate inference computation of LLMs over heterogeneous GPUs and network connections as a max-flow problem on directed, weighted graphs... a mixed integer linear programming (MILP) algorithm... evaluation on several heterogeneous clusters ranging from 24 to 42 GPU nodes shows that Helix improves serving throughput by up to 3.3x and reduces prompting and decoding latency by up to 66% and 24%, respectively." ↩↩↩
arXiv 2509.26182, abstract: "Parallax decomposes planning into (i) model allocation, which places layers of each replica across diverse GPUs to jointly optimize latency and throughput under memory and link-bandwidth constraints, and (ii) request-time GPU pipeline selection, which stitches layers from different replicas into end-to-end execution chains... evaluated on open-source LLMs deployed over real volunteer nodes." GradientHQ/parallax (Apache-2.0, verified against src/scheduling/README.md and the layer_allocation.py/request_routing.py source, 2026-07-06): a GreedyLayerAllocator (maximize pipeline count, minimize stages) and a DynamicProgrammingLayerAllocator (choose pipeline count k to maximize k^2 / s*(k)) for layer allocation, both followed by a water-filling rebalance across each pipeline's nodes proportional to compute and bandwidth; a DynamicProgrammingRouting router (RTT as edge cost, per-node layer latency as vertex cost) that detects "turning points" and truncates shards to the path actually used; dynamic node join/leave via a non-blocking event queue, heartbeat-based eviction on a configurable timeout, and automatic global rebalancing on a coefficient-of-variation load-imbalance threshold; leftover capacity beyond a full pipeline is used to replicate the lightest-loaded layers rather than sit idle. Node backends: SGLang and vLLM for GPUs, MLX LM for Apple Silicon Macs, over a P2P transport (Lattica, a separate GradientHQ project) rather than a WAN overlay of the kind Overlay & mesh networking documents for training. ↩↩↩↩↩↩↩↩
Ryabinin and Gusev, arXiv 2002.04013, abstract: "we propose Learning@home: a novel neural network training paradigm designed to handle large amounts of poorly connected participants," the decentralized Mixture-of-Experts lineage that precedes Petals. ↩
A self-published, non-peer-reviewed technical report and its accompanying open-source implementation (see References; not independently reproduced in this KB, though this page's own executed simulation above independently derives a crossover of the same order of magnitude for one illustrative parameterization) document, on pipeline-parallel deployments of large MoE models across untrusted consumer and prosumer GPUs in multiple countries: an acceptance-gated speculative-decoding law for WAN links reporting a crossover near a per-token draft-acceptance rate of about 0.8 for their tested chunk depth, with the strongest cited published novel-text drafter reaching about 0.74; an optimization sequence moving from a latency-bound baseline through speculative decoding, a direct-return topology, and asynchronous pipelining, to a further gain from compiling the draft model into a static execution graph once the network was no longer the dominant cost; a roughly 40 percent latency cut from relocating a layer-free coordinator into the same region as the rest of the pipeline (the repository's topology.py implements this placement as an exact Held-Karp tour search for small node counts and a nearest-neighbor-plus-2-opt heuristic above that, over a measured latency graph with the coordinator as the tour's depot); speculative-sampling rejection (the repository's specsample.py) making nonzero-temperature and top-p/top-k sampling provably lossless, not only greedy decoding, by specializing the same accept/residual rule this KB's own speculative-decoding.md proves for a general drafter to the case of a deterministic (point-mass) proposal distribution; a mid-generation middle-stage failure healed by relaunching only the pre-warmed spare and the dead stage's predecessor, with every other surviving stage keeping its loaded weights and simply re-handshaking its links, followed by a real re-prefill of the prompt plus all previously committed tokens on the healed topology before decoding resumes (the repository's heal.py; a bounded-cost repair with one user-visible pause, not a free or instant resume); signed per-stage activation hash-chain receipts (the repository's receipt.py, ed25519-signed, checked for full non-overlapping layer-count coverage, fail-closed), proving attribution and coverage at a measured cost under 1 percent of stage compute but, in the repository's own words, functioning today as an audit trail rather than a proof of correct computation, with a cryptographic proof-of-compute named as a future upgrade to the same slot; a separate layer-block challenge (the repository's challenge.py) that spot-checks a suspect node's output against a trusted node's recompute of the same deterministic, seeded input, compared by cosine similarity rather than exact hash because floating-point output is not bit-reproducible across different GPUs, kernels, or quantization runtimes, backed by an economic response (strike, reputation, ejection, withheld pay) rather than a cryptographic one, and named by the repository's own roadmap documentation as still genuine, open research at scale, not a solved problem; and alongside all of this, the repository's own transport documentation (wire.py) acknowledging that a node must still decrypt and therefore see the activations it processes, naming that leakage risk as an open, unsolved problem rather than claiming it away. ↩↩↩↩↩↩↩↩↩↩↩
Semantic Parallelism, arXiv 2503.04398, abstract: expert parallelism's efficiency "is largely bounded by inter-device communication" from all-to-all collectives; proposes model-data co-scheduling (offline expert clustering by co-activation tendency, online inter-request rebatching for Attention-DP, online intra-request token reshuffling for Attention-TP), implemented in SGLang as Sem-MoE, "effectively reduce[s] the all-to-all communication volume in EP and achieve[s] superior inference throughput compared to existing solutions." ↩
MoEShard, arXiv 2503.08467, abstract: "achieves perfect load balancing through tensor sharding of MoE experts," row- and column-wise decomposition of expert matrices rather than capacity-factor heuristics or token dropping, "demonstrating speedups of up to 6.4x in time to first token (TTFT)" versus DeepSpeed on encoder-based MoE architectures. ↩