Markdown

Mesh LLM and Skippy stage splits¶

Scope: running one OpenAI-compatible inference endpoint over a peer-to-peer mesh of machines that may sit in different datacentres, on different continents, behind NAT, and on different vendors' GPUs, using Mesh LLM and its embedded Skippy staged runtime. Covers the QUIC/iroh transport and gossip control plane, Nostr and invite-token discovery, layer-package weight distribution, the latency-aware stage-count planner, the activation wire-dtype correctness gate, recurrent-state topology affinity, and the trust boundary of a public mesh. This is a per-technology page under cross-WAN model-parallel inference, which covers the pattern in general; read that page first for the Petals/HexGen/Helix/Parallax lineage and for the speculative-decoding and verification mechanics that Mesh LLM does not currently implement.

Mesh LLM is the worked example of the layer cross-WAN model-parallel inference names but does not open: the "purpose-built transport" a decentralized pipeline actually runs on. It is Apache-2.0, written in Rust, and under active development (HEAD c14e458, 2026-07-12). Every claim on this page was checked against that tree's source, not only its documentation; where its own docs and its code disagree, this page follows the code and says so.

What it is¶

A single mesh-llm binary that is simultaneously a node, a router, and an OpenAI-compatible API server. Start one; start more; they find each other and become one logical endpoint at http://localhost:9337/v1. A request is routed by its model field to whichever peer serves that model. When no single machine can hold the model, the coordinator plans a Skippy stage split: contiguous layer ranges assigned to different peers, with the forward pass running as a pipeline across them.

Three things distinguish it from the systems in the cross-WAN lineage:

The transport is the product. Petals, HexGen, Helix and Parallax are schedulers and placement engines; their transport is gloo/TCP or a separate project (Parallax's Lattica). Mesh LLM ships the transport: QUIC via iroh 1.0.1, with NAT hole-punching, relay fallback, dial-by-public-key, gossip membership, and admission control, all in the same binary.
Layers are a distribution format, not just a partition. A layer package is a Hugging Face repo holding a model-package.json manifest plus GGUF fragments, so a peer downloads only the fragments for the layer range it was assigned. Weight distribution to peers is a solved, content-addressed problem rather than an assumed precondition.
It refuses to be a datacentre. Once any peer reports a stage-transfer latency, the planner ranks candidates by estimated decode-network TPOT first, so it will pick a lower-context two-stage topology over a higher-context four-stage one. It is engineered around the assumption that every added hop is a loss, which is the correct assumption for a WAN and the opposite of a fast-fabric engine's. (With no latency inputs at all it falls back to highest-context-first, so this behaviour depends on the mesh actually measuring its links.)

llama.cpp does the tensor math. Skippy owns the stage ABI, the activation wire format, and topology planning. Mesh owns discovery, admission, gossip, routing, and the /v1 surface.

Why use it¶

It is the only one of these systems that treats NAT and the public internet as the normal case. Nodes on home connections, behind carrier-grade NAT, on cellular, join by scanning a token. That is a genuinely different operating envelope from a rented multi-region cluster with public IPs.
One binary spans CUDA, ROCm, Vulkan, Metal, and CPU. A mesh can mix an H100 box, a Mac Studio, and a Windows workstation. Compare prima.cpp, which also spans heterogeneous devices but as a single-cluster, single-owner system.
Its honest engineering artifacts are unusually good. The repository publishes negative results (an abandoned async-forwarding experiment, an MoE expert-sharding feature its own roadmap calls disappointing), per-family correctness gates, and a benchmark log that shows splitting a model making it slower. That makes it a far better teacher than a vendor benchmark page.

When to use it (and when not)¶

Use it when capacity is genuinely scattered and cannot be consolidated: a volunteer pool, a set of workstations across offices, or GPUs stranded below the model's memory footprint at every individual site. Use it when the alternative is not serving the model at all.

Do not use it to make inference faster. This is the single most important thing to understand about the entire pattern, and Mesh LLM's own numbers say it plainly. Splitting a model across more machines makes decode slower, monotonically, every time. The repository's benchmark page reports GLM-4.7-Flash-Q4_K_M (17 GB) over Wi-Fi, on hardware it describes only as "an M4 Max and a Mac mini M4" (which does not account for the third row, so treat the setup as underspecified):

Configuration	tok/s
Solo, no mesh	68
2-node split (85/15)	21
3-node split (62/31/8)	12 to 13

Adding a second machine cost roughly 69% of throughput; adding a third cost about 82%. Two caveats on these specific figures. They come from docs/BENCHMARKS.md, whose surrounding text attributes its overhead to "per-token RPC latency" and discusses stock llama.cpp RPC transfer volumes, indicating the legacy RPC split rather than the current Skippy stage path; the page never labels the table itself, so read that attribution as inference from context, not as a stated fact. And the project's AGENTS.md states that there is "no external llama-server / rpc-server runtime lane" and forbids reintroducing one, so the RPC path is no longer a supported serving mode; the name nonetheless still appears in the events crate, the TUI, the installers, and CI, so "deleted entirely" would overstate it.

The direction, however, is not in doubt, and it does not rest on these numbers: it follows from the planner's own latency model, reproduced and executed below.

Prefer region-replica routing whenever a full replica fits at every site. Prefer a single bigger box, or a smaller/quantized model, whenever that is available. Splitting is what you do when you have run out of better options.

Do not put confidential prompts on a public mesh. See the trust boundary in Running it in production, below.

Architecture¶

flowchart TB
  subgraph CP["Control plane: iroh QUIC, ALPN mesh-llm/1"]
    G["Gossip 0x01<br/>PeerAnnouncement: VRAM, GPUs,<br/>models, RTT, capabilities"]
    A["Admission: quarantine until a valid<br/>GossipFrame with gen=1 arrives"]
    D["Discovery: Nostr kind 31990<br/>(public) or invite token (private)"]
  end

  subgraph DP["Data plane: ALPN skippy-stage/2"]
    S0["stage 0<br/>layers 0..10<br/>embeddings, driver"]
    S1["stage 1<br/>layers 10..20"]
    S2["stage 2<br/>layers 20..30"]
    S3["stage 3<br/>layers 30..40<br/>logits, readout"]
  end

  C["OpenAI client"] --> S0
  S0 -->|"activation frame<br/>hidden x dtype per token<br/>4 KiB at f16"| S1
  S1 -->|"activation frame"| S2
  S2 -->|"activation frame"| S3
  S3 -->|"predicted token, DIRECT return<br/>(stage protocol gen 3)"| S0
  S0 --> C

  D -.-> A
  A -.-> G
  G -.->|"VRAM and RTT feed<br/>the topology planner"| S0

Two properties of this picture carry the whole design.

The KV cache never crosses a boundary. Each stage holds the KV for its own layer range only, for the sessions routed through it. What crosses is one hidden-state vector per token per boundary: hidden_size x dtype_bytes. For the 40-layer, hidden-2048 model in Mesh LLM's own lab runs, that is 8 KiB per token per boundary at f32 and 4 KiB at f16. This is what makes the pattern tractable at all, and it is the same locality property inference parallelism strategies documents for pipeline parallelism generally.

The return hop is direct. Stage protocol generation 3 sends the predicted token straight from the readout stage back to stage 0, rather than walking it back up the chain. For a four-stage topology that cuts the decode hot path from six serialized hops to four. This is why the planner's cost model multiplies by the stage count, not the stage count minus one.

How to use it¶

Join the public mesh and serve whatever the hardware can hold:

mesh-llm serve --auto

Start a private mesh with a specific model; the command prints an invite token:

mesh-llm serve --model Qwen3-8B-Q4_K_M

Join it from another machine:

mesh-llm serve --join <token>

Run a model too large for any single peer, from a published layer package. Pin an immutable revision for anything you care about:

mesh-llm serve --model hf://meshllm/Qwen3-235B-A22B-UD-Q4_K_XL-layers@<revision> --split

Then talk to it like any OpenAI endpoint:

curl http://localhost:9337/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3-8B-Q4_K_M","messages":[{"role":"user","content":"hello"}]}'

On a multi-homed Linux or Docker host, pin the mesh interface explicitly. iroh will otherwise discover and advertise a Docker bridge address such as 172.17.0.1, and if every host has the same bridge address, peers race the wrong local path:

mesh-llm serve --split --bind-ip 10.1.2.3 --bind-port 47916 --model Qwen3-8B-Q4_K_M

Check that every stage came up before trusting the endpoint. Stage 0 is not routable until all downstream stages report ready:

curl -s http://localhost:3131/api/status | jq '{state:.node_state, stages:.runtime.stages}'
mesh-llm doctor split --model-ref meshllm/Qwen3-8B-Q4_K_M-layers --port 3131

These commands are transcribed from the repository's docs/MESHES.md, docs/SKIPPY_SPLITS.md, and docs/CLI.md at HEAD. They are reference templates: they have not been executed against a live multi-node mesh in this KB, and flags on a project moving this fast should be re-checked against mesh-llm --help before use.

How to develop with it: the stage-count decision is the whole ball game¶

Everything else is detail. The one decision that determines whether a cross-datacentre mesh works or is unusable is how many stages to split into, and Mesh LLM's coordinator answers it with a model simple enough to state in one line. From crates/skippy-coordinator/src/topology.rs:462:

fn estimate_decode_network_ms_per_token(nodes: &[UsableNode]) -> Option<u32> {
    let hop_latency = nodes
        .iter()
        .filter_map(|node| node.stage_transfer_latency_ms)
        .max()?;
    Some(hop_latency.saturating_mul(nodes.len() as u32))
}

That is: estimated_decode_network_ms_per_token = max(hop_latency) * stage_count. The slowest link sets the pace, and every added stage multiplies it. Candidates that meet the target TPOT beat candidates that miss it; within a bucket, lower estimated network TPOT wins ahead of context length, parallel lanes, and spare VRAM. Without latency inputs the planner falls back to a priority order that encodes the same belief: highest valid context, then fewest nodes that can run the model, then most parallel lanes. Extra nodes are never added just to buy lanes.

The model below reproduces the planner's published stage-count table from the formula, derives why the split axis must be pipeline and not tensor parallelism, and then checks the derivation against Mesh LLM's own measured decode spans.

# wan_split_axis.py -- validated. Why pipeline (layer) parallelism survives a WAN link and
# tensor parallelism cannot, derived from first principles and cross-checked against Mesh LLM's
# own published and measured numbers. The load-bearing point is NOT bytes: it is the number of
# SERIALIZED network round trips one decode token must pay. Run: python3 wan_split_axis.py
from __future__ import annotations

LAYERS = 40      # Qwen3.6-35B-A3B as staged in Mesh LLM's four-host lab runs
HIDDEN = 2048    # not assumed: recovered below from two independent published frame sizes


def activation_bytes_per_token(hidden: int, dtype_bytes: int) -> int:
    """Pipeline parallelism: one stage boundary carries one hidden-state vector per token."""
    return hidden * dtype_bytes


def pp_decode_network_ms_per_token(stages: int, hop_ms: float) -> float:
    """The estimator implemented at skippy-coordinator/src/topology.rs:462.
    Stage protocol gen 3 returns the predicted token DIRECTLY to stage 0, so N stages cost
    (N-1) forward hops + 1 return hop = N hops. Hence `nodes.len()`, not `nodes.len() - 1`."""
    return stages * hop_ms


def tp_collectives_per_token(layers: int) -> int:
    """Tensor parallelism: TWO blocking all-reduces per layer per token, one after the
    attention output projection and one after the MLP down projection. No rank may proceed
    until every rank has contributed."""
    return 2 * layers


def tp_decode_network_ms_per_token(layers: int, hop_ms: float) -> float:
    """Each all-reduce costs at least one link latency. This is a generous FLOOR: it charges a
    ring all-reduce a single hop_ms, ignoring that a real ring over N ranks is 2(N-1) sequential
    steps. Even this floor is catastrophic on a WAN."""
    return tp_collectives_per_token(layers) * hop_ms


def tp_allreduce_bytes_per_token(hidden: int, layers: int, dtype_bytes: int, ranks: int) -> int:
    """Ring all-reduce moves 2*(N-1)/N * payload bytes per rank, twice per layer."""
    payload = hidden * dtype_bytes
    return int(2 * layers * 2 * (ranks - 1) / ranks * payload)


if __name__ == "__main__":
    # 1) Recover hidden size from two INDEPENDENT published numbers in docs/skippy/DATA_FLOW.md:
    #    "decode frame 8 KiB/token" (f32), and "max prefill frame ~2.0 MiB" at chunk size 256,
    #    f32. Agreement is strong evidence these are real measurements, not marketing.
    assert (8 * 1024) // 4 == HIDDEN
    assert (2 * 1024 * 1024) // (256 * 4) == HIDDEN

    # Their q8 claim: one boundary payload falls from 4096 B (fp16) to 2052 B, "about 49.9%".
    fp16_frame = activation_bytes_per_token(HIDDEN, 2)
    q8_frame = HIDDEN * 1 + 4                       # one int8 per element, plus a 4-byte scale
    assert (fp16_frame, q8_frame) == (4096, 2052)
    assert abs(100 * (1 - q8_frame / fp16_frame) - 49.9) < 0.05

    # 2) Reproduce the planner's own published stage-count table exactly. TOPOLOGY_PLANNER.md
    #    assumes 10 ms per inter-stage transfer; a 30 tok/s target needs the whole decode loop
    #    under 33.3 ms/token.
    HOP_MS = 10.0
    for stages, floor_ms, best_tps in [(2, 20.0, 50.0), (3, 30.0, 33.3),
                                       (4, 40.0, 25.0), (5, 50.0, 20.0)]:
        got = pp_decode_network_ms_per_token(stages, HOP_MS)
        assert got == floor_ms, (stages, got, floor_ms)
        assert abs(1000.0 / got - best_tps) < 0.05

    # Their conclusion, re-derived rather than quoted: 4 stages cannot reach 30 tok/s even
    # before a single FLOP of local compute is counted. Two stages leave real headroom.
    budget_ms = 1000.0 / 30.0
    assert pp_decode_network_ms_per_token(4, HOP_MS) > budget_ms
    assert pp_decode_network_ms_per_token(2, HOP_MS) < budget_ms

    # Monotonicity: adding a stage NEVER lowers the network floor. This is exactly why the
    # planner ranks "fewest nodes that fit" above extra context and extra parallel lanes.
    floors = [pp_decode_network_ms_per_token(s, HOP_MS) for s in range(1, 9)]
    assert all(b > a for a, b in zip(floors, floors[1:]))

    # 3) The split-axis comparison. Same model, same nodes, same link.
    print(f"Model: {LAYERS} layers, hidden {HIDDEN} (recovered from two independent frames)\n")
    print(f"{'link':>22} {'PP hops':>8} {'PP ms':>8} {'PP tok/s':>9} | "
          f"{'TP collectives':>14} {'TP ms':>8} {'TP tok/s':>9}")
    for label, hop_ms in [("NVLink ~0.005ms", 0.005), ("LAN 0.1ms", 0.1),
                          ("Thunderbolt 0.2ms", 0.2), ("metro DC 2ms", 2.0),
                          ("cross-region 10ms", 10.0), ("intercontinental 80ms", 80.0)]:
        pp_ms = pp_decode_network_ms_per_token(4, hop_ms)
        tp_ms = tp_decode_network_ms_per_token(LAYERS, hop_ms)
        print(f"{label:>22} {4:>8} {pp_ms:>8.2f} {1000/pp_ms:>9.1f} | "
              f"{tp_collectives_per_token(LAYERS):>14} {tp_ms:>8.1f} {1000/tp_ms:>9.2f}")

    # The ratio is fixed by the MODEL's shape, not the network: TP pays 2L serialized
    # collectives where PP pays `stages` hops. Identical multiple at every link latency.
    ratios = {hop: tp_decode_network_ms_per_token(LAYERS, hop)
                   / pp_decode_network_ms_per_token(4, hop)
              for hop in (0.005, 0.1, 2.0, 10.0, 80.0)}
    assert all(abs(r - ratios[0.005]) < 1e-9 for r in ratios.values())
    assert abs(ratios[10.0] - (2 * LAYERS) / 4) < 1e-9
    print(f"\nTP/PP serialized-round-trip ratio = 2L/stages = {ratios[10.0]:.0f}x, IDENTICAL at every")
    print("link latency. It is set by the model's DEPTH, not by the network. A faster link makes")
    print("both faster and does not change which axis you must choose.")

    # Bytes tell a weaker story than round trips. Report both so the point is not overstated.
    pp_bytes = activation_bytes_per_token(HIDDEN, 2)
    tp_bytes = tp_allreduce_bytes_per_token(HIDDEN, LAYERS, 2, ranks=4)
    print(f"\nBytes/token/node (f16): PP boundary {pp_bytes:,} B | TP all-reduce {tp_bytes:,} B "
          f"({tp_bytes // pp_bytes}x)")
    print("Bytes are a 120x problem. Serialized round trips are a 20x problem that MULTIPLIES the")
    print(f"link's latency, and latency is what a WAN has. On a 10ms link TP's 80 blocking")
    print(f"collectives cost {tp_decode_network_ms_per_token(LAYERS, 10.0)/1000:.1f}s per token before any FLOP is issued.")

    # 4) Adversarial check against an INDEPENDENT system's published measurements. Mesh LLM's own
    #    stage spans are NOT used here: the only run that publishes a decode-level breakdown is a
    #    Wi-Fi/LAN run its own repository flags as "diagnostics only" (not GPU-offloaded), so it
    #    cannot carry a load-bearing claim. Petals (arXiv 2312.08361, Table 3, BLOOM-176B, 3
    #    servers, single-batch) publishes exactly the two-way sweep needed instead. The paper
    #    defines this column as generation LATENCY (steps/s), not throughput.
    gbit_5ms, mbit_5ms, mbit_100ms = 1.71, 1.66, 1.23
    ms = lambda steps_per_s: 1000.0 / steps_per_s

    # Isolate each effect while holding the other fixed.
    d_bandwidth = ms(mbit_5ms) - ms(gbit_5ms)      # 10x LESS bandwidth, RTT held at <5 ms
    d_latency = ms(mbit_100ms) - ms(mbit_5ms)      # ~+95 ms RTT, bandwidth held at 100 Mbit/s
    assert d_latency > 10 * d_bandwidth, (d_latency, d_bandwidth)

    # The model says decode_ms = compute + hops * rtt, so d(decode)/d(rtt) recovers the HOP COUNT.
    # If the model is right, a 3-server Petals chain must show a slope of a few serialized hops.
    implied_hops = d_latency / (100.0 - 5.0)
    assert 1.5 < implied_hops < 3.5, implied_hops   # a 3-server chain: 2 to 3 serialized hops

    print(f"\nIndependent check, Petals BLOOM-176B over the internet (3 servers):")
    print(f"  10x LESS bandwidth (RTT fixed): {d_bandwidth:6.1f} ms/step")
    print(f"  ~+95 ms RTT (bandwidth fixed) : {d_latency:6.1f} ms/step  "
          f"({d_latency/d_bandwidth:.0f}x worse)")
    print(f"  => implied serialized round trips per token = {d_latency:.1f}/95 = {implied_hops:.2f}")
    print(f"  A 3-server chain should cost 2 to 3 hops. The measured slope says {implied_hops:.2f}.")
    print("  The hop-count model is confirmed by an INDEPENDENT system's published numbers,")
    print("  and so is the asymmetry: latency binds, bandwidth is slack.")
    print("\nALL ASSERTIONS PASSED")

Executed output:

Model: 40 layers, hidden 2048 (recovered from two independent frames)

                  link  PP hops    PP ms  PP tok/s | TP collectives    TP ms  TP tok/s
       NVLink ~0.005ms        4     0.02   50000.0 |             80      0.4   2500.00
             LAN 0.1ms        4     0.40    2500.0 |             80      8.0    125.00
     Thunderbolt 0.2ms        4     0.80    1250.0 |             80     16.0     62.50
          metro DC 2ms        4     8.00     125.0 |             80    160.0      6.25
     cross-region 10ms        4    40.00      25.0 |             80    800.0      1.25
 intercontinental 80ms        4   320.00       3.1 |             80   6400.0      0.16

TP/PP serialized-round-trip ratio = 2L/stages = 20x, IDENTICAL at every
link latency. It is set by the model's DEPTH, not by the network. A faster link makes
both faster and does not change which axis you must choose.

Bytes/token/node (f16): PP boundary 4,096 B | TP all-reduce 491,520 B (120x)
Bytes are a 120x problem. Serialized round trips are a 20x problem that MULTIPLIES the
link's latency, and latency is what a WAN has. On a 10ms link TP's 80 blocking
collectives cost 0.8s per token before any FLOP is issued.

Independent check, Petals BLOOM-176B over the internet (3 servers):
  10x LESS bandwidth (RTT fixed):   17.6 ms/step
  ~+95 ms RTT (bandwidth fixed) :  210.6 ms/step  (12x worse)
  => implied serialized round trips per token = 210.6/95 = 2.22
  A 3-server chain should cost 2 to 3 hops. The measured slope says 2.22.
  The hop-count model is confirmed by an INDEPENDENT system's published numbers,
  and so is the asymmetry: latency binds, bandwidth is slack.

ALL ASSERTIONS PASSED

Four results fall out, and the fourth is the one that should change how you provision.

First, the model reproduces Mesh LLM's published stage-count table exactly, so the table is arithmetic, not a benchmark, and it transfers to your own hop latency unchanged.

Second, the choice of split axis is not a tuning decision, it is forced by the model's depth. Tensor parallelism needs 2L blocking all-reduces per token where pipeline parallelism needs stages hops. For a 40-layer model split four ways that is a 20x ratio in serialized round trips, and it is the same 20x at every link latency, from NVLink to intercontinental. A faster link does not rescue tensor parallelism across a WAN; it just moves both curves. On a 10 ms cross-region link, TP's 80 collectives cost 0.8 s per token before any arithmetic happens. This is why every system in this lineage, from Petals to Mesh LLM, splits by layer.

Third, bytes are the less interesting axis. TP moves 120x more bytes, but bytes buy bandwidth, and bandwidth is the thing a WAN actually has. Round trips multiply latency, and latency is the thing it does not.

The instructive contrast is exo, which makes the opposite architectural bet: tensor parallelism, over RDMA on Thunderbolt 5. Its RDMA and tensor-parallel machinery is real and visible in its source, not marketing. Its numbers are another matter: the README claims "up to 1.8x speedup on 2 devices and 3.2x speedup on 4 devices," linking an X post that gives a model and a driver (Kimi K2 Thinking, MLX tensor parallel) but no baseline or operation, and separately claims a "99% reduction in latency between devices," linking a different X post that, read directly, does not contain that figure at all; the underlying 300-to-3-microsecond arithmetic behind "99%" sits in a third, uncited post from the same announcement thread. exo does ship a real benchmark harness and a written methodology, but no result artifact in the repository backs any of these three figures, and two of the three citations do not survive being followed. Take the architecture, not the numbers.

The architecture is the point anyway: exo bets on tensor parallelism precisely because Thunderbolt-5 RDMA puts it in the sub-millisecond regime where paying 2L collectives per token is affordable, and it targets a room, not a WAN. Move that same design onto a 10 ms link and the table above prices it at 1.25 tok/s. The two projects are not disagreeing; they are optimizing for opposite ends of the latency axis, and each is coherent for its own end. The decision rule is therefore mechanical rather than ideological: tensor-parallel inside a fast fabric, pipeline-parallel across it, with the boundary where 2L * hop_ms stops fitting the per-token budget.

The canonical statement of this cost asymmetry is not new, and it is worth citing rather than rediscovering. Megatron-LM's scaling paper (arXiv 2104.04473, section 3.2) puts it exactly: with pipeline parallelism the communication "between every pair of consecutive devices (for either the forward or backward pass) for each microbatch is bsh", while with tensor parallelism tensors of size bsh "need to be all-reduced among t model replicas twice each in the forward and backward pass for each layer, leading to a total communication of 8bsh(t-1)/t per layer per device for each microbatch." Note the per-pass qualifier: bsh is what crosses a boundary on one pass, not per microbatch overall. The derivation above is the decode-time, forward-only specialization of that result, and it agrees with it. (Megatron's own section 4.1 then reduces the pipeline figure to bsh/t with a scatter-gather optimization across its InfiniBand cards, which is a fast-fabric trick and does not change the ratio that matters here.)

Fourth, the model survives contact with an independent system's published numbers. Petals is the natural check because it publishes the one sweep that separates the two variables: holding RTT fixed and cutting bandwidth tenfold costs it 17.6 ms per decoding step, while holding bandwidth fixed and adding about 95 ms of RTT costs it 210.6 ms. Latency is twelve times more expensive than a tenfold bandwidth cut. Better still, the slope is diagnostic: if decode time is compute + hops * rtt, then the derivative with respect to RTT recovers the hop count, and Petals' measured slope implies 2.22 serialized round trips per token for a three-server chain. That is exactly the two-to-three hops the model predicts. An independent system, a different codebase, a different model, and the hop-count model still falls out of its data.

A note on what this page does not use. Mesh LLM's repository does publish a decode-level stage breakdown, and an earlier draft of this page built its headline claim on it. That was a mistake, and it is worth naming: those spans come from a Wi-Fi/LAN run over .local hostnames that the repository itself flags as diagnostic (it is not GPU-offloaded, and the project notes that hostname-based runs "were observed to choose mixed routes"). Its own Thunderbolt runs, on 10.0.0.x, are roughly three times faster on the same corpus. A number from a run the project tells you not to use is not evidence, however convenient its shape. The claim stands on the derivation, on Petals, and on Mesh LLM's own unambiguous statement in its protocol notes: "decode is usually RTT-bound, while prefill is more sensitive to activation bytes."

That is the whole conclusion. Decode is latency-bound, so a cross-datacentre pipeline is not a tuning problem but a different regime, and speculative decoding is the only lever that touches the binding term, because it is the only one that commits more than one token per round trip. Mesh LLM ships it ([defaults.speculative], plus native MTP defaulting on).

Activation wire dtype is a correctness gate, not a compression knob¶

The obvious optimization is to shrink the activation frame. Mesh LLM's own certification data says: not globally, you cannot.

The usual default is f16, which halves the payload against f32. q8 halves it again (4096 B to 2052 B for a hidden-2048 model, a 49.9% cut) but changes the next token for some families and not others. This is not a knob you can set globally, and Mesh LLM's answer is a per-family certification record rather than a policy.

Read from crates/skippy-topology/capabilities/reviewed-family-capabilities.json at HEAD, the reviewed set holds 96 records across 95 model families, and the verdicts split almost evenly:

`q8_wire_validation`	Records
`validated`	48
`rejected` (changes the next token)	40
`untested`	8

Even f16 is not universal: 94 of the 96 records default to it, but gemma and apertus default to f32, so the conservative payload halving is itself withheld for two families. About 42% of reviewed records cannot take q8 without changing their output.

The two traps in that file are worth internalizing, because they are exactly the kind of thing a careless lookup gets wrong:

deepseek is rejected; deepseek2 is validated.
gemma3 is rejected; gemma3n is validated.

Adjacent names, opposite verdicts. The planner rejects an unsafe combination before launch, not at the first divergent token. Treat activation compression the way you would treat a quantization change: as something requiring per-family correctness evidence, looked up exactly, never inferred from a family's neighbours. Mesh LLM's own experiment log also found q8 was not reliably faster (one smoke 23.8% faster, a second 20.0% slower, the sweep abandoned as too noisy), so for many families the compression is not even buying what it costs.

Recurrent and hybrid models make the topology sticky¶

For a dense transformer, any stage can serve any sequence: the only per-sequence state is the KV cache, which stays local to the layers that own it. For recurrent and hybrid models (Falcon-H1, Qwen3Next), the layer range owns a recurrent state that is far too large to ship per token (Mesh LLM measures roughly 76 to 79 MB). The rule the planner enforces:

activation crosses topology boundaries
recurrent state defines topology affinity

A stage owning recurrent layers becomes a sticky owner: every future token for that sequence must route back to it. That is a routing constraint, not merely a placement one, and it interacts badly with node churn, since a sticky owner leaving kills its sequences rather than degrading them. No page in this KB previously covered this: it is a class of constraint that only appears once state-space and hybrid models enter a distributed serving pool.

How to maintain it¶

Readiness is explicit and ordered. Downstream stages load first; stage 0 publishes its route only after every required stage reports ready. A loaded runtime is not a routable one. Do not infer health from a running process.
A stage failure replans the whole topology. Mesh withdraws the routable target, quarantines the failed peer, and rebuilds with a fresh run id from the remaining eligible peers. In-flight requests fail; there is no resume. This is a deliberate choice (stage state is spread across per-stage KV, sequence ids, and transport links) and it is the honest one, but it means a churny pool converts node failures directly into user-visible request failures. Compare the finer-grained healing in cross-WAN, which relaunches only a spare and the dead stage's predecessor and then re-prefills.
Watch the planner's own estimate. estimated_decode_network_ms_per_token and decode_tpot_target_met are exposed in split planning; a topology that misses its TPOT target is telling you it will not hit your latency SLO before you serve a single token.
Pin immutable package revisions. hf://namespace/repo@revision, not a moving branch. Only immutable refs are eligible for peer artifact transfer at all.

Running it in production: the trust boundary is the real limit¶

Mesh LLM is unusually careful about what its security mechanisms do and do not prove, and the honest summary is that a public mesh gives you no protection against a peer that lies about its output.

Release attestation is build provenance, not runtime attestation. The repository states this outright: a signed attestation proves a peer's binary was published by a trusted release signer. "It does not prove the remote process is actually running unmodified code, nor that the host OS or hardware has not been tampered with." An invalid binary still follows the normal startup path unless mesh policy requires certified builds.
Node reputation is local-only and is not a trust system. routing_affinity.target_reputation counters are process-local routing health, used to avoid peers that recently timed out. They are explicitly not gossiped, not persisted, and not proof of model honesty. The repository's own future-work section lists Sybil resistance and spoofing as unanswered.
There is no output verification of any kind. No signed activation receipts, no recompute challenge, no proof of inference. The mechanisms that cross-WAN model-parallel inference documents (attribution/coverage receipts, and a separate trusted-recompute challenge) have no counterpart here. A peer serving a smaller or different model, or returning plausible garbage, is not detected.
Confidentiality is impossible by construction. A stage must see the activations it computes. On a public mesh, a peer holding layers 10 to 20 sees the hidden states of every token routed through it.

The defensible production posture, then:

Deployment	Posture
Public mesh (`--publish`, `--auto`)	Non-sensitive prompts only. Treat as a capability demo or a volunteer/community service, not a business system.
Private mesh, machines you own	The realistic production case. Invite tokens, `--trust-policy allowlist`, owner keys, and `--mesh-discovery-mode mdns` to keep discovery off public relays.
Cross-datacentre, machines you own	Viable. Use the TPOT planner honestly: at a 10 ms inter-DC hop, two stages is your budget, three is marginal, four cannot hit 30 tok/s.

Peer artifact transfer (fetching missing layer-package fragments from the coordinating peer) is disabled by default on public meshes for good reason; enable it only inside a trust boundary via MESH_LLM_ARTIFACT_TRANSFER=trusted.

Failure modes¶

Expecting a speedup from splitting. Every added stage multiplies the decode network floor. The repository's own figures show a 2-node split at roughly 31% of solo throughput and a 3-node split at about 18%. Split to fit a model, never to accelerate one.
Reading docs/BENCHMARKS.md as current. Those figures describe the legacy llama.cpp RPC split, which no longer exists in the serving runtime. The direction holds; the numbers do not describe Skippy.
Reading "total tok/s" as decode throughput. Mesh LLM's benchmark tables report total tok/s including prompt tokens. In run lab-prefill-balanced-mixed192-kv-chunk128-credit-20260426-071734, "30.33 total tok/s" sits in the same row as 0.99 generated tok/s (1536 generated tokens over 1545 s), because that corpus generates only 8 tokens per request. The metric is legitimate for the prefill work it was built for, and badly misleading if compared against a vLLM decode number. Note also that this is a different run from the clean-replay one whose stage spans are often quoted; the repository flags this one as carrying 45 dropped telemetry events.
Reading the "zero inter-device communication during decode" line literally. docs/EXO_COMPARISON.md ends a paragraph with that clause, and taken alone it is simply false: every decode token crosses every stage boundary. In fairness to the source, the sentence immediately before it says the opposite ("each device decodes independently once it receives its activation frame from the prior stage"), so the paragraph contradicts its own last clause rather than the project misunderstanding its own design. What contiguous-layer splitting avoids is an all-reduce inside a layer, not communication as such. Quote the clause at your peril.
Letting iroh pick the wrong interface. On Docker or multi-NIC Linux hosts it will advertise bridge addresses like 172.17.0.1; if every host shares that address, peers race the wrong local path. Pin --bind-ip.
Splitting a recurrent or hybrid model without planning for sticky owners. Falcon-H1 and Qwen3Next stages own unmovable recurrent state; the owner's departure kills its sequences.
Putting a sensitive prompt on a public mesh. Every stage sees every activation it computes, and nothing verifies that any peer ran the model it claimed to.

Open questions and validation¶

None of the throughput numbers on this page were reproduced in this KB on live multi-node hardware. They are the repository's own measurements, checked for internal arithmetic consistency (which they pass, exactly) rather than independently re-run. The derived latency model, by contrast, is executed above and depends on no vendor number.
Mesh LLM ships speculative decoding and native MTP but publishes no staged, cross-WAN speculative benchmark. Given that decode is RTT-bound, this is the single measurement that would most change the picture. Note that speculative decoding pays at essentially any acceptance rate here, since it amortizes the round trip: the cross-WAN page's executed model puts a threshold near 0.78 only on the further question of whether keeping several speculative chunks in flight beats resolving them one at a time, which is a different and narrower claim.
The only WAN datapoint in the repository (Sydney to Queensland, about 20 ms RTT, 10 to 25 tok/s) is from the deleted RPC path. There is no published Skippy measurement across a real WAN, which is precisely the regime this KB cares about most.
Whether the q8 activation exactness failures are a scalar-codec artifact (the repo suspects this) or a genuine sensitivity of those families is unresolved, and it matters: a vectorized codec that preserved exactness would halve WAN activation payload for every family.

References¶

Mesh LLM (Apache-2.0), verified against HEAD c14e458, 2026-07-12: https://github.com/Mesh-LLM/mesh-llm
Topology planner, latency-aware stage count and family capability records: https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/skippy/TOPOLOGY_PLANNER.md
Stage data flow and activation frame sizes: https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/skippy/DATA_FLOW.md
Skippy stage-split operator guide: https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/SKIPPY_SPLITS.md
Mesh workflows, discovery, invite tokens, trust policy and the release-attestation boundary: https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/MESHES.md
Node message protocol, gossip frames, admission and stream types: https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/design/message_protocol.md
Local node reputation, and its explicit non-goals: https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/NODE_REP.md
Experiment log, four-host stage spans, chunk and wire-dtype sweeps, abandoned async forwarding: https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/skippy/EXPERIMENTS.md
Layer package manifest and artifact spec: https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/specs/layer-package-repos.md
iroh (QUIC transport, hole-punching, relay fallback, dial by public key), pinned at 1.0.1 in the workspace Cargo.lock: https://github.com/n0-computer/iroh
Narayanan et al., Megatron-LM (arXiv 2104.04473, SC'21); section 3.2 for the pipeline-versus-tensor communication volumes quoted above: https://arxiv.org/abs/2104.04473
Borzunov et al., Distributed Inference and Fine-tuning of LLMs Over The Internet (arXiv 2312.08361, NeurIPS 2023), Table 3, the source of the Petals bandwidth-versus-latency cross-check executed above: https://arxiv.org/abs/2312.08361
Nostr NIPs. Mesh LLM publishes a replaceable event of custom kind 31990 with an expiration tag (crates/mesh-client/src/network/nostr.rs). That number is the NIP-89 application-handler kind, but the repository never cites NIP-89 and its payload is not NIP-89 conformant, so treat it as kind-number reuse rather than protocol adherence: https://github.com/nostr-protocol/nips