Copy for agents Markdown

P2P transport for decentralized inference

Scope: the data-plane substrate that carries activations between inference stages on machines in different datacentres, on different networks, and behind NAT. Covers why QUIC rather than TCP, NAT traversal and relay fallback, peer discovery (DHT, Nostr, mDNS, invite tokens), gossip membership and admission control, and the arithmetic that decides whether a link can carry a decode loop at all. Every system in cross-WAN model-parallel inference needs this layer and none of the papers specify it; this page is the layer itself. For the WireGuard overlay that carries a training rendezvous and low-frequency sync, see overlay and mesh networking, which is the control-plane substrate and says explicitly not to put a decode hot path on it.

What it is

Between "the scheduler assigned layers 10 to 20 to node B" and "node B computed them" sits an unglamorous layer that decides whether the whole system works: find node B, connect to it through two NATs neither of you control, prove it is really node B, carry activation frames and control messages and multi-gigabyte weight transfers over the same connection without any of them blocking the others, and notice within seconds when it dies.

The scheduling literature treats this as solved. It is not solved; it is assumed. Petals builds on hivemind, whose Kademlia DHT rides on go-libp2p and which traverses NAT with libp2p Circuit Relay. Parallax uses a separate project (Lattica). Mesh LLM ships QUIC via iroh in the same binary as the inference engine. exo, targeting a room rather than a WAN, has moved off libp2p entirely: it pins zenoh and discovers peers with a bespoke IPv6 UDP multicast beacon. These are four different answers to the same unspecified question, and the choice has direct, measurable consequences for the latency budget the scheduler is optimizing against.

That exo detail is a warning in itself. Mesh LLM's own comparison document still describes exo as using "libp2p, mDNS, bootstrap peers"; exo's current tree contains no libp2p crate in any Cargo.toml, and its swarm.rs opens with the comment "Compat shim for the old libp2p code." Transport stacks in this space turn over fast enough that a year-old secondary description is likely to be wrong. Check the Cargo.toml, not the comparison table.

The honest baseline for what happens when you skip this layer is llama.cpp's own RPC backend, which is the naive design (expose a port that accepts tensor operations) and whose README says so plainly:

This example and the RPC backend are currently in a proof-of-concept development stage. As such, the functionality is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!

That is the correct warning, and it is why every serious decentralized-inference system builds a real transport instead. Mesh LLM's own contributor guide now states there is "no external llama-server / rpc-server runtime lane" and forbids reintroducing one, though the name still lingers in its events crate, TUI, installers, and CI.

Why the transport is the product

Because the transport's round-trip time is the scheduler's cost function. Mesh LLM's coordinator prices a candidate topology as estimated_decode_network_ms_per_token = max(hop_latency_ms) * stage_count and rejects topologies that miss the target. The transport does not merely carry the workload; it sets the budget the workload is planned against. A 2 ms improvement in hop latency is worth more than any amount of kernel tuning, because it multiplies by the stage count on every single token.

And the binding constraint is latency, not bandwidth. That claim is worth deriving rather than asserting, because it is counterintuitive and it inverts the usual instinct to buy a fatter pipe.

Core knowledge: the link is idle, and that is the problem

SWARM Parallelism (arXiv 2301.11913) reports finding "configurations where training larger models becomes less communication-intensive," which sounds like good news for anyone splitting a model over a slow link. It is good news, and it is good news about exactly one of the two terms that matter. The model below derives the effect from the transformer's shape, then draws the line.

# wan_bandwidth_vs_latency.py -- validated. Bigger models are EASIER to distribute over a slow
# link per byte moved (the SWARM effect, derived here from the transformer's shape rather than
# quoted), and this helps decode exactly zero. Then: what actually does.
# Run: python3 wan_bandwidth_vs_latency.py
from __future__ import annotations

DTYPE = 2  # f16 activation wire

# (name, total_params, hidden_size, layers) -- published dense-model shapes.
MODELS = [
    ("Llama-3 8B",    8.0e9,  4096,  32),
    ("Llama-3 70B",  70.0e9,  8192,  80),
    ("Llama-3 405B", 405.0e9, 16384, 126),
]


def boundary_bytes_per_token(hidden: int, dtype: int = DTYPE) -> int:
    """A pipeline boundary carries one hidden-state vector per token. Scales as O(h)."""
    return hidden * dtype


def stage_flops_per_token(total_params: float, stages: int) -> float:
    """Forward-pass compute for the params one stage holds: the standard 2*N FLOPs per token.
    Scales as O(h^2) for a transformer, whose params per layer are ~12h^2."""
    return 2.0 * (total_params / stages)


def arithmetic_intensity(total_params: float, hidden: int, stages: int) -> float:
    """FLOPs of local compute earned per byte pushed across a stage boundary.
    = (2P/S) / (h * dtype). Numerator is O(h^2), denominator is O(h): this GROWS with h."""
    return stage_flops_per_token(total_params, stages) / boundary_bytes_per_token(hidden)


def decode_hops_per_token(stages: int) -> int:
    """Serialized network round trips one decode token must pay. Independent of model size."""
    return stages


def specdec_hops_per_token(stages: int, tokens_committed: float) -> float:
    """Speculative decoding commits several tokens per verification round trip, so the hop cost
    is AMORTIZED across them. This is the only lever that touches the latency term."""
    return stages / tokens_committed


if __name__ == "__main__":
    STAGES = 4

    print(f"Pipeline boundary economics, {STAGES} stages, f16 activation wire\n")
    print(f"{'model':>14} {'hidden':>7} {'boundary B/token':>17} {'stage FLOPs/token':>19} {'FLOPs per byte':>16}")
    intensities = []
    for name, params, hidden, layers in MODELS:
        ai = arithmetic_intensity(params, hidden, STAGES)
        intensities.append(ai)
        print(f"{name:>14} {hidden:>7,} {boundary_bytes_per_token(hidden):>17,} "
              f"{stage_flops_per_token(params, STAGES):>19,.3g} {ai:>16,.3g}")

    # 1) The SWARM effect, derived: boundary arithmetic intensity GROWS with model size.
    assert intensities[0] < intensities[1] < intensities[2], intensities
    print(f"\nLlama-3 405B does {intensities[2]/intensities[0]:.1f}x more compute per boundary byte than 8B.")
    print("Compute per stage is O(h^2); the boundary payload is O(h). The ratio grows as O(h).")

    # 2) Adversarial: is that the h^2-over-h scaling, or an artifact of these three real shapes?
    #    Hold architecture fixed, vary ONLY hidden size, using params = 12*h^2*L.
    L = 64
    synth = {h: arithmetic_intensity(12 * h * h * L, h, STAGES) for h in (1024, 2048, 4096, 8192)}
    for h_small, h_big in ((1024, 2048), (2048, 4096), (4096, 8192)):
        assert abs(synth[h_big] / synth[h_small] - 2.0) < 1e-9   # exactly linear in h
    print(f"Controlled check (params = 12*h^2*L, L={L} fixed): doubling hidden size doubles")
    print("FLOPs-per-boundary-byte EXACTLY, at every size. The O(h) scaling is structural.")

    # 3) THE LINE. That was a BANDWIDTH result. The latency term is the serialized hop count,
    #    and it does not depend on model size at all.
    for name, params, hidden, layers in MODELS:
        assert decode_hops_per_token(STAGES) == STAGES          # identical for 8B and 405B
    print(f"\nDecode hops per token = {decode_hops_per_token(STAGES)} for EVERY model above.")
    print("Scaling the model up buys nothing on the latency term. Not a little: exactly nothing.")

    # 4) Which term binds? Compare a decode step (1 token) with a prefill chunk (many tokens)
    #    on a 10 ms cross-region hop over a 1 Gbit/s link.
    HOP_MS, LINK_GBPS = 10.0, 1.0
    bytes_per_ms = LINK_GBPS * 1e9 / 8 / 1000
    hidden = 8192                                               # Llama-3 70B
    print(f"\nOn a {HOP_MS:.0f} ms hop over a {LINK_GBPS:.0f} Gbit/s link:")
    print(f"{'workload':>22} {'bytes/hop':>12} {'transmit ms':>12} {'latency ms':>11} {'bound by':>12}")
    for label, tokens in [("decode (1 token)", 1), ("prefill chunk (256)", 256),
                          ("prefill chunk (2048)", 2048)]:
        payload = boundary_bytes_per_token(hidden) * tokens
        transmit_ms = payload / bytes_per_ms
        bound = "LATENCY" if transmit_ms < HOP_MS else "bandwidth"
        print(f"{label:>22} {payload:>12,} {transmit_ms:>12.2f} {HOP_MS:>11.1f} {bound:>12}")

    decode_transmit = boundary_bytes_per_token(hidden) / bytes_per_ms
    assert decode_transmit < HOP_MS / 50, decode_transmit       # decode payload is latency-trivial
    print(f"\nA decode token's 16 KiB transmits in {decode_transmit:.3f} ms, then waits {HOP_MS:.0f} ms for the")
    print("round trip. The link is ~99% idle during decode. Buying bandwidth changes NOTHING.")
    print("The pipe is not full. It is empty, and waiting.")

    # 5) So what fixes decode? Only committing more tokens per round trip. With per-token
    #    acceptance alpha and K draft tokens, speculative decoding commits, in expectation,
    #    E[tokens] = (1 - alpha^(K+1)) / (1 - alpha) per verification round trip.
    def expected_tokens(alpha: float, K: int) -> float:
        return (K + 1.0) if alpha >= 1.0 else (1.0 - alpha ** (K + 1)) / (1.0 - alpha)

    print(f"\nSpeculative decoding, {STAGES} stages, {HOP_MS:.0f} ms hop (network floor only):")
    print(f"{'alpha':>7} {'K':>3} {'E[tokens]/round':>16} {'hops/token':>11} {'network tok/s':>14}")
    base_tps = 1000.0 / (decode_hops_per_token(STAGES) * HOP_MS)
    for alpha, K in [(0.0, 0), (0.6, 4), (0.74, 4), (0.8, 8), (0.9, 8)]:
        e = expected_tokens(alpha, K)
        hops = specdec_hops_per_token(STAGES, e)
        print(f"{alpha:>7.2f} {K:>3} {e:>16.2f} {hops:>11.2f} {1000.0/(hops*HOP_MS):>14.1f}")

    assert abs(expected_tokens(0.0, 8) - 1.0) < 1e-9            # hopeless drafter: 1 token/round
    assert abs(expected_tokens(1.0, 8) - 9.0) < 1e-9            # perfect drafter: K+1 tokens/round
    good = 1000.0 / (specdec_hops_per_token(STAGES, expected_tokens(0.9, 8)) * HOP_MS)
    assert good > 4 * base_tps, (good, base_tps)
    print(f"\nWithout speculation the network floor is {base_tps:.1f} tok/s. At alpha=0.9, K=8 it is")
    print(f"{good:.1f} tok/s, a {good/base_tps:.1f}x gain bought purely by committing more tokens per round")
    print("trip. Speculative decoding is not an optimization here. It is the only lever that")
    print("touches the term that actually binds.")

    print("\nALL ASSERTIONS PASSED")

Executed output:

Pipeline boundary economics, 4 stages, f16 activation wire

         model  hidden  boundary B/token   stage FLOPs/token   FLOPs per byte
    Llama-3 8B   4,096             8,192               4e+09         4.88e+05
   Llama-3 70B   8,192            16,384             3.5e+10         2.14e+06
  Llama-3 405B  16,384            32,768            2.02e+11         6.18e+06

Llama-3 405B does 12.7x more compute per boundary byte than 8B.
Compute per stage is O(h^2); the boundary payload is O(h). The ratio grows as O(h).
Controlled check (params = 12*h^2*L, L=64 fixed): doubling hidden size doubles
FLOPs-per-boundary-byte EXACTLY, at every size. The O(h) scaling is structural.

Decode hops per token = 4 for EVERY model above.
Scaling the model up buys nothing on the latency term. Not a little: exactly nothing.

On a 10 ms hop over a 1 Gbit/s link:
              workload    bytes/hop  transmit ms  latency ms     bound by
      decode (1 token)       16,384         0.13        10.0      LATENCY
   prefill chunk (256)    4,194,304        33.55        10.0    bandwidth
  prefill chunk (2048)   33,554,432       268.44        10.0    bandwidth

A decode token's 16 KiB transmits in 0.131 ms, then waits 10 ms for the
round trip. The link is ~99% idle during decode. Buying bandwidth changes NOTHING.
The pipe is not full. It is empty, and waiting.

Speculative decoding, 4 stages, 10 ms hop (network floor only):
  alpha   K  E[tokens]/round  hops/token  network tok/s
   0.00   0             1.00        4.00           25.0
   0.60   4             2.31        1.73           57.6
   0.74   4             2.99        1.34           74.8
   0.80   8             4.33        0.92          108.2
   0.90   8             6.13        0.65          153.1

Without speculation the network floor is 25.0 tok/s. At alpha=0.9, K=8 it is
153.1 tok/s, a 6.1x gain bought purely by committing more tokens per round
trip. Speculative decoding is not an optimization here. It is the only lever that
touches the term that actually binds.

ALL ASSERTIONS PASSED

The whole page follows from this run.

Prefill and decode are different workloads on the same wire. Prefill pushes a chunk of hundreds or thousands of tokens and is bandwidth-bound: at 256 tokens it needs 33 ms of transmit time against a 10 ms hop. Decode pushes one 16 KiB vector, transmits it in 0.13 ms, and then waits 10 ms. The link is essentially idle during decode. Provisioning a fatter pipe to fix decode latency is the single most common and most expensive mistake in this pattern. Petals measured exactly this asymmetry on a real internet deployment, and Mesh LLM's own protocol notes concede the same thing in one line: "decode is usually RTT-bound, while prefill is more sensitive to activation bytes."

The SWARM effect is real and it is a bandwidth effect. A 405B model earns 12.7x more local compute per boundary byte than an 8B one, and the controlled check shows the scaling is exactly linear in hidden size, structural to the transformer rather than incidental to anyone's cluster. Bigger models genuinely are easier to spread over a slow link. This makes prefill scale gracefully and does nothing whatsoever for decode, whose hop count is stages regardless of whether the model is 8B or 405B.

SWARM states the general form of this directly in the section it titles "The Square-Cube Law of Distributed Training": compute for naive matrix multiplication scales as O(n^3) while the communication phase "requires at most O(n^2) time", so "as we increase the model size, the computation time grows faster than communication time." That is a simplified model of stacked n x n matmuls rather than a transformer-specific result, which is why the derivation above re-does it in the transformer's own terms. Its measurements are the empirical version, and the best published evidence that model size buys WAN tolerance (relative GPU utilization, 500 Mbit/s link, batch 1):

RTT	base (h=768)	xxlarge (h=4096)	GPT-3 (h=12288)
none	18.0%	32.1%	82.1%
10 ms	11.8%	28.9%	79.3%
100 ms	2.78%	14.9%	60.2%
200 ms	1.53%	10.1%	48.5%

At 100 ms, which SWARM notes is "roughly equivalent to training on different continents," a GPT-3-shaped model still holds 60.2% utilization while a small one collapses to 2.78%. Resist the temptation to read that as a 21.7x latency-tolerance gap: most of it exists already at zero latency (82.1% versus 18.0%, a 4.56x gap in baseline efficiency). The tolerance claim has to be about what each model retains under load, and there the honest figures are that the small model keeps 15.4% of its zero-latency utilization while GPT-3 keeps 73.3%, a 4.75x difference. SWARM computes no such ratio itself; this is the KB's arithmetic on its table. The direction is the durable point: a 400B model is a far better candidate for cross-datacentre splitting than a 7B one. The small model is the one you should keep in a single box.

Petals reports the same asymmetry from the serving side, and it is the cleanest independent confirmation available. Its NeurIPS evaluation states it in one sentence: "For inference, performance does not depend much on bandwidth or sequence length but degrades with higher latency. In turn, fine-tuning forward passes for large batches are affected by both bandwidth and latency." That is exactly the prefill/decode split derived above, found independently and from the opposite direction: the single-token path is latency-bound, and the large-batch forward path (which is what prefill is) is the one that also feels bandwidth. Concretely, on Petals' BLOOM-176B three-server rows, cutting the link from 1 Gbit/s to 100 Mbit/s cost it almost nothing (1.71 to 1.66 decoding steps/s, the paper's own generation-latency measure, not a throughput figure), while going from under 5 ms to 100 ms RTT on that same 100 Mbit/s link cost a quarter of it (1.66 to 1.23). Two independent systems, one training and one serving, measured the same thing: latency binds, bandwidth is slack.

Speculative decoding is therefore not one optimization among several; it is the only one that touches the binding term. Everything else (compression, bandwidth, faster kernels) attacks terms that are already slack. This is why Mesh LLM ships speculative decoding and native MTP defaulting on, and why its own protocol notes concede "decode is usually RTT-bound, while prefill is more sensitive to activation bytes."

Note what the table above does and does not say about acceptance rates. Speculative decoding pays here at essentially any acceptance rate, because even a mediocre drafter commits more than one token per round trip: at α = 0.60 the executed run already shows 57.6 tok/s against a 25.0 tok/s floor, a 2.3x gain. The roughly-0.78 threshold in the cross-WAN page's pipelining model answers a different and narrower question: whether keeping several speculative chunks in flight at once beats resolving them one at a time. Do not read it as a floor below which speculation stops helping. It is not.

Core knowledge: why QUIC and not TCP

flowchart TB
  subgraph One["One QUIC connection, independent streams"]
    S1["activation frames<br/>latency-critical, tiny"]
    S2["gossip and control<br/>small, periodic"]
    S3["layer-package artifacts<br/>multi-GB bulk transfer"]
  end
  Note["TCP: a lost packet in the 4 GB weight download<br/>head-of-line blocks the decode activation<br/>frame queued behind it.<br/>QUIC: streams are independent."]
  One --> Note

A decentralized inference node multiplexes three traffic classes with violently different requirements over one link to each peer: latency-critical activation frames, periodic control and gossip, and bulk weight transfer. On a single TCP connection these share one ordered byte stream, so a single dropped packet in a multi-gigabyte weight download stalls the decode activation frame queued behind it, because TCP must deliver bytes in order. This is head-of-line blocking, and on a decode loop that is already 90% wait, it is fatal. iroh's README names the property directly: QUIC gives "concurrent streams with stream priorities, a datagram transport and avoid head-of-line-blocking out of the box."

Four QUIC properties earn their place here, all specified in RFC 9000 and RFC 9001:

• Independent streams. A loss on one stream does not stall the others. This is the whole reason to be here.
• Fewer handshake round trips. TCP plus TLS 1.3 costs two round trips before the first byte of payload; QUIC folds transport and crypto into one. On a reconnect, TLS 1.3 over TCP can do 0-RTT but still pays the TCP handshake, where QUIC pays nothing. Either way the saving is one round trip, so about 10 ms on a 10 ms link, and a churny volunteer pool reconnects constantly.
• Connection migration. A QUIC connection is keyed by a connection ID, not by the four-tuple, so a laptop moving from Wi-Fi to cellular keeps its connection. A TCP connection dies. For a mesh whose nodes are consumer machines, this is the difference between a dropped request and an uninterrupted one.
• Always-authenticated encryption. There is no unencrypted QUIC. Contrast llama.cpp's RPC port, which has neither authentication nor encryption and says so.

QUIC is not on this list because it helps NAT traversal, and it is worth being blunt about that, because the folklore says otherwise. The largest published measurement of decentralized hole punching (arXiv 2510.27500, on libp2p's DCUtR in production IPFS) sets out to refute exactly that folklore, and reports "statistically indistinguishable success rates for both TCP and QUIC (about 70%)." Two caveats keep this honest: the paper offers no test statistic behind "statistically indistinguishable" beyond observing that both curves sit near 70%, and its headline scale (4.4M attempts, 85,000 networks, 167 countries) describes the remote peers punched toward, not the measurement fleet, which ran from 859 networks in 39 countries. The direction is nonetheless well-evidenced and matches the mechanism: being UDP-based buys you nothing at the NAT. Choose QUIC for head-of-line blocking, handshake cost, and connection migration; do not choose it expecting a better hole-punch rate.

Mesh LLM layers a protocol on top of this with two ideas worth stealing. ALPN is the version gate: control traffic negotiates mesh-llm/1 and latency-sensitive activation traffic negotiates skippy-stage/2, so an incompatible peer fails at connection setup rather than at the first malformed frame. And one connection carries many logical streams, distinguished by a one-byte prefix, with a hard 8 MiB frame cap so a malicious peer cannot exhaust memory by declaring an enormous frame.

Core knowledge: NAT traversal and the relay fallback

Two consumer machines behind separate NATs cannot simply connect. The escalation ladder every P2P system implements:

1. Direct connection if a routable path exists (same LAN, public IP, or an already-open mapping).
2. Hole punching otherwise: both peers, coordinated by a third party they can each reach, simultaneously send packets outbound, so each NAT creates a mapping that lets the other's packets in.
3. Relay when hole punching fails: a public server forwards the traffic.

iroh describes exactly this ladder: "The fastest route is a direct connection, so if necessary, iroh tries to hole-punch. Should this fail, it can fall back to an open ecosystem of public relay servers."

A large minority of your peers will end up on a relay, and the number is worse than the headline. The DCUtR study puts hole-punch success at "70% ± 7.1%" (the paper gives no definition for that interval, so read it as dispersion around a fitted line rather than a confidence bound), with 97.6% of the successes landing on the first attempt. Hole punching generally fails when either side sits behind a symmetric NAT, which assigns a different external port per destination, so the port the coordinator observed is not the port the peer will actually use.

Two things stop that 70% from being the number you can plan against, and the paper is candid about both. It is conditional: the study excludes roughly 29% of its data points, mostly cases where the earlier relay-reservation or public-address steps failed outright, so 70% "reflect[s] the performance of the DCUtR hole-punching mechanism itself, conditional on the success of these earlier stages, rather than the end-to-end success rate." And the authors twice warn it is likely optimistic, because their client sampling and honeypot-based peer discovery both pre-filter for reachability. Compounding the conditional gives a realistic end-to-end direct-connection rate nearer 50 to 55%. Plan for something close to a coin flip, not a 30% tail.

That matters more here than in a file-sharing network, because of how the planner prices a topology. A relayed connection routes through a third location, and that latency lands directly in max(hop_latency_ms) * stage_count. A single relayed peer sets the pace for the entire pipeline, because the cost model takes the maximum hop latency, not the mean. A mesh is only as fast as its worst-connected stage. Even taking the optimistic 70% at face value and assuming links fail independently (a simplification, since peers behind the same NAT type are correlated), a four-stage chain has four distinct peer-to-peer links and therefore about a 76% chance of containing at least one relayed hop. A two-stage split has only one link, since the forward and return legs are the same peer pair on the same QUIC connection, so its exposure is the per-link rate itself: about 30% optimistically, and closer to half on the end-to-end figure. Relay is not the edge case, and that is why decentralized meshes underperform their hardware.

The transport choice this pushes people toward is convergent. Prime Intellect's prime-iroh ("asynchronous P2P communication backend for decentralized pipeline parallelism") and its prime-vllm ("modded vLLM to run pipeline parallelism over public networks") independently landed on iroh, the same QUIC stack Mesh LLM uses, for the same reasons: direct connections where possible, hole punching, relay fallback. Two unrelated teams building WAN pipeline-parallel inference converged on the same transport, which is a meaningful signal even though neither publishes latency benchmarks for it.

Practical consequences:

• Measure per-peer RTT and feed it to the planner, distinguishing direct from relayed paths. A peer on a relay may be worse than no peer at all, because adding it as a stage multiplies its latency across every token.
• Prefer a LAN-scoped discovery mode when the mesh is a LAN. Mesh LLM's --mesh-discovery-mode mdns skips Nostr relays, public iroh relays, and STUN probing entirely.
• Do not let the transport choose your interface. On multi-NIC Linux and docker run --network host machines, iroh discovers and advertises Docker or CNI bridge addresses such as 172.17.0.1. If every host in the mesh has the same bridge address, peers race the wrong local path and connect to themselves or to nothing. Pin it (--bind-ip). This is a real, reported failure and it is not obvious from a stack trace.

Core knowledge: discovery, membership, and admission

Discovery (how do I find any peer at all?) and membership (who is in the mesh right now, and what can they do?) are separate problems, and the systems in this lineage answer them differently.

Mechanism	Used by	Trust and reach
Kademlia DHT over libp2p	hivemind, and Petals on top of it (servers announce their active blocks to the DHT)	Public, permissionless, no central server; a joining peer needs bootstrap addresses
Nostr relay events	Mesh LLM (--publish)	Public; a replaceable event of custom kind 31990 carrying an expiration tag, so it ages out if the node exits
mDNS	Mesh LLM (mdns mode)	LAN only; no public infrastructure touched
Bespoke UDP multicast beacon	exo (IPv6 group ff12::e0a1:de89, over zenoh)	LAN only; not a standard protocol, so nothing else can discover it
Bootstrap peers	hivemind	Explicit addresses, configured out of band
Invite token	Mesh LLM (private meshes)	Private by default; possession of the token is the admission credential

The load-bearing insight is that public discovery makes admission control mandatory. The moment your node is discoverable by anyone, anyone can open a connection to it. Mesh LLM's answer is worth studying because it is small and correct:

• Quarantine until gossip. Under the default trust policy, a newly connected peer may open only the gossip, route-table, and passive HTTP-tunnel streams (a stricter RequireOwned/Allowlist policy narrows this to gossip alone). Everything else is refused until it sends a valid gossip frame. An unauthenticated peer cannot reach the inference path, the plugin channel, or the artifact transfer.
• Bind the application identity to the transport identity. A gossip frame's sender_id must equal the QUIC TLS peer identity, so a peer cannot claim to be another node. Forged "peer leaving" frames naming a different node are rejected outright.
• Require confirmation before acting on a peer's claim about a third party. A "peer down" report triggers an independent reachability check before the named peer is evicted. Without this, any node can evict any other node by lying.
• Gate on protocol generation. Every frame carries gen, and a mismatch is rejected at the frame level rather than producing subtle downstream corruption.
• Cap frame size (8 MiB) so a peer cannot request an unbounded allocation.

Membership is also where the scheduler gets its inputs. Mesh LLM's gossip PeerAnnouncement carries per-GPU VRAM, reserved bytes, memory bandwidth, compute hints, hostname, the models the peer has on disk, and the observed RTT. The gossip payload is the planner's input vector, which is why a transport that cannot gossip hardware facts cannot support a hardware-aware placement algorithm, and why the placement papers quietly assume one.

Don't-miss checklist

• Measure and record per-peer hop RTT, direct versus relayed, and feed it to the placement planner. The maximum, not the mean, is what sets the pace.
• Assume something near half of peers fail to get a direct path end to end, and provision for it. The often-quoted 70% hole-punch rate is conditional on earlier steps already succeeding. Do not design as though hole punching usually works.
• Split the big model, keep the small one home. WAN tolerance scales with hidden size; a 7B model across a WAN is the worst of both worlds.
• Pin the mesh interface explicitly on any multi-NIC or Docker host.
• Separate the traffic classes. Bulk weight transfer must not be able to head-of-line block an activation frame. If you are on TCP, you need separate connections; on QUIC you get it free.
• Never expose a tensor-op RPC port (llama.cpp rpc-server) to an untrusted network. Its own documentation forbids it.
• Bind application identity to transport identity. A peer's claimed node id must be its TLS identity, or your gossip is forgeable.
• Verify a third-party claim before acting on it. Eviction on an unverified "peer down" report is a denial-of-service primitive.
• Budget the stage count against measured RTT before you split, not after. At a 10 ms hop, two stages is a real budget and four cannot reach 30 tok/s at all.
• Reach for speculative decoding first, not compression. It is the only lever on the binding term.

Failure modes

• Buying bandwidth to fix decode latency. The link is roughly 99% idle during decode. This is the expensive mistake, and the derivation above prices it at zero.
• One relayed peer silently setting the pace. The planner takes max(hop_latency), so a single peer that failed to hole-punch degrades every token for everyone. Detect relayed paths and treat them as a placement penalty, not a curiosity.
• Head-of-line blocking from mixing bulk and latency-critical traffic on one ordered stream. Symptom: decode latency spikes exactly when a new peer is downloading weights. Cause: TCP.
• Assuming hole punching always works. Symmetric NAT defeats it. Plan for a relay path and measure how many of your peers are on one.
• Advertising a Docker bridge address. Every host claims 172.17.0.1, peers race the wrong path, and the mesh forms but does not work.
• Public discovery without admission control. Discoverable means reachable. Quarantine unauthenticated peers away from every stream except the one that authenticates them.
• Trusting gossip you did not bind to a transport identity. Unbound gossip lets any peer impersonate any other, evict rivals, or advertise VRAM it does not have to attract stage assignments.

Open questions and validation

• No numbers on this page were measured on live WAN hardware in this KB. The latency and bandwidth model is executed and self-contained; the RTT figures used to illustrate it are assumptions, and the one real WAN datapoint in Mesh LLM's repository (Sydney to Queensland, about 20 ms, 10 to 25 tok/s) is from a code path that no longer exists. Measuring a real Skippy-class stage pipeline across two datacentres is the single most valuable experiment this KB has not run.
• The 70% hole-punch rate is measured on IPFS/libp2p traffic, not on an inference mesh. Whether an inference peer pool (likelier to be datacentre-hosted, less likely to be residential) traverses better is unverified, and it is the single number that most determines whether a public mesh is viable.
• Activation compression has two different success criteria and the literature conflates them. Petals states that dynamic blockwise quantization of the hidden states "halves the bandwidth requirements without any noticeable effect on generation quality." That is a quality claim, and worth noting that the paper publishes no quality evaluation specifically for the activation quantization (its zero-shot accuracy table measures 8-bit weight compression, a separate mechanism, and it would be a mistake to borrow those numbers as evidence for this one). Mesh LLM meanwhile rejects q8 activations for about 42% of its reviewed model families on an exactness criterion: it changes the next token. Both positions can be right at once, and which you need depends on whether you owe a user reproducible outputs or merely good ones. Per the model above, neither helps decode.
• Connection migration's behavior under a stage assignment is untested here: a node that migrates networks mid-request keeps its QUIC connection, but whether its hop latency changes enough to invalidate the topology the planner chose is an open question.
• Verification of an untrusted stage's output is not solved by the transport and is not attempted by Mesh LLM. The most credible deployed answer is Prime Intellect's TOPLOC (arXiv 2501.16007), which commits to intermediate activations with a locality-sensitive hash, compresses a proof to 258 bytes per 32 tokens, and is used in INTELLECT-2 to verify "rollouts from untrusted inference workers." Its reported "100% accuracy" at detecting a modified model, prompt, or precision needs four caveats the abstract omits: three from its own section 6, and a fourth from its methodology. Section 6 states it cannot detect speculative decoding (so a provider swapping in a cheap decoder passes), that the fp8-versus-bf16 margin is small enough that reliable discrimination may need matching device and attention configurations, and that minor prompt tweaks and lightly fine-tuned models are "significantly harder" to catch. Section 5.2 adds the fourth: the accept/reject thresholds were chosen from the same error statistics (Tables 2 and 5) later used to report the 100%-accuracy result, so the figure is in-sample. Treat it as a strong deterrent, not a proof. Whether it composes with a Mesh-LLM-style stage pipeline is untested here.

References

• RFC 9000, QUIC: A UDP-Based Multiplexed and Secure Transport (streams, connection migration, connection IDs): https://www.rfc-editor.org/rfc/rfc9000.html
• RFC 9001, Using TLS to Secure QUIC (handshake, 0-RTT): https://www.rfc-editor.org/rfc/rfc9001.html
• iroh: QUIC peer-to-peer connections with hole punching, relay fallback, and dial-by-public-key: https://github.com/n0-computer/iroh
• Mesh LLM node message protocol: stream types, gossip frames, admission, anti-spoofing, frame caps: https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/design/message_protocol.md
• Mesh LLM mesh workflows: discovery modes, invite tokens, relay behavior, interface pinning: https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/MESHES.md
• llama.cpp RPC backend, with its own "fragile and insecure, never run on an open network" warning: https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc
• hivemind: decentralized training over the internet, DHT on go-libp2p, fault-tolerant forward and backward passes: https://github.com/learning-at-home/hivemind
• Borzunov et al., Petals: Collaborative Inference and Fine-tuning of Large Models (arXiv 2209.01188, ACL 2023 System Demos), Kademlia DHT block announcement, libp2p Circuit Relay for NAT traversal, and dynamic blockwise activation quantization: https://arxiv.org/abs/2209.01188
• Borzunov et al., Distributed Inference and Fine-tuning of LLMs Over The Internet (arXiv 2312.08361, NeurIPS 2023), the fuller evaluation, including "performance does not depend much on bandwidth or sequence length but degrades with higher latency": https://arxiv.org/abs/2312.08361
• Ryabinin et al., SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient (arXiv 2301.11913, ICML 2023); section 3.1 states the "Square-Cube Law of Distributed Training" and Table 1 measures WAN tolerance against model size: https://arxiv.org/abs/2301.11913
• Narayanan et al., Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (arXiv 2104.04473, SC'21); section 3.2 gives the canonical pipeline-versus-tensor communication volumes (bsh between consecutive devices per pass, versus 8bsh(t-1)/t per layer per device per microbatch). Its section 4.1 is a separate scatter-gather optimization that lowers the pipeline figure to bsh/t on a fast fabric: https://arxiv.org/abs/2104.04473
• Trautwein et al., Challenging Tribal Knowledge: a Large Scale Measurement Campaign on Decentralized NAT Traversal (arXiv 2510.27500); libp2p DCUtR in production IPFS, 4.4M analyzed attempts (of 6.25M tracked), hole-punch success reported as 70% plus or minus 7.1% and indistinguishable between TCP and QUIC. The rate is conditional on relay reservation and public-address discovery already succeeding, which the paper excludes about 29% of data points for, and the authors note it is likely optimistic: https://arxiv.org/abs/2510.27500
• Prime Intellect prime-iroh, an iroh-based P2P backend for decentralized pipeline parallelism: https://github.com/PrimeIntellect-ai/prime-iroh
• Prime Intellect prime-vllm, vLLM modified to run pipeline parallelism over public networks: https://github.com/PrimeIntellect-ai/prime-vllm
• Ong et al., TOPLOC: a Locality Sensitive Hashing Scheme for Trustless Verifiable Inference (arXiv 2501.16007), activation commitments at 258 bytes per 32 tokens; its section 6 lists the limits, including an inability to detect speculative decoding: https://arxiv.org/abs/2501.16007
• INTELLECT-2 (arXiv 2505.07291), the globally distributed RL run that deploys TOPLOC to verify "rollouts from untrusted inference workers": https://arxiv.org/abs/2505.07291
• exo, whose current tree pins zenoh and discovers peers over a bespoke IPv6 UDP multicast beacon rather than the libp2p stack older comparisons describe: https://github.com/exo-explore/exo
• Nostr NIP-89, application handler events (kind 31990): https://github.com/nostr-protocol/nips/blob/master/89.md

Related: Mesh LLM and Skippy stage splits · Cross-WAN model-parallel inference · Non-colocated inference: which pattern? · Overlay & mesh networking (geo-distributed) · Speculative decoding · Inference parallelism strategies · Remote GPU verification · Region-replica routing