Markdown

Decentralized inference with Petals¶

Scope: Petals (bigscience-workshop/petals, MIT), the system that runs large language models across a swarm of volunteer GPUs, BitTorrent-style. This page covers the block-span serving model (each server hosts a contiguous span of transformer layers), the DHT-based discovery and Dijkstra min-latency routing the client uses to chain servers into a full model, the privacy trade-off of a public swarm, and the project's current dormant status. It is a foundational reference among this knowledge base's decentralized and distributed serving pages, alongside Prima.cpp, cross-WAN model-parallel inference, and split inference privacy over WAN; it extends serving open-weight models.

Examined at commit 22afba62 of bigscience-workshop/petals, MIT. Petals is dormant: its last commit is from September 2024 and it pins transformers==4.43.1, so treat it as a reference design rather than a maintained tool for current models. The min-latency routing model below is executed and asserted as standalone Python, modeled on src/petals/client/routing/sequence_manager.py (_make_sequence_with_min_latency, _build_inference_graph). No swarm was run: Petals needs volunteer GPU servers and a DHT, so the "up to 10x faster than offloading" claim is the project's own, not measured here.

What it is¶

Petals runs a large model that would not fit on your machine by distributing its layers across many machines. The model's transformer blocks are partitioned into contiguous spans; each volunteer server hosts one span (say blocks 0 to 20) on its GPU. A client wanting to run the whole model finds a set of servers whose spans cover every block, chains them into a route, and streams activations hop by hop: your machine holds only the embeddings and the head, and the heavy per-layer compute happens on the swarm. This lets a desktop generate text with Llama 3.1 (up to 405B), Mixtral 8x22B, Falcon, or BLOOM by borrowing the swarm's collective GPU memory.

Discovery is over a distributed hash table (hivemind DHT): servers announce which block spans they serve, and the client's RemoteSequenceManager reads the DHT to learn who serves what. Routing is a shortest-path problem. _make_sequence_with_min_latency builds a graph whose nodes are (server, block) pairs and whose edges carry network delays (RTT-derived) plus a cache-allocation penalty when a server lacks KV room, then runs Dijkstra from a virtual start to end to pick the lowest-latency chain that covers every block. A max_throughput mode optimizes differently for batch work. You can also host a span yourself to add capacity, or run a private swarm among machines you trust.

Why it matters¶

Collective memory beats offloading. For a model too large for local VRAM, streaming activations to swarm servers that keep layers resident is faster than swapping layers to disk or CPU on one machine; the project reports up to 10x over offloading.
A clean decentralized-serving design. The block-span partition, DHT discovery, and shortest-path routing are the canonical pattern for volunteer or geo-distributed inference, and later systems build on the same ideas.
Latency-aware routing. The Dijkstra route accounts for real network delays and cache pressure, so the client picks a fast chain rather than any covering chain.
Self-hostable capacity. Anyone can serve a span and expand the swarm, or stand up a private swarm for sensitive data.
Foundational reference. Even dormant, Petals is the clearest worked example of BitTorrent-style model serving, which is why it belongs in a distributed-inference reading list.

When to use it (and when not)¶

Use it as the reference design for decentralized or volunteer-GPU inference, and as a starting point for a private swarm among trusted machines when you understand it is unmaintained. The routing and block-span ideas transfer directly to any system that spreads a model across the network.

Do not deploy it for current production serving: it is dormant, pinned to a 2024 transformers, and will not support newer model architectures without work. Do not send sensitive data through the public swarm: your activations are processed by other people's machines, so the project itself points you to a private swarm for anything confidential (this is the exact concern the split inference privacy page addresses). Do not expect the routing to hide a coverage gap: as the model below shows, a missing block is a hard error, not a silent partial route. For a maintained home-cluster alternative, see Prima.cpp.

Architecture¶

flowchart TB
  CLIENT["Client: embeddings + head"] --> DHT["hivemind DHT: who serves which blocks"]
  DHT --> ROUTE["RemoteSequenceManager: Dijkstra min-latency route"]
  ROUTE --> S1["Server A: blocks 0-1"]
  S1 --> S2["Server B: blocks 2-3"]
  S2 --> CLIENT
  S1 -. announces span .-> DHT
  S2 -. announces span .-> DHT

Min-latency routing, executed¶

The client's routing is a shortest path over a (server, block) graph. Edges carry one-way network delay (RTT/2 plus a serialization overhead constant) and, on the entry edge to a server, a cache-allocation penalty when that server lacks KV room. Petals also forces a chosen server to the end of its span before allowing a switch (to avoid an O(N^2) graph), so switches happen only at span boundaries. The model below reproduces that and shows the cache penalty, not raw RTT, deciding the route.

import heapq

OVERHEAD = 0.018      # serialization overhead per hop (empirical constant in petals)
ALLOC_PENALTY = 10.0  # added on entry when a server has no KV cache room


def dijkstra(graph, src, dst):
    dist, prev, pq = {src: 0.0}, {}, [(0.0, src)]
    while pq:
        d, u = heapq.heappop(pq)
        if u == dst:
            break
        if d > dist.get(u, float("inf")):
            continue
        for v, w in graph.get(u, []):
            nd = d + w
            if nd < dist.get(v, float("inf")):
                dist[v], prev[v] = nd, u
                heapq.heappush(pq, (nd, v))
    if dst not in dist:
        raise ValueError("MissingBlocks: no chain covers every layer")
    path, node = [], dst
    while node != src:
        path.append(node)
        node = prev[node]
    return list(reversed(path)), dist[dst]


def build_and_route(n_blocks, servers, client_rtt, cache_ok):
    serving = {b: [p for p, (s, e) in servers.items() if s <= b < e] for b in range(n_blocks)}
    missing = [b for b in range(n_blocks) if not serving[b]]
    if missing:
        raise ValueError(f"MissingBlocks: {missing}")
    g = {}
    def edge(a, b, w): g.setdefault(a, []).append((b, w))

    def entry_delay(p):
        return client_rtt[p] / 2 + OVERHEAD + (0 if cache_ok[p] else ALLOC_PENALTY)

    for p in serving[0]:
        edge("start", (p, 0), entry_delay(p))
    for b in range(1, n_blocks):
        for p in serving[b]:
            if servers[p][0] <= b - 1:                     # same server, next block: free
                edge((p, b - 1), (p, b), 0.0)
        for q in serving[b - 1]:                           # switch only at a span boundary
            if servers[q][1] == b:
                for p in serving[b]:
                    if p != q:
                        edge((q, b - 1), (p, b), entry_delay(p))
    for p in serving[n_blocks - 1]:
        edge((p, n_blocks - 1), "end", client_rtt[p] / 2)

    path, cost = dijkstra(g, "start", "end")
    spans = []
    for node in path:
        if node in ("start", "end"):
            continue
        if not spans or spans[-1] != node[0]:
            spans.append(node[0])
    return spans, cost


# A and B both serve the whole model; A is nearer but has NO cache room.
servers = {"A": (0, 4), "B": (0, 4)}
rtt = {"A": 0.02, "B": 0.20}
spans, cost = build_and_route(4, servers, rtt, {"A": False, "B": True})
print(f"A cache-full -> route {' => '.join(spans)} ({cost:.3f}s)")
assert spans == ["B"]                                       # +10s penalty outweighs A's low RTT

spans2, cost2 = build_and_route(4, servers, rtt, {"A": True, "B": True})
print(f"cache freed  -> route {' => '.join(spans2)} ({cost2:.3f}s)")
assert spans2 == ["A"] and cost2 < cost                     # now the low-RTT server wins

half = {"A": (0, 2), "B": (2, 4)}                            # split coverage -> two-hop chain
spans3, _ = build_and_route(4, half, {"A": 0.02, "B": 0.02}, {"A": True, "B": True})
print(f"split 0-1 | 2-3 -> route {' => '.join(spans3)}")
assert spans3 == ["A", "B"]

try:
    build_and_route(6, half, {"A": 0.02, "B": 0.02}, {"A": True, "B": True})
    raise AssertionError("should have raised")
except ValueError as e:
    print("coverage gap ->", str(e))
print("OK: routing is Dijkstra over (peer, block); the cache penalty rides the entry "
      "edge, servers are used to their span end before a switch, and gaps fail loudly")

Executed output:

A cache-full -> route B (0.218s)
cache freed  -> route A (0.038s)
split 0-1 | 2-3 -> route A => B
coverage gap -> MissingBlocks: [4, 5]
OK: routing is Dijkstra over (peer, block); the cache penalty rides the entry edge, servers are used to their span end before a switch, and gaps fail loudly

Two design lessons fall out. First, the route is driven by cache pressure and network delay together, not by proximity alone: a nearby server with no KV room loses to a farther server that can actually hold the request. Second, coverage is checked up front; a swarm that does not collectively serve every block raises MissingBlocks rather than returning a broken partial chain, which is the right failure mode for a system where servers come and go.

How to use it¶

The client API mirrors transformers (reference template, pinned to the 2024 stack):

# Reference template, unexecuted (requires a running Petals swarm and transformers==4.43.1).
from transformers import AutoTokenizer
from petals import AutoDistributedModelForCausalLM

model_name = "meta-llama/Meta-Llama-3.1-405B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoDistributedModelForCausalLM.from_pretrained(model_name)  # connects to the swarm

inputs = tokenizer("A cat sat", return_tensors="pt")["input_ids"]
outputs = model.generate(inputs, max_new_tokens=5)
print(tokenizer.decode(outputs[0]))

To add capacity, run a server that hosts part of a model on your GPU; to keep data off the public swarm, launch a private swarm among machines you trust (the project's wiki documents both). Gated models like Llama 3.1 need a Hugging Face login first.

How to develop with it¶

The routing is the piece worth studying: src/petals/client/routing/sequence_manager.py builds the inference graph and runs Dijkstra, sequence_info.py tracks which spans cover which block, and inference_session.py streams the forward pass along the chosen chain. To adapt Petals to a new model family you implement the block partition in src/petals/models. Because it is dormant, expect to update the transformers pin and the model integrations yourself; treat the codebase as a design to port rather than a dependency to add.

How to maintain it¶

There is effectively no upstream maintenance: the last commit is September 2024 and the transformers==4.43.1 pin will not load newer architectures. If you build on it, fork and update the pin and model code, and re-validate the DHT and routing against your hivemind version. For current work, prefer a maintained decentralized or home-cluster system and use Petals as the reference for how the routing should behave.

How to run it in production¶

Petals is not a production system in 2026; running it means a private swarm you maintain among trusted machines, or a research reproduction. If you do, keep sensitive workloads on a private swarm (never the public one, since servers see your activations), size the swarm so every block is covered with redundancy (so a departing server does not trigger MissingBlocks), and account for the WAN reality that decode is latency-bound: each token traverses the whole server chain, so a long chain over high-RTT links is slow regardless of per-server compute. The cross-WAN model-parallel inference and Prima.cpp pages cover the maintained end of this design space.

Failure modes¶

Dormant and pinned. September 2024 last commit, transformers==4.43.1; newer models need code changes.
Public-swarm privacy. Your activations are processed by strangers' machines. Use a private swarm for anything sensitive.
Coverage gaps. A swarm that does not collectively serve every block raises MissingBlocks; production swarms need redundant coverage.
WAN decode latency. Every token crosses the full server chain, so high-RTT links and long chains dominate latency, which no per-server GPU speed fixes.
Volunteer churn. Servers join and leave; the routing re-plans, but a run can stall if the swarm thins mid-generation.

References¶

Petals repository (BigScience), pinned commit 22afba62: https://github.com/bigscience-workshop/petals
Routing (src/petals/client/routing/sequence_manager.py): https://github.com/bigscience-workshop/petals/blob/main/src/petals/client/routing/sequence_manager.py
Petals paper: https://arxiv.org/abs/2209.01188
Swarm health dashboard: https://health.petals.dev
hivemind (the DHT it builds on): https://github.com/learning-at-home/hivemind