Skip to content
Markdown

Recipe: Mesh LLM production deployment

Scope: the operational bring-up this KB's conceptual page does not cover: installing and supervising mesh-llm so it survives disconnects and crashes, securing a private mesh with owner identity and a trust policy, gating readiness before trusting the endpoint, turning on reliability guardrails, monitoring a running mesh, reproducing WAN conditions in a local test lab before a real cross-region rollout, and upgrading safely. Read Mesh LLM and Skippy stage splits first: it has the performance model (when splitting helps versus hurts), the trust-boundary analysis, and the architecture this recipe assumes. This page assumes you already decided splitting is the right call and now need to run it as more than a demo.

Every command below is transcribed from the project's own current documentation (docs/CLI.md, docs/USAGE.md, docs/MESHES.md, docs/BENCHMARKS.md, docs/NODE_REP.md, docs/specs/mesh-setup-installer.md, docs/specs/vram-accounting.md, .skills/deploy-linux-gpu/SKILL.md, docker/skippy-wan-lab/README.md, and the identity design docs under docs/design/), at commit 2c2e808be11f (2026-07-15T03:39:14Z), Apache-2.0, 1,924 commits, actively developed; the latest tagged release is v0.73.1, so re-check flag names against your installed version if it predates this commit. There is no associated academic paper and no independent third-party production writeup, this is the project's own documentation, reviewed for internal consistency, not corroborated externally. Mesh LLM moves fast; re-verify every flag against mesh-llm --help / mesh-llm <command> --help before depending on it, and treat anything marked "design-stage" below as unconfirmed until you have checked it against your installed version yourself.

Executed here (2026-07-17, GPU-less x86_64 Linux host, installed release binary mesh-llm 0.73.1; the live repo was re-checked the same day, HEAD 8e5bb12, v0.73.1 still the latest tag): the pinned install.sh flow; the exact flag surface of every command this page quotes, via --help at v0.73.1; auth init, auth status, and the full auth trust/revoke-node/revoke-owner incident set; config validate; a real single-node serve with a small GGUF driven through /api/status, a real /v1/chat/completions request, and a live runtime guardrails switch; the owner-control runtime bootstrap and get-config round trip; a second process joining by invite token with owner attestation verified; a SIGKILL peer-death transcript and a mid-request node-death transcript; doctor split; uninstall --dry-run; and a genuine cross-version canary-then-rollback (update --version v0.73.0 -> v0.73.1 -> v0.73.0, --flavor cpu, two real releases fetched from GitHub, not a same-version reinstall, in an isolated HOME). Also executed: a real release-attestation-required mesh, with a real admitted peer and a real rejected peer (the rejection is silent on the rejected side, see step 2); metrics-server built from source and fed real OTLP telemetry from a live two-node split mesh, standalone, without the Docker WAN lab (see step 9); and models certify against a real, resolvable layer package, which panics in both --package-only and --api-base modes rather than returning a report, a genuine bug on this pinned CPU release binary (see step 8). Still not validated here: every GPU path, a real multi-host split across physically separate machines, the Docker WAN lab itself, and setup --service. See the executed-transcripts section below, including the real command failures found while verifying.

What it is

A production bring-up is five concerns the conceptual page treats as given: the process must survive an SSH disconnect or a crash, a private/business mesh needs an identity and trust policy beyond a bare invite token, an endpoint must not serve traffic before every stage it depends on is actually ready, an operator needs a way to see the mesh is healthy without SSHing into every box, and an upgrade must not silently break a running deployment. Mesh LLM ships real, documented mechanisms for most of this; this recipe assembles them into one path instead of leaving each as an isolated doc page.

Why it matters

Skipping any one of these concerns creates a named failure mode later in this page: a foreground process or shell-backgrounded & job can end on disconnect; a detached tmux session survives a normal SSH disconnect but provides no crash or reboot restart; a leaked invite token does not establish node ownership; an open API port does not prove every split stage is ready; disabled guardrails enforce nothing; and a fleet-wide upgrade without a certified canary increases rollback risk.

When to use it (and when not)

Use this recipe once Mesh LLM and Skippy stage splits has told you splitting is warranted: capacity is scattered, no single node holds the model, or a node is thrashing against its memory limit. Do not use this recipe as a reason to split a model that already fits on one machine; re-read that page's network-floor arithmetic first. If a full replica fits everywhere you have capacity, you do not need any of this.

Architecture

flowchart TB
  subgraph Owner["Operator: owner identity"]
    OK["mesh-llm auth init<br/>owner keystore"]
  end
  OK -->|"owner-key, node-label"| Seed["Seed / coordinator node<br/>systemd/supervisor supervised<br/>mesh-llm serve --split --model &lt;layer-package&gt;"]
  Seed -->|"invite / bootstrap token"| W1["Worker node 1<br/>supervised, --join &lt;token&gt;"]
  Seed -->|"invite / bootstrap token"| W2["Worker node 2<br/>supervised, --join &lt;token&gt;"]
  Seed --> Ready{"All stages ready?<br/>/api/status"}
  Ready -->|"no"| Wait["Do not route traffic yet"]
  Ready -->|"yes"| LB["Client traffic:<br/>http://seed:9337/v1"]
  Seed -.->|"OTLP telemetry"| Metrics["metrics-server<br/>(from the Docker WAN lab,<br/>self-hosted, not built into serve)"]
  Seed -.->|"/api/status"| Ops["Operator: node reputation,<br/>guardrail mode, readiness"]

How: implement, integrate, maintain

1. Install and verify the binary on every node

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/2c2e808be11fd417e3b864db907db92151f13ee2/install.sh | \
  MESH_LLM_REQUIRE_CHECKSUM=1 sh -s -- --no-setup
~/.local/bin/mesh-llm update --version v0.73.1 --flavor cuda
~/.local/bin/mesh-llm --version
nvidia-smi --query-gpu=name,memory.total,compute_cap,driver_version --format=csv,noheader

The installer auto-detects the GPU and CUDA major version and pulls the matching build. ~/.local/bin is not always on a non-interactive SSH PATH; use the full path or a login shell (ssh -tt host 'bash -lc "..."').1 If mesh-llm --version fails at the loader before you ever reach the update line, the auto-detection picked a flavor the host cannot satisfy; the executed transcripts below reproduce that failure verbatim and show the recovery.

The public two-stage install/setup flow is install.sh then mesh-llm setup; the bootstrap script owns only fetching and installing the executable (with checksum verification, best-effort by default, fail-closed only with MESH_LLM_REQUIRE_CHECKSUM=1), and mesh-llm setup owns everything after that: native runtime install, and service registration.2

~/.local/bin/mesh-llm setup --service --verbose

--service is supported for Linux systemd user units and macOS launchd agents; it is a hard, explicit error on Windows (setup cannot install the background service on windows, crates/mesh-llm-commands/src/setup/service.rs), and the systemd user unit is written under ~/.config/systemd/user/mesh-llm.service (setup/service_paths.rs).2 The exact unit file is written and managed by the installer, not hand-authored here; use --verbose to print its actual path and status on your machine rather than assuming a path.

Executed: real transcripts from a GPU-less host (v0.73.1)

Everything in this section ran for real on GPU-less x86_64 Linux against the pinned commit's install.sh and the v0.73.1 release binary. Output is pasted from the runs; the CLI's emoji line-prefixes are stripped, home directories are shortened to ~, and elisions are marked with ....

First real finding: the freshly installed binary would not even run --version.

$ ~/.local/bin/mesh-llm --version
~/.local/bin/mesh-llm: error while loading shared libraries: libnccl.so.2: cannot open shared object file: No such file or directory

The host has no GPU (/dev/nvidiactl absent) but has the CUDA toolkit installed, and install.sh treats any of nvidia-smi, nvcc, /dev/nvidiactl, or /proc/driver/nvidia/gpus as NVIDIA evidence (probe_nvidia, install.sh lines 249 to 255 at the pinned commit). It therefore selected the CUDA build on a host whose loader cannot satisfy it: this host ships CUDA driver userspace libraries but no libnccl.so.2, and the CUDA-flavor binary requires NCCL at load time. The old escape hatch is gone; the installer prints warning: MESH_LLM_INSTALL_FLAVOR is deprecated in install.sh and is ignored and installs the probed flavor regardless. And neither mesh-llm setup nor mesh-llm update --flavor cpu can rescue this state, because both live inside the binary that does not load. Two recoveries were exercised for real: pointing LD_LIBRARY_PATH at an existing libnccl.so.2 from an unrelated Python package already on disk made the binary start (fragile, and still the wrong flavor for a GPU-less host), and reinstalling from the plain CPU release asset, which worked cleanly and uses supported artifacts:

BASE=https://github.com/Mesh-LLM/mesh-llm/releases/download/v0.73.1
ASSET=mesh-llm-v0.73.1-x86_64-unknown-linux-gnu.tar.gz       # the cpu flavor carries no suffix
curl -fsSLO "$BASE/$ASSET" && curl -fsSLO "$BASE/$ASSET.sha256"
echo "$(awk '{print $1}' "$ASSET.sha256")  $ASSET" | sha256sum -c -   # printed: OK
tar -xzf "$ASSET"                                            # unpacks as mesh-bundle/mesh-llm
mv -f mesh-bundle/mesh-llm ~/.local/bin/mesh-llm
~/.local/bin/mesh-llm --version                              # printed: mesh-llm 0.73.1

The flavor-to-asset naming rule is asset_name() in install.sh: the CPU flavor is the bare x86_64-unknown-linux-gnu tarball; cuda, rocm, and vulkan carry suffixes.

Every command and flag this page quotes exists at v0.73.1. Checked against the installed binary's own help output: top-level, serve, auth and all its subcommands, runtime guardrails/bootstrap/get-config/apply-config, models certify, doctor split, update, uninstall, setup, and config. Help-surface quirks worth knowing: serve does not appear in the top-level command list (it is the documented preferred entrypoint anyway); --split, --bind-ip, and --bind-port are shown by mesh-llm --help-advanced rather than serve --help; and the owner-control subcommands plus runtime guardrails are absent from runtime --help's command list even though all of them parse and run. config validate takes --config-path, not a positional path. runtime get-config without --endpoint fails at the parser (error: the following required arguments were not provided: --endpoint <ENDPOINT>), so step 7's ControlEndpointRequired failure code is reachable only by API callers, not by this CLI.

With the binary working, a genuine single-node, non-split serve against a small real GGUF model (SmolLM2-135M-Instruct.Q4_K_M, ~105 MB, no GPU needed):

mesh-llm serve --gguf ./SmolLM2-135M-Instruct.Q4_K_M.gguf --port 19337 --console 13131

03:20:29 WARN  Configuration UI is read-only: no owner identity found. ...
03:20:32 INFO  Invite created for mesh 7e6e8b903b23171c780ba869557c9736: eyJpZCI6...
03:20:32 INFO  Waiting for peers...
03:20:32 OK    startup plan ready: 1 process(es), 2 endpoint(s), 1 model(s)
03:20:32 OK    api ready at http://127.0.0.1:19337
03:20:32 OK    web console ready at http://127.0.0.1:13131
03:20:36 INFO  Model loaded: local-gguf/sha256-e826c9a862671bf6
03:20:36 OK    Mesh runtime ready (1 model(s))
03:20:36 OK    model local-gguf/sha256-e826c9a862671bf6 ready on port 42479

Readiness, per the real /api/status response (step 5's gate, confirmed live): "node_state":"serving", "llama_ready":true, "runtime":{"models":[{"status":"ready", ...}]}. A real chat completion against the real running server:

$ curl -s http://localhost:19337/v1/chat/completions -H 'Content-Type: application/json' \
    -d '{"model":"local-gguf/sha256-e826c9a862671bf6","messages":[{"role":"user","content":"Say pong."}],"max_tokens":10}'
{"choices":[{"finish_reason":"length","index":0,"message":{"content":"Pong: The classic game of playing two-", ...}}],
 "usage":{"completion_tokens":10,"prompt_tokens":34,"total_tokens":44}, ...}

Guardrail default posture, confirmed live and then live-switched exactly as step 6 documents, no restart:

$ curl -s localhost:13131/api/status | jq '.runtime.openai_guardrails.mode'
"disabled"
$ mesh-llm runtime guardrails --mode metrics --port 13131
{"mode": "metrics", "status": {"mode": "metrics", ...}, "updated_models": 0}
$ curl -s localhost:13131/api/status | jq '.runtime.openai_guardrails.mode'
"metrics"

A second run on the same host verified the serve-time form too: a node started with --mesh-guardrails metrics reported "metrics" at the first poll of .runtime.openai_guardrails.mode, and runtime guardrails --mode enforce --port 3131 then switched it live to "enforce", confirmed by re-polling /api/status. Both of step 6's paths hold at v0.73.1.

Second real finding: models certify against this same live endpoint failed, no layer package found for "local-gguf/sha256-e826c9a862671bf6". Runtime certification (step 8) expects a proper model reference the catalog resolved (a Hub ref or a registered layer package), not the ad hoc local-gguf/sha256-<hash> identifier a raw --gguf file gets assigned at serve time. If your production path uses --gguf directly rather than a catalog/Hub model reference, models certify --api-base may not have a package to certify against; verify this against your exact model-loading path before relying on certify as the post-deploy gate step 8 describes, and prefer serving from a resolvable model reference (a Hub ref or registered layer package) specifically so certification works.

/api/status's local_instances field also surfaced something worth knowing operationally: it lists every mesh-llm process on the host, not just the one you started, each with its own PID, API port, and is_self flag. On a shared or multi-tenant host, do not assume the mesh-llm instance you launched is the only one influencing what /api/status reports.

Owner identity and the trust store, executed. auth init prompts Enter passphrase (empty for none):, and an empty answer writes an unencrypted keystore (auth status then reports Encrypted: no), so run it interactively in production and give it a passphrase. It prints the owner ID, writes ~/.mesh-llm/owner-keystore.json, and auth status also reports the trust store path, the active trust policy, and counts of trusted and revoked entries. Step 2's incident-response set ran for real:

$ mesh-llm auth trust add "$OWNER_ID" --label studio
Trusted owner d7ec51da... in ~/.mesh-llm/trusted-owners.json
$ mesh-llm auth revoke-node --node-id "$ENDPOINT_ID" --reason compromised
Revoked node ID 1111...
Updated trust store ~/.mesh-llm/trusted-owners.json
$ mesh-llm auth revoke-owner "$OWNER_ID" --reason compromised
Revoked owner d7ec51da... in ~/.mesh-llm/trusted-owners.json
$ mesh-llm auth trust list
Trusted owners:
- none
Revoked owners:
- d7ec51da... (compromised)
Revoked node IDs:
- 1111... (compromised)

revoke-owner also removes the owner from the trusted list; running trust remove afterwards reports the owner was not present. Every command here accepts --trust-store <path>, which makes disposable stores for incident drills trivially cheap.

Third real finding: this page's previous step 7 config example did not validate. v0.73.1 ships mesh-llm config validate (this page previously did not mention it). Validating step 7's example exactly as previously written failed:

$ mesh-llm config validate --config-path config.toml
Config invalid: config.toml
  error InvalidValue at owner_control.advertise_addr: owner_control.advertise_addr must use the same port as owner_control.bind
config validation failed

The example had been transcribed from the project's own docs/USAGE.md (lines 186 to 188 at the pinned commit), which pairs bind = "0.0.0.0:7447" with advertise_addr = "203.0.113.10:18443". The shipped validator (crates/mesh-llm-config/src/validate.rs) rejects exactly that pairing, and also requires bind whenever advertise_addr is set. Upstream documentation and upstream code disagree; the code wins. Step 7's example is corrected below, and with matching ports the same file passes (Config valid). Gate every config change on config validate before restarting a node.

Owner-control round trip on a live node.

$ mesh-llm runtime bootstrap --port 3131 --json
{
  "enabled": true,
  "endpoint": "eyJpZCI6...",
  "local_only": true,
  "requires_explicit_remote_endpoint": true
}
$ EP=$(mesh-llm runtime bootstrap --port 3131 --json | jq -r .endpoint)
$ mesh-llm runtime get-config --port 3131 --endpoint "$EP" --json | jq '{revision: .snapshot.revision}'
{
  "revision": 0
}

The same bootstrap payload is served at GET /api/runtime/control-bootstrap, and the full management-API endpoint list, including /api/status and /api/diagnostics/split-readiness, is documented at the top of crates/mesh-llm-host-runtime/src/api/mod.rs. get-config returned the node's full config snapshot with revision, config_hash, hostname, and node_id; that revision is what apply-config --expected-revision compares against.

Invite-token join, owner attestation, and a real peer-death transcript. A second process on the same host (separate $HOME, same owner keystore copied over, exactly the flow auth init prints) joined as a --client using the seed's invite token:

03:17:40 INFO  Connected to bootstrap peer; awaiting mesh admission
03:18:10 INFO  Running as client — proxying requests to mesh
03:18:12 INFO  Models available in mesh: ["local-gguf/sha256-ced8a651bc6cb794"]
03:18:12 OK    client ready
03:18:12 OK    api ready at http://127.0.0.1:9338

The seed's /api/status listed the peer with its node_label and "owner_verified": true. This validates the invite-token join and owner-attestation paths, not the split worker join (no layer package was involved). The joiner then got kill -9 while idle, and the seed's log shows what liveness detection actually looks like:

03:18:18  (joiner SIGKILLed at this timestamp)
03:21:48  WARN  Heartbeat: d02f42cea1 unreachable (1/2), will retry
03:22:58  WARN  Heartbeat: d02f42cea1 unreachable (2 failures), removing + broadcasting death
03:22:58  WARN  Peer d02f42cea1 died — removing and broadcasting

.peers in the seed's /api/status kept listing the dead node until the second heartbeat strike, 4 minutes 40 seconds after SIGKILL, and the reputation counters stayed at zero throughout because nothing routed a request through the dead peer; they penalize request failures, not membership. Two operational consequences: never use the coordinator's peer list as a liveness signal (health-check every node's own /api/status directly), and expect a several-minute window in which a dead idle peer still looks like a member.

Killing the serving node mid-request. With a /v1/chat/completions request in flight on the single-node mesh, the serving process got kill -9. The in-flight request failed with curl exit code 52 (empty reply, zero response bytes), and the readiness probe flipped to a connection error:

$ curl -sS -m 5 http://localhost:3131/api/status
curl: (7) Failed to connect to localhost port 3131 after 0 ms: Couldn't connect to server

That connection error is exactly the signal the load balancer's continuous health check has to catch. This is the single-node analogue of the split-mesh stage-failure mode; the split version still needs its own commissioning transcript (below).

Upgrade, split diagnostics, decommission, executed.

$ mesh-llm update --version v0.73.1 --flavor cpu
Updated to v0.73.1
$ mesh-llm --version
mesh-llm 0.73.1

$ mesh-llm doctor split --model-ref local-gguf/sha256-ced8a651bc6cb794 --port 3131
Split readiness: waiting_for_peers
Model: local-gguf/sha256-ced8a651bc6cb794
Eligible participants: 1
Excluded peers: 0
Recommended next steps:
  - Start at least one more worker/host with --model ... --split and join it to this mesh.
  - When testing multiple nodes on one machine, use distinct --port, --console, and --bind-port values for every process.

$ mesh-llm uninstall --dry-run
Mesh uninstall dry run
  Steps    9 planned
  Config   preserved

update --version <tag> --flavor <flavor> is the exact mechanism step 11 uses for both canary and rollback, exercised here as a real cross-version canary and rollback between two different published releases (transcript in step 11). doctor split reports an honest waiting_for_peers with remediation steps, and its multiple-processes-per-machine hint (distinct --port, --console, --bind-port) is what made the two-process mesh above possible.

Not exercised here, and what commissioning must capture. No GPU path ran on this host; the CUDA build does not even load, and the verbatim loader error above is that boundary. Also not exercised: a real multi-host split across physically separate machines (steps 3 and 4 with a layer package; the split mesh executed in step 9 used two processes on one host), and the Docker WAN lab's own Compose stack specifically (a plain metrics-server binary was run standalone instead, step 9), and setup --service (it mutates the host's systemd user session, so it was verified in source instead, cited in step 1). Release-attestation mesh creation and models certify against a real layer package WERE exercised (steps 2 and 8): the latter found a genuine panic, not a pass, so do not treat it as a validated gate. At commissioning on real GPU nodes, capture and file two transcripts before calling the deployment production-ready: a happy-path transcript (every stage ready in /api/status, one real completion through stage 0, models certify --api-base passing without panicking), and at least one failure transcript (SIGKILL a worker while a request is in flight; expect the in-flight request to fail outright and the topology to replan, record the detection-to-replan latency, and confirm the load balancer pulled the endpoint on the readiness probe rather than on user complaints).

2. Decide your mesh's security posture before the first serve

Three distinct mesh types exist, and the choice is made at creation time and cannot be changed in place, changing any of these later derives a brand-new mesh with a new mesh id:3

Mesh type Who can join Confirmed shipped?
Unrestricted (legacy) Anyone with the plain invite token Yes
Private, invite-token Anyone with the token; still no identity binding Yes
Requirement-aware, release-attestation-required Only peers presenting a trusted release attestation, via a signed bootstrap token Yes, executed here: a real admission and a real rejection, see below 
# Unrestricted / legacy private mesh
mesh-llm serve --model Qwen3-8B-Q4_K_M

# Requirement-aware public mesh requiring a trusted release signature
mesh-llm serve --model Qwen3-8B-Q4_K_M --publish \
  --require-release-attestation \
  --release-signer-key ed25519:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
  --owner-key ~/.mesh-llm/owner-keystore.json \
  --owner-required \
  --trust-policy require-owned \
  --node-label lab-a

# On a joining node, use the signed bootstrap token printed by the seed.
SIGNED_BOOTSTRAP_TOKEN="${SIGNED_BOOTSTRAP_TOKEN:?set to the signed token printed by the seed}"
mesh-llm serve --join "$SIGNED_BOOTSTRAP_TOKEN"

Two constraints are explicit in the project documentation. Release attestation is build provenance, not runtime attestation. It proves that a trusted release signer published the peer's binary; it does not prove the remote process is unmodified or that the host is uncompromised.3 Verifying a local packaged binary is not a plain mesh-llm subcommand; it requires a source checkout and Cargo: cargo run -p xtask -- release-attestation inspect --binary <path> --public-key-file <release-signing-public-key.json>.4 missing is expected for an unstamped development build; a packaged release binary can report valid or invalid.

Executed here: a real release-attestation-required mesh, a real admitted peer, and a real rejection. A throwaway ed25519 signer keypair was generated with the project's own xtask tool, and a spare copy of the installed v0.73.1 CPU release binary was stamped with it:

cargo run -p xtask -- release-attestation generate-keypair \
  --private-key-out signer.key --public-key-out signer.pub
cargo run -p xtask -- release-attestation stamp \
  --binary ./attested-mesh-llm --signing-key-file signer.key

Inspecting the real, unmodified v0.73.1 binary against this new key confirms it already carries an embedded attestation from the project's own release signer, not a blank slate:

$ cargo run -p xtask -- release-attestation inspect --binary ~/.local/bin/mesh-llm --public-key-file signer.pub --json
{"status":"invalid","signer_key_id":"ed25519:8266cc52d104c663fa19d139e32dc169776728b0fe9d17f89cc6917aaaa23ff1", "error":"supplied public key does not match embedded signer_key_id", ...}

invalid, not missing: the shipped binary is signed, just not by the throwaway key. Inspecting the stamped copy against the matching key confirms the opposite direction ("status":"valid"). A seed was then started with that throwaway key as the mesh's only allowed signer (--require-release-attestation --release-signer-key ed25519:<throwaway-pubkey> --owner-required --trust-policy require-owned). The first attempt ran the seed from its own real, installed binary and failed immediately: the policy is enforced against the genesis node too, not only against joiners:

mesh gossip rejected for peer 385744e10d [release_signer_untrusted]: the peer's certified build proof was signed by an untrusted release signer.

Restarting the seed from the stamped copy fixed that. Decoding the seed's own invite token (base64 then JSON) confirmed the embedded genesis policy carried exactly the throwaway key as its sole allowed_signer_keys entry. Two peers then tried to join with that token:

  • A peer running the stamped binary was admitted for real: the seed's /api/status lists it in peers[] with "release_attestation":{"status":"valid","signer_key_id":"ed25519:<throwaway-pubkey>",...} and "owner":{"status":"verified","verified":true}.
  • A peer running the project's real, unmodified release binary (a legitimate signer, just not the one this mesh's policy allows) reached "node_state":"client" and printed api ready locally, but never actually joined the gossip mesh: the seed logged the same release_signer_untrusted rejection, and the rejected peer's own /api/status shows "peers": [] permanently. A rejected release-attestation peer does not hard-fail or exit; it silently stays an ungossiped island that still looks locally "ready." Health-check the seed's peer list for this peer's node ID, not the rejected node's own status output, to catch it.

Initialize owner identity once per operator, then attach it to every node you run:

OWNER_ID="${OWNER_ID:?set to the trusted owner identity}"
mesh-llm auth init                      # generates/saves the owner keypair
mesh-llm auth status                    # shows identity/keystore status

mesh-llm serve --model Qwen3-14B \
  --owner-key ~/.mesh-llm/owner-keystore.json \
  --node-label studio \
  --trust-policy allowlist \
  --trust-owner "$OWNER_ID"

auth init, auth status, auth sign-node, auth renew-node, auth rotate-node, the runtime trust flags, and the local trust/revocation commands are present in the v0.73.1 command parser, and the init/status/trust/revoke set was exercised for real against the installed binary (transcripts above). The prose in docs/CLI.md does not enumerate every trust subcommand, so verify syntax with the installed binary's mesh-llm auth --help and mesh-llm auth trust --help.4 auth init prompts for a passphrase; an empty answer (including any non-interactive run) writes the keystore unencrypted, which auth status reports as Encrypted: no. Answer the prompt in production.

OWNER_ID="${OWNER_ID:?set to the trusted owner identity}"
ENDPOINT_ID="${ENDPOINT_ID:?set to the node endpoint identity}"

mesh-llm auth trust add "$OWNER_ID" --label studio
mesh-llm auth trust list
mesh-llm auth trust remove "$OWNER_ID"
mesh-llm auth revoke-node --node-id "$ENDPOINT_ID" --reason compromised
mesh-llm auth revoke-owner "$OWNER_ID" --reason compromised

These commands update a local trust store. Apply the same revocation state to every node that enforces the policy, and keep network isolation and process termination in the incident procedure while that distribution completes. Two behaviors verified live: revoke-owner also removes the owner from the trusted list, and every command here accepts --trust-store <path>, so incident-procedure drills can run against a disposable store instead of the real one.

3. Bring up the seed/coordinator node under supervision

A shell-backgrounded mesh-llm serve & can receive a hangup when its SSH shell exits. A detached tmux session normally survives that disconnect, but it does not restart the process after a crash or host reboot. On images that ship supervisor (common on rented GPU providers), use it for restart policy and boot persistence:1

cat > /etc/supervisor/conf.d/mesh-llm.conf <<'EOF'
[program:mesh-llm]
command=/root/.local/bin/mesh-llm serve --model unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XL --auto
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=30
stdout_logfile=/var/log/mesh-llm.log
stderr_logfile=/var/log/mesh-llm.log
environment=HOME="/root"
EOF
supervisorctl reread && supervisorctl update && supervisorctl start mesh-llm
supervisorctl status mesh-llm

Where there is no supervisor, prefer mesh-llm setup --service (systemd user unit / launchd agent, step 1) over ad hoc backgrounding. sudo loginctl enable-linger "$USER" makes a systemd user unit survive reboot before the user logs in.1

For a split deployment, the seed node becomes the coordinator by starting the mesh and printing the invite/bootstrap token; pin an immutable layer-package revision for anything you care about, only immutable refs are eligible for peer artifact transfer at all:5

REVISION="${REVISION:?set to an immutable layer-package revision}"
MODEL_REF="hf://meshllm/Qwen3-235B-A22B-UD-Q4_K_XL-layers@${REVISION}"
mesh-llm serve \
  --model "$MODEL_REF" \
  --split \
  --bind-ip 10.1.2.3 --bind-port 47916

4. Bring up worker nodes and join

JOIN_TOKEN="${JOIN_TOKEN:?set to the seed's invitation or bootstrap token}"
REVISION="${REVISION:?set to the seed's immutable layer-package revision}"
MODEL_REF="hf://meshllm/Qwen3-235B-A22B-UD-Q4_K_XL-layers@${REVISION}"

mesh-llm serve --split --join "$JOIN_TOKEN" --model "$MODEL_REF" \
  --bind-ip 10.1.2.4 --bind-port 47916

On multi-interface Linux hosts, especially docker run --network host systems, iroh (the mesh's QUIC transport) can discover and advertise a Docker/CNI bridge address such as 172.17.0.1. If every host has the same bridge address, peers race the wrong local path; --bind-ip on every node avoids this.3 --bind-port, paired with --bind-ip, gives you a fixed UDP port for firewall/NAT rules instead of an ephemeral one.4

5. Gate on readiness before trusting the endpoint

Stage 0 is not routable until every required downstream stage reports ready; a loaded runtime is not a routable one.5

curl -s http://localhost:3131/api/status | jq '{state:.node_state, stages:.runtime.stages}'
mesh-llm doctor split --model-ref meshllm/Qwen3-235B-A22B-UD-Q4_K_XL-layers --port 3131

Use the /api/status stage-readiness result for the load balancer's continuous health check; an open API port is insufficient. Run models certify after deployment and upgrades as a deeper functional gate, not as the load balancer's per-probe health endpoint.

One nuance verified live: .runtime.stages is null on a node that is not running a split; it populates per stage only on Skippy split runs. A probe that checks only stages therefore proves nothing on a mono-node deployment. On non-split nodes gate on node_state == "serving" plus the served model's "status": "ready" under .runtime.models (both observed on the executed node above).

6. Turn on reliability guardrails and confirm they are actually active

Mesh guardrails default to disabled. Enabling them is a separate, explicit step, both at server start and on an already-running node:64

# At server start:
mesh-llm serve --model MiniMax-M2.5-Q4_K_M --mesh-guardrails metrics   # or: enforce

# Or switch a running node without restart:
mesh-llm runtime guardrails --mode metrics --port 3131

# Always verify the posture actually took, do not assume the flag applied:
curl -s localhost:3131/api/status | jq '.runtime.openai_guardrails'

disabled, metrics, and enforce are the three documented modes; request-level mesh_guardrails flags on an individual API call cannot upgrade a server that started disabled, so the server-side posture is the one that matters for a production gate.4

7. Set production config: telemetry and the owner-control plane

~/.mesh-llm/config.toml is where a production node's non-default behavior should live, rather than accumulating as ad hoc CLI flags. Two sections matter specifically for production operation:11

# ~/.mesh-llm/config.toml
version = 1

[telemetry]
enabled  = true
endpoint = "http://localhost:4317"       # your OTLP collector, not a Mesh LLM-hosted service

[owner_control]
bind           = "0.0.0.0:7447"          # owner-control QUIC listen address
advertise_addr = "203.0.113.10:7447"     # advertised to authorized operator clients; port MUST match bind

An earlier revision of this page paired bind port 7447 with advertise_addr port 18443, transcribed from upstream docs/USAGE.md's own example. That configuration is invalid: the shipped validator rejects it with owner_control.advertise_addr must use the same port as owner_control.bind (crates/mesh-llm-config/src/validate.rs; failure reproduced and correction validated live, transcripts above). Upstream doc and upstream code disagree here, and the code wins. Gate every config edit before restart:

mesh-llm config validate --config-path ~/.mesh-llm/config.toml

Config precedence, highest to lowest: an explicit request field, then a per-model [[models]] entry, then [defaults.*], then family/topology policy, then built-in runtime defaults. Request-level defaults are merged only at the OpenAI frontend boundary when a field is absent or null; they never reach StageConfig, protobuf, or lower runtime layers. Editing this file affects future starts or reloads, not an already-running session.11

Owner-control is a separate protocol lane (mesh-llm-control/1) from the public mesh plane used for join, gossip, routing, and inference; it exists specifically for config and inventory mutation by an authorized operator, and it is not required for ordinary mesh participation.11 A new client needs an explicit bootstrap endpoint token before it can use it:

CONTROL_ENDPOINT="${CONTROL_ENDPOINT:?set from the bootstrap response}"

# Read the local bootstrap policy
mesh-llm runtime bootstrap --port 3131 --json
curl -s localhost:3131/api/runtime/control-bootstrap | jq .

# Read and apply config through owner-control, with optimistic-concurrency revisioning
mesh-llm runtime get-config --port 3131 --endpoint "$CONTROL_ENDPOINT" --json
mesh-llm runtime apply-config \
  --port 3131 \
  --endpoint "$CONTROL_ENDPOINT" \
  --expected-revision 7 \
  --config /absolute/path/to/config.toml \
  --json

A client with no explicit endpoint gets ControlEndpointRequired, by design, there is no silent fallback to a legacy config path.11 The other failure codes worth knowing before an incident, not during one:

Error Meaning
ControlUnsupported Target does not speak mesh-llm-control/1 at that endpoint
ControlUnavailable Endpoint token, listener, network path, or local owner-key unlock failed
Unauthorized The same-owner handshake failed, both sides must share an owner identity
RevisionConflict apply-config used a stale --expected-revision; re-read and retry
LegacyJsonUnsupported An old mesh-plane config frame hit the new owner-control lane

If the local bootstrap payload reports enabled=false, that means a signed same-owner keystore is required before owner-control requests can succeed on that machine, a prerequisite report, not a silent pass.11

Sizing note. Before pinning fit_target_mib/gpu_layers/tensor_split values in [[models]] entries, understand that Mesh LLM distinguishes rated VRAM (a user-facing capacity class like 32 GB), system-reported VRAM (raw driver-reported bytes, the internal source of truth), reserved VRAM (true driver/runtime-reserved bytes where the platform reports them, live used-memory counters are not this), and allocatable VRAM (system-reported minus reserved). Internal fit decisions use allocatable bytes; only the rated class is meant for user-facing labels.12 No documented minimum RAM or disk footprint exists anywhere in the project's docs; size hosts from the model/layer-package weight size plus KV budget, not from a published floor.

8. Certify the deployment before calling it production-ready

models certify gives a repeatable confidence check distinct from "the process is running." Package-only mode checks artifact integrity and local stage materialization; runtime mode additionally proves the live OpenAI-compatible endpoint by requiring real text-bearing responses from both Chat Completions and the Responses API, not just an HTTP 200:4

mesh-llm models certify hf://meshllm/Qwen3-8B-Q4_K_M-layers \
  --package-only --report-out cert-package.json

mesh-llm models certify unsloth/Qwen3-8B-GGUF:Q4_K_M \
  --api-base http://127.0.0.1:9337 --json

Run the runtime form after every deploy and after every upgrade (step 11), not just once at initial bring-up.

Executed here: both certify modes panic on a genuine layer package, on this pinned CPU release binary. Testing this properly requires more than the --gguf case already shown failing above with a handled error; it requires the actual Hub layer-package format models certify expects. Serving hf://meshllm/Qwen3-0.6B-Q4_K_M-layers directly (non-split, single node; the package format loads standalone as well as split) reaches a genuine "status":"ready" and answers a real /v1/chat/completions request. Certifying that live endpoint crashes instead of returning a report, in both modes:

$ mesh-llm models certify hf://meshllm/Qwen3-0.6B-Q4_K_M-layers --api-base http://127.0.0.1:19446 --json
thread 'tokio-rt-worker' (517544) panicked at crates/skippy-ffi/src/lib.rs:693:14:
MeshLLM native runtime library has not been loaded
task 19 panicked with message "MeshLLM native runtime library has not been loaded"

--package-only against the same package reference panics identically (no live server involved at all), with the same backtrace bottoming out in skippy_runtime::package::materialize_layer_package_details called from mesh_llm_host_runtime::inference::skippy::certification. Reproduced in both a fresh, isolated HOME and this host's real, already-populated ~/.mesh-llm/~/.cache/mesh-llm/native-runtimes, ruling out a missing-native-runtime-cache explanation. The certify subcommand's package-materialization path appears to assume the native llama.cpp/Skippy library is already loaded in-process, a state serve establishes at startup but the standalone certify command does not perform itself. Separately, the doc's own second example above (unsloth/Qwen3-8B-GGUF:Q4_K_M, a plain community Hub ref rather than a -layers package) was also tested directly with a same-shaped small model (unsloth/Qwen3-0.6B-GGUF:Q4_K_M) and fails the same way the --gguf-served node did earlier in this page (no layer package found for "unsloth/Qwen3-0.6B-GGUF:Q4_K_M"). Neither example command in this section, as written, produced a passing certification on v0.73.1 in this environment. Do not treat models certify as a validated production gate until you have run it successfully, end to end, against your own exact model reference and package format.

9. Monitor continuously

The most reliable per-node health signal without extra infrastructure is /api/status itself, polled on a schedule:

curl -s http://localhost:3131/api/status | jq '{
  state: .node_state,
  guardrails: .runtime.openai_guardrails,
  reputation: .routing_affinity.target_reputation
}'

routing_affinity.target_reputation exposes penalized_targets (how many local model+target entries currently carry a health penalty) and routes_penalized (how many route orderings moved a penalized candidate behind cleaner ones). This is process-local routing health, not a mesh-wide trust or correctness signal, do not treat a clean reputation counter as proof a peer is honest; it only means that peer has not recently timed out or returned unavailable to this node.7

Nor is it a liveness signal: in the executed peer-death transcript above, a SIGKILLed idle peer left both counters at zero, because reputation penalizes request failures and no request touched the dead peer. Membership eviction is heartbeat-driven with two strikes, and took 4 minutes 40 seconds; poll each node's own /api/status for liveness rather than the coordinator's peers list.

For real telemetry beyond point-in-time polling, point [telemetry] (step 7) at your own OTLP collector, or use the project's own reproducible reference implementation, metrics-server (OTLP/gRPC ingest), demonstrated in the Docker WAN lab (step 10) on ports :18080 (HTTP) and :14317 (OTLP/gRPC):8

RUN_ID="${RUN_ID:?set to the metrics-server run identifier}"
curl -s "http://127.0.0.1:18080/v1/runs/${RUN_ID}/status" | jq
curl -s "http://127.0.0.1:18080/v1/runs/${RUN_ID}/report.json" | jq

docker/skippy-wan-lab/README.md describes the storage as "the DuckDB file"; the shipped crate disagrees with its own project's docs here. crates/metrics-server/Cargo.toml depends on rusqlite (bundled SQLite), store.rs opens the file with rusqlite::Connection, and a real running instance below prints db=.../metrics.sqlite. It is SQLite, not DuckDB; plan tooling (backup, ad hoc queries) around the file format you actually get.

Executed here: metrics-server built from source and fed real OTLP telemetry from a live two-node split mesh, standalone, no Docker required. metrics-server is a plain workspace crate; nothing about it requires the WAN lab's Compose stack:

cargo build --release -p metrics-server
./target/release/metrics-server serve \
  --db metrics.sqlite --http-addr 127.0.0.1:28080 --otlp-grpc-addr 127.0.0.1:28317

A coordinator plus one joining worker (serve --gguf ... --split --debug --skippy-metrics-otlp-grpc http://127.0.0.1:28317) formed a real two-stage split mesh on this host. --debug is required for any of this to emit telemetry at all: skippy_telemetry_options (mesh-llm-host-runtime) only turns on OTLP export when --debug is set; --skippy-metrics-otlp-grpc alone does nothing. Before any inference request, the run already existed implicitly from startup spans:

$ curl -s http://127.0.0.1:28080/v1/runs/mesh-split-1784305178457013359/status
{"run_id":"mesh-split-1784305178457013359","status":"implicit","request_count":0,"span_count":9,"metric_export_count":0,"log_export_count":0}

A real /v1/chat/completions request through the split mesh's stage-0 raised the counts for real, confirmed by re-polling the same endpoint:

{"run_id":"mesh-split-1784305178457013359","status":"implicit","request_count":1,"span_count":36,"metric_export_count":0,"log_export_count":0}

report.json breaks this down per stage (skippy.stage_id, skippy.layer_start/skippy.layer_end, skippy.topology_id) and per request (stage_request_summaries, with per-stage span counts and start/end timestamps), confirming the ingested telemetry is genuinely structured around the split topology rather than a flat request counter. The run id is generated by the runtime itself (mesh-split-<nanos>, also visible in /api/status's runtime.stages[].run_id), not something you choose up front; there is no list-runs endpoint on metrics-server, so read the run id from a stage node's own /api/status rather than guessing it.

Gap to plan around: [telemetry] sends OTLP, there is no documented built-in Prometheus /metrics exporter on the serve path itself. If your production stack standardizes on Prometheus, run an OTLP-to-Prometheus bridge (e.g. an OpenTelemetry Collector with a Prometheus exporter) in front of [telemetry].endpoint, or scrape /api/status with a small custom exporter, rather than assuming a native /metrics endpoint exists.

10. Reproduce WAN conditions in a test lab before a real cross-region rollout

Before trusting a topology on an actual multi-region link, the project ships a Docker Compose lab that runs a four-stage CPU-backed chain with real Linux tc netem traffic shaping between stages and metrics-server wired in:8

TARGET_IP="${TARGET_IP:?set to the remote deployment target}"
cp docker/skippy-wan-lab/.env.example docker/skippy-wan-lab/.env

# Calibrate shaping parameters from a real target's measured RTT/bandwidth
scripts/skippy-wan-calibrate.sh "$TARGET_IP" docker/skippy-wan-lab/.env.link

# Bring the lab up (enforces the model-package invariant; do not use raw `docker compose`)
docker/skippy-wan-lab/up.sh

# Verify and exercise it
curl -s http://127.0.0.1:9337/v1/models | jq
docker/skippy-wan-lab/prompt.sh --max-new-tokens 64

Use WAN_ENABLE=0 to disable shaping without changing the compose topology, useful for isolating whether a problem is the network model or the serving stack. This is the closest thing to a pre-production integration test this project ships; there is no separate CI-integrated load-test harness documented for multi-node WAN behavior beyond this lab.

11. Upgrade and roll back

PREVIOUS_VERSION="${PREVIOUS_VERSION:?set to the recorded pre-canary release}"

# Canary upgrade to a pinned release and backend bundle.
mesh-llm update --version v0.73.1 --flavor cuda
systemctl --user restart mesh-llm

# Roll back by installing the previously recorded release.
mesh-llm update --version "$PREVIOUS_VERSION" --flavor cuda
systemctl --user restart mesh-llm

Record the current version before the canary update. After each update or downgrade, restart the process with its actual supervisor (systemctl --user above, or supervisorctl restart mesh-llm), wait for /api/status stage readiness, rerun runtime models certify, and compare guardrail and reputation fields with the baseline. The shipped update --version <VERSION> --flavor <FLAVOR> path performs both forward installs and downgrades; a separate rollback verb is unnecessary.4

A real cross-version canary and rollback, executed here (isolated HOME, generic x86_64-unknown-linux-gnu CLI binary, --flavor cpu, two genuinely different real releases fetched from GitHub, not a same-version reinstall):

$ mesh-llm --version
mesh-llm 0.73.0

$ mesh-llm update --version v0.73.1 --flavor cpu
⬇️ Installing mesh-llm v0.73.0 -> v0.73.1 (cpu)...
🔎 Detecting host runtime profile
⬇️  Downloading native runtime meshllm-native-runtime-linux-x86_64-cpu
✅ Installed meshllm-native-runtime-linux-x86_64-cpu
   version: 0.73.1
   flavor: cpu
✅ Updated to v0.73.1

$ mesh-llm --version
mesh-llm 0.73.1

$ mesh-llm update --version v0.73.0 --flavor cpu
⬇️ Downgrading mesh-llm v0.73.1 -> v0.73.0 (cpu)...
🔎 Detecting host runtime profile
⬇️  Downloading native runtime meshllm-native-runtime-linux-x86_64-cpu
✅ Installed meshllm-native-runtime-linux-x86_64-cpu
   version: 0.73.0
   flavor: cpu
✅ Pruned 1 native runtime cache version(s)
   removed: .../native-runtimes/0.73.1
✅ Updated to v0.73.0

$ mesh-llm --version
mesh-llm 0.73.0

Two things the real transcript shows that the command alone does not: the CLI itself detects direction (Installing for a forward canary versus Downgrading for a rollback, exact same update --version invocation shape either way, per the "no separate rollback verb" claim above), and each update prunes the native-runtime cache down to the currently-selected version rather than accumulating every version ever installed. This validates the version-swap mechanism itself, end to end, on two real published releases; a live-GPU node still needs its own commissioning transcript for the models certify/guardrail/reputation comparison step, since that requires an actual loaded model and running mesh, not just the binary swap.

Uninstall / decommission a node

mesh-llm uninstall --dry-run     # print the cleanup plan first, always
mesh-llm uninstall --yes         # stops tracked processes, removes the service, keeps ~/.mesh-llm config
mesh-llm uninstall --yes --purge-config   # also removes ~/.mesh-llm identity/config data

Default uninstall preserves ~/.mesh-llm (including node identity) unless you pass --purge-config; decide deliberately whether a decommissioned machine should keep or lose its identity before it might return to service.4

How to maintain it

  • Certificate lifetime. The project's identity design recommends node certificates that live for hours to days, not indefinitely, with automatic renewal in the final 24 to 36 hours of a 7-day default lifetime; shorten it further for more sensitive deployments.9 Certificate renewal itself is meant to be automatic; node-key and owner-key rotation are meant to stay manual, explicit operator actions.
  • Prepare both incident-response playbooks. For a compromised node, run auth revoke-node for its certificate or endpoint ID in every enforcing trust store, isolate the host, and rotate its node key before return to service. For a compromised owner keystore, run auth revoke-owner throughout the mesh, create a new owner identity, and re-enroll the trusted machines. Treat every host that stored the old owner key as suspect.9
  • Watch guardrail mode after every upgrade or config change. A mode can silently reset to disabled; re-check /api/status rather than assuming a prior runtime guardrails --mode call persisted.
  • Watch node churn and reputation counters, not just uptime. A rising penalized_targets/routes_penalized count on a node that "looks up" is an earlier signal than a user-visible latency complaint.
  • A stage failure replans the whole topology; there is no mid-request resume. In-flight requests on a failed stage fail outright, the mesh withdraws the routable target and rebuilds from remaining eligible peers.10 Plan client-side retry for this, do not expect the mesh to resume a request transparently.

How to run it in production: posture checklist

Deployment Minimum posture
Public mesh (--publish, --auto) Non-sensitive prompts only; do not run this recipe's identity/trust steps as a substitute for treating it as a demo/community service.
Private mesh, machines you own The realistic production floor: owner identity (auth init), a non-off --trust-policy, invite tokens kept off public channels, guardrails at metrics or enforce, supervised processes, readiness gating before routing traffic, and models certify before and after every change.
Cross-datacentre, machines you own All of the above, plus: validate the topology in the Docker WAN lab against your actual measured inter-region RTT before rollout, and size stage count from the network-floor arithmetic in the conceptual page, not from what merely "works" in a low-latency test.

Peer artifact transfer (fetching missing layer-package fragments from the coordinating peer) is disabled by default on public meshes; enable it only inside a trust boundary via MESH_LLM_ARTIFACT_TRANSFER=trusted.5

Failure modes

  • The installer's GPU probe selects a build the host cannot load. Confirmed by direct reproduction: on a GPU-less host that merely had the CUDA toolkit installed, probe_nvidia (which accepts nvidia-smi, nvcc, /dev/nvidiactl, or /proc/driver/nvidia/gpus as evidence) selected the CUDA build, which then failed to load with libnccl.so.2: cannot open shared object file. mesh-llm setup and mesh-llm update cannot rescue this state, because they live inside the binary that will not load; reinstall from the correct flavor's release asset (transcript above) and verify mesh-llm --version runs clean before trusting any later step.
  • Treating the coordinator's peer list as a liveness signal. Heartbeat eviction is a two-strike process; in the executed run a SIGKILLed idle peer stayed in /api/status's peers list for 4 minutes 40 seconds, with reputation counters untouched the whole time. Health-check every node's own /api/status directly.
  • models certify assumes a resolvable model reference, not a raw --gguf path's ad hoc identifier. Confirmed by direct reproduction: certifying a model served via --gguf failed with no layer package found for "local-gguf/sha256-<hash>". Serve from a Hub reference or a registered layer package specifically when you need step 8's certification gate to work, not a bare local file path.
  • models certify panics, in both modes, against a genuine layer package on this pinned CPU release binary. Confirmed by direct reproduction against hf://meshllm/Qwen3-0.6B-Q4_K_M-layers, a real, resolvable, currently-serving layer package, not a raw --gguf path: both --package-only and --api-base crash with MeshLLM native runtime library has not been loaded (crates/skippy-ffi/src/lib.rs:693) instead of returning a certification report. Reproduced in a fresh isolated HOME and this host's real, already-populated native-runtime cache alike. Treat models certify as unverified for your deployment until you have run it to a real pass, not just a clean exit.
  • A release-attestation-required mesh enforces its policy against the genesis node too, and a rejected peer fails silently. Confirmed by direct reproduction: starting the seed itself from a binary whose embedded signer key is not in the mesh's own allowed_signer_keys gets the seed's own gossip rejected. A joining peer in the same situation reaches "node_state":"client" and prints api ready locally, with no error, while never actually appearing in the seed's peers[]. Check the seed's peer list for the peer's node ID, not the peer's own status output, to detect this.
  • Process is not restarted after a crash or reboot. A detached tmux session survives a normal SSH disconnect but provides no restart policy. Use supervisorctl or mesh-llm setup --service for a production process; reserve tmux new -d for short-lived diagnostics.1
  • --headless mistaken for "run quietly in the background." It only disables the embedded web console; it does nothing for process supervision and is a recurring operator mistake.1
  • Docker/multi-NIC bridge address collision. iroh advertises a bridge address like 172.17.0.1; if every host shares it, peers race the wrong local path. Pin --bind-ip on every node.3
  • Trusting /v1/models the instant the port opens. Model load after a mesh join can take minutes; poll GPU VRAM usage and the stage-readiness endpoint, do not conclude "not serving" from an early check, and do not route production traffic before readiness confirms.15
  • Assuming release attestation proves the remote process is safe to trust. It proves the binary was signed by a trusted release signer, nothing about the running process, host, or hardware.3
  • Assuming the prose CLI guide is exhaustive. The v0.73.1 parser ships auth trust add/list/remove and auth revoke-node/auth revoke-owner, although docs/CLI.md does not enumerate every subcommand. Exercise the installed binary's --help and a disposable trust store before relying on the incident procedure.4
  • Leaving guardrails at the disabled default and assuming reliability enforcement is on. It is an explicit opt-in at both serve time and via runtime guardrails; a request-level flag cannot upgrade a server that started disabled.4
  • Confusing package certification with runtime certification. models certify --package-only proves artifact integrity, not that a live endpoint actually serves correct completions; use --api-base for a real production readiness gate.4
  • Assuming a Prometheus /metrics endpoint exists on the production serve path. It does not appear in this project's documented surface; metrics-server/OTLP is the shipped telemetry path, and it is lab/benchmark infrastructure you run alongside the deployment, not a flag on serve.

References

  • Mesh LLM (Apache-2.0), as of commit 2c2e808be11f (2026-07-15); repo re-checked 2026-07-17 (HEAD 8e5bb12, v0.73.1 still the latest tag, tag commit 43103c5): https://github.com/Mesh-LLM/mesh-llm
  • v0.73.1 release assets (per-flavor tarballs and .sha256 sidecars; the executed transcripts installed mesh-llm-v0.73.1-x86_64-unknown-linux-gnu.tar.gz): https://github.com/Mesh-LLM/mesh-llm/releases/tag/v0.73.1
  • install.sh at the pinned commit (probe_nvidia lines 249 to 255, deprecated MESH_LLM_INSTALL_FLAVOR, asset_name flavor naming): https://github.com/Mesh-LLM/mesh-llm/blob/2c2e808be11fd417e3b864db907db92151f13ee2/install.sh
  • Config validator (owner-control bind/advertise_addr same-port rule reproduced above): https://github.com/Mesh-LLM/mesh-llm/blob/2c2e808be11fd417e3b864db907db92151f13ee2/crates/mesh-llm-config/src/validate.rs
  • Management API endpoint list (/api/status, /api/runtime/control-bootstrap, /api/diagnostics/split-readiness): https://github.com/Mesh-LLM/mesh-llm/blob/2c2e808be11fd417e3b864db907db92151f13ee2/crates/mesh-llm-host-runtime/src/api/mod.rs
  • Setup service module (systemd user unit path, hard Windows error): https://github.com/Mesh-LLM/mesh-llm/blob/2c2e808be11fd417e3b864db907db92151f13ee2/crates/mesh-llm-commands/src/setup/service.rs
  • xtask release-attestation tool (generate-keypair/stamp/inspect subcommands, used above to create a throwaway signer and stamp/inspect binaries): https://github.com/Mesh-LLM/mesh-llm/blob/2c2e808be11fd417e3b864db907db92151f13ee2/tools/xtask/src/main.rs
  • metrics-server crate (OTLP/gRPC ingest over tonic, SQLite storage via rusqlite, /v1/runs/{run_id}/status and report.json): https://github.com/Mesh-LLM/mesh-llm/blob/2c2e808be11fd417e3b864db907db92151f13ee2/crates/metrics-server/src/store.rs
  • CLI User Guide (all serve/auth/models/runtime guardrails/uninstall/update commands and switches): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/CLI.md
  • Mesh Workflows (mesh types, release attestation, bootstrap tokens, owner/trust flags, multi-interface binding): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/MESHES.md
  • Skippy stage-split operator guide (readiness, immutable layer-package refs): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/SKIPPY_SPLITS.md
  • Benchmarks and guardrail evidence scaffolding (--mesh-guardrails, runtime guardrails, corpus runner): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/BENCHMARKS.md
  • Local Node Reputation (process-local routing health, explicit non-goals): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/NODE_REP.md
  • Mesh setup installer spec (install.sh/mesh-llm setup, service setup boundary, checksum verification): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/specs/mesh-setup-installer.md
  • deploy-linux-gpu operator skill (install, supervisor config, systemd, verification, troubleshooting table): https://github.com/Mesh-LLM/mesh-llm/blob/main/.skills/deploy-linux-gpu/SKILL.md
  • Skippy WAN Docker Lab (tc netem shaping, metrics-server, reproducible pre-production test harness): https://github.com/Mesh-LLM/mesh-llm/blob/main/docker/skippy-wan-lab/README.md
  • Node Owner Identity design proposal (owner/node identity split, certificate lifecycle, status: proposal at time of review): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/design/NODE_OWNER_IDENTITY.md
  • Identity Incident Response draft runbook (node-compromise and owner-compromise response flows): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/design/IDENTITY_INCIDENT_RESPONSE.md
  • Usage Guide (config.toml reference, owner-control plane bootstrap/get-config/apply-config, failure-mode table): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/USAGE.md
  • VRAM accounting spec (rated/system-reported/reserved/allocatable definitions): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/specs/vram-accounting.md

Related: Mesh LLM and Skippy stage splits · Cross-WAN model-parallel inference · P2P transport for decentralized inference · Remote GPU verification (rented hardware) · GPU confidential computing and attestation · Non-colocated inference: which pattern? · Glossary

  1. deploy-linux-gpu SKILL.md: installer auto-detects GPU/CUDA; binary at ~/.local/bin/mesh-llm; supervisor config example; mesh-llm setup --service installs a systemd user unit at ~/.config/systemd/user/mesh-llm.service; sudo loginctl enable-linger "$USER" for reboot survival; model load can take minutes, poll before concluding failure; --headless only disables the web console. ↩↩↩↩↩↩

  2. docs/specs/mesh-setup-installer.md: two-stage install.sh then mesh-llm setup flow; checksum verification is best-effort by default, MESH_LLM_REQUIRE_CHECKSUM=1 fails closed; --service/--no-service/--skip-runtime/--verbose setup switches; service setup supported for Linux systemd user units and macOS launchd, a hard error on Windows. ↩↩

  3. docs/MESHES.md: unrestricted vs. private-invite-token vs. requirement-aware mesh types; changing mesh requirements derives a new mesh id; release attestation is build provenance, not runtime attestation; --bind-ip/--bind-port for multi-interface/Docker hosts; owner-key/trust-policy/trust-owner private-mesh example; auth trust add/list/remove listed as related commands; peer artifact transfer disabled by default on public meshes, MESH_LLM_ARTIFACT_TRANSFER=trusted to enable inside a trust boundary. ↩↩↩↩↩

  4. Mesh LLM commit 2c2e808be11fd417e3b864db907db92151f13ee2: docs/CLI.md documents models certify, guardrails, update --version <VERSION> --flavor <FLAVOR>, and the Cargo/xtask release-attestation inspection path. crates/mesh-llm-cli/src/parser.rs additionally defines auth trust add/list/remove, auth revoke-node, and auth revoke-owner; crates/mesh-llm-commands/src/auth.rs implements them against the local trust store. https://github.com/Mesh-LLM/mesh-llm/blob/2c2e808be11fd417e3b864db907db92151f13ee2/crates/mesh-llm-cli/src/parser.rs ↩↩↩↩↩↩↩↩↩↩↩

  5. docs/SKIPPY_SPLITS.md: stage 0 not routable until all stages ready; prefer immutable hf://namespace/repo@revision package refs for production split runs; /api/status and mesh-llm doctor split readiness checks; peer artifact transfer is disabled by default on public meshes, enabled via MESH_LLM_ARTIFACT_TRANSFER=trusted for trusted or lab deployments. ↩↩↩↩

  6. docs/BENCHMARKS.md: guardrail evidence scaffolding, --mesh-guardrails metrics/runtime guardrails --mode metrics and verifying via /api/status. ↩

  7. docs/NODE_REP.md: routing_affinity.target_reputation is process-local routing health only, explicitly not gossiped, not persisted, not proof of peer identity or model honesty. ↩

  8. docker/skippy-wan-lab/README.md: four-stage CPU-backed Docker lab with tc netem shaping and metrics-server on :18080/:14317; up.sh enforces the model-package invariant; WAN_ENABLE=0 disables shaping without changing topology. ↩↩

  9. docs/design/IDENTITY_INCIDENT_RESPONSE.md and docs/design/NODE_OWNER_IDENTITY.md (design status at review time): node-compromise versus owner-compromise response flows and a recommended 7-day certificate lifetime with renewal in the final 24 to 36 hours. The corresponding local revocation commands are implemented in the v0.73.1 parser and command handler, even though the prose CLI guide does not enumerate them. ↩↩

  10. Mesh LLM and Skippy stage splits: a stage failure replans the whole topology with no mid-request resume; peer artifact transfer disabled by default on public meshes. ↩

  11. docs/USAGE.md: [telemetry]/[owner_control] config.toml sections; config precedence order and the request-defaults-merge-only-at-the-frontend rule; owner-control as a separate mesh-llm-control/1 protocol lane distinct from the public mesh plane; runtime bootstrap/get-config/apply-config commands and REST equivalents; the ControlEndpointRequired/ControlUnsupported/ControlUnavailable/Unauthorized/RevisionConflict/LegacyJsonUnsupported failure table; enabled=false bootstrap payload as an explicit prerequisite report. ↩↩↩↩↩

  12. docs/specs/vram-accounting.md: rated vs. system-reported vs. reserved vs. allocatable VRAM definitions; internal fit decisions use system_reported_bytes - reserved_bytes where a true reserved value is available; user-facing labels show the rated capacity class. ↩