Markdown

Recipe: Mesh LLM production deployment¶

Scope: the operational bring-up this KB's conceptual page does not cover: installing and supervising mesh-llm so it survives disconnects and crashes, securing a private mesh with owner identity and a trust policy, gating readiness before trusting the endpoint, turning on reliability guardrails, monitoring a running mesh, reproducing WAN conditions in a local test lab before a real cross-region rollout, and upgrading safely. Read Mesh LLM and Skippy stage splits first: it has the performance model (when splitting helps versus hurts), the trust-boundary analysis, and the architecture this recipe assumes. This page assumes you already decided splitting is the right call and now need to run it as more than a demo.

Every command below is transcribed from the project's own current documentation (docs/CLI.md, docs/USAGE.md, docs/MESHES.md, docs/BENCHMARKS.md, docs/NODE_REP.md, docs/specs/mesh-setup-installer.md, docs/specs/vram-accounting.md, .skills/deploy-linux-gpu/SKILL.md, docker/skippy-wan-lab/README.md, and the identity design docs under docs/design/), at commit 2c2e808be11f (2026-07-15T03:39:14Z), Apache-2.0, 1,924 commits, actively developed; the latest tagged release is v0.73.1, so re-check flag names against your installed version if it predates this commit. None of it has been executed against a live multi-node mesh in this KB. There is no associated academic paper and no independent third-party production writeup, this is the project's own documentation, reviewed for internal consistency, not corroborated externally. Mesh LLM moves fast; re-verify every flag against mesh-llm --help / mesh-llm <command> --help before depending on it, and treat anything marked "design-stage" below as unconfirmed until you have checked it against your installed version yourself.

What it is¶

A production bring-up is five concerns the conceptual page treats as given: the process must survive an SSH disconnect or a crash, a private/business mesh needs an identity and trust policy beyond a bare invite token, an endpoint must not serve traffic before every stage it depends on is actually ready, an operator needs a way to see the mesh is healthy without SSHing into every box, and an upgrade must not silently break a running deployment. Mesh LLM ships real, documented mechanisms for most of this; this recipe assembles them into one path instead of leaving each as an isolated doc page.

Why it matters¶

Skipping any one of these turns a working proof of concept into an outage generator, and each has a specific, named failure mode later in this page: a process backgrounded with & or bare tmux dies the moment the SSH session that started it closes; a private mesh with only an invite token has no way to answer "does this node belong to us" if the token leaks; a client that trusts /v1/models the instant the API port opens can route a request into a stage that has not finished loading; a guardrail mode left at its disabled default provides none of the reliability enforcement the project ships; and an upgrade applied to every node at once with no certification step first has no rollback plan if the new version regresses a model family's correctness.

When to use it (and when not)¶

Use this recipe once Mesh LLM and Skippy stage splits has told you splitting is warranted: capacity is scattered, no single node holds the model, or a node is thrashing against its memory limit. Do not use this recipe as a reason to split a model that already fits on one machine; re-read that page's network-floor arithmetic first. If a full replica fits everywhere you have capacity, you do not need any of this.

Architecture¶

flowchart TB
  subgraph Owner["Operator: owner identity"]
    OK["mesh-llm auth init<br/>owner keystore"]
  end
  OK -->|"owner-key, node-label"| Seed["Seed / coordinator node<br/>systemd/supervisor supervised<br/>mesh-llm serve --split --model &lt;layer-package&gt;"]
  Seed -->|"invite / bootstrap token"| W1["Worker node 1<br/>supervised, --join &lt;token&gt;"]
  Seed -->|"invite / bootstrap token"| W2["Worker node 2<br/>supervised, --join &lt;token&gt;"]
  Seed --> Ready{"All stages ready?<br/>/api/status"}
  Ready -->|"no"| Wait["Do not route traffic yet"]
  Ready -->|"yes"| LB["Client traffic:<br/>http://seed:9337/v1"]
  Seed -.->|"OTLP telemetry"| Metrics["metrics-server<br/>(from the Docker WAN lab,<br/>self-hosted, not built into serve)"]
  Seed -.->|"/api/status"| Ops["Operator: node reputation,<br/>guardrail mode, readiness"]

How: implement, integrate, maintain¶

1. Install and verify the binary on every node¶

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | sh
~/.local/bin/mesh-llm --version
nvidia-smi --query-gpu=name,memory.total,compute_cap,driver_version --format=csv,noheader

The installer auto-detects the GPU and CUDA major version and pulls the matching build. ~/.local/bin is not always on a non-interactive SSH PATH; use the full path or a login shell (ssh -tt host 'bash -lc "..."').¹

The public two-stage install/setup flow is install.sh then mesh-llm setup; the bootstrap script owns only fetching and installing the executable (with checksum verification, best-effort by default, fail-closed only with MESH_LLM_REQUIRE_CHECKSUM=1), and mesh-llm setup owns everything after that: native runtime install, and service registration.²

mesh-llm setup --service    # requests background service setup (systemd user unit on Linux)
mesh-llm setup --verbose    # print exact service paths, commands, log locations

--service is supported for Linux systemd user units and macOS launchd agents; it is a hard, explicit error on Windows.² The exact unit file is written and managed by the installer, not hand-authored here; use --verbose to print its actual path and status on your machine rather than assuming a path.

2. Decide your mesh's security posture before the first `serve`¶

Three distinct mesh types exist, and the choice is made at creation time and cannot be changed in place, changing any of these later derives a brand-new mesh with a new mesh id:³

Mesh type	Who can join	Confirmed shipped?
Unrestricted (legacy)	Anyone with the plain invite token	Yes
Private, invite-token	Anyone with the token; still no identity binding	Yes
Requirement-aware, release-attestation-required	Only peers presenting a trusted release attestation, via a signed bootstrap token	Yes, creation and join flow; see the caveat below on verifying your own binary

# Unrestricted / legacy private mesh
mesh-llm serve --model Qwen3-8B-Q4_K_M

# Requirement-aware public mesh requiring a trusted release signature
mesh-llm serve --model Qwen3-8B-Q4_K_M --publish \
  --require-release-attestation \
  --release-signer-key ed25519:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
  --owner-key ~/.mesh-llm/owner-keystore.json \
  --owner-required \
  --trust-policy require-owned \
  --node-label lab-a

# Join via the signed bootstrap token this prints
mesh-llm serve --join <signed-bootstrap-token>

Two things the project's own docs are explicit about, and that this recipe will not soften: release attestation is build provenance, not runtime attestation. It proves a peer's binary was published by a trusted release signer; it does not prove the remote process is running unmodified code or that the host has not been tampered with.³ And verifying your own packaged binary's attestation is not a plain mesh-llm subcommand; it requires a full source checkout and Cargo: cargo run -p xtask -- release-attestation inspect --binary <path> --public-key-file <release-signing-public-key.json>.⁴ missing is the normal, expected result for an unstamped local/dev build; only a packaged release binary can report valid or invalid.

Initialize owner identity once per operator, then attach it to every node you run:

mesh-llm auth init                      # generates/saves the owner keypair
mesh-llm auth status                    # shows identity/keystore status

mesh-llm serve --model Qwen3-14B \
  --owner-key ~/.mesh-llm/owner-keystore.json \
  --node-label studio \
  --trust-policy allowlist \
  --trust-owner <owner-id>

auth init/auth status and the --owner-key/--owner-required/--node-label/--trust-policy/--trust-owner runtime flags are documented in the current CLI reference.⁴ auth sign-node, auth renew-node, and auth rotate-node also appear with documented switches in that same reference.⁴ Treat auth trust add/list/remove and auth revoke-node/auth revoke-owner differently: they appear in the project's mesh-workflow docs and its identity design/incident-response drafts, but not in the CLI command reference itself, which is the practical, current-tense guide to what actually ships.³⁹ Confirm with mesh-llm auth --help on your installed version before building an incident-response process around them; if they are not yet present, your compromise-response plan needs a fallback (at minimum: stop the node, block its endpoint at the network layer, and rotate the owner keystore manually).

3. Bring up the seed/coordinator node under supervision¶

A bare mesh-llm serve & or a tmux session dies the moment the SSH connection or the tmux server goes away. On images that ship supervisor (common on rented GPU providers), use it; it restarts on crash and persists across disconnects:¹

cat > /etc/supervisor/conf.d/mesh-llm.conf <<'EOF'
[program:mesh-llm]
command=/root/.local/bin/mesh-llm serve --model unsloth/Qwen3.6-27B-GGUF:UD-Q4_K_XL --auto
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=30
stdout_logfile=/var/log/mesh-llm.log
stderr_logfile=/var/log/mesh-llm.log
environment=HOME="/root"
EOF
supervisorctl reread && supervisorctl update && supervisorctl start mesh-llm
supervisorctl status mesh-llm

Where there is no supervisor, prefer mesh-llm setup --service (systemd user unit / launchd agent, step 1) over ad hoc backgrounding. sudo loginctl enable-linger "$USER" makes a systemd user unit survive reboot before the user logs in.¹

For a split deployment, the seed node becomes the coordinator by starting the mesh and printing the invite/bootstrap token; pin an immutable layer-package revision for anything you care about, only immutable refs are eligible for peer artifact transfer at all:⁵

mesh-llm serve \
  --model hf://meshllm/Qwen3-235B-A22B-UD-Q4_K_XL-layers@<revision> \
  --split \
  --bind-ip 10.1.2.3 --bind-port 47916

4. Bring up worker nodes and join¶

mesh-llm serve --split --join <token> --model hf://meshllm/Qwen3-235B-A22B-UD-Q4_K_XL-layers@<revision> \
  --bind-ip 10.1.2.4 --bind-port 47916

On multi-interface Linux hosts, especially docker run --network host systems, iroh (the mesh's QUIC transport) can discover and advertise a Docker/CNI bridge address such as 172.17.0.1. If every host has the same bridge address, peers race the wrong local path; --bind-ip on every node avoids this.³ --bind-port, paired with --bind-ip, gives you a fixed UDP port for firewall/NAT rules instead of an ephemeral one.⁴

5. Gate on readiness before trusting the endpoint¶

Stage 0 is not routable until every required downstream stage reports ready; a loaded runtime is not a routable one.⁵

curl -s http://localhost:3131/api/status | jq '{state:.node_state, stages:.runtime.stages}'
mesh-llm doctor split --model-ref meshllm/Qwen3-235B-A22B-UD-Q4_K_XL-layers --port 3131

Do not put a load balancer's health check on the API port opening; put it on this stage-readiness check, or on models certify (step 8).

6. Turn on reliability guardrails and confirm they are actually active¶

Mesh guardrails default to disabled. Enabling them is a separate, explicit step, both at server start and on an already-running node:⁶⁴

# At server start:
mesh-llm serve --model MiniMax-M2.5-Q4_K_M --mesh-guardrails metrics   # or: enforce

# Or switch a running node without restart:
mesh-llm runtime guardrails --mode metrics --port 3131

# Always verify the posture actually took, do not assume the flag applied:
curl -s localhost:3131/api/status | jq '.runtime.openai_guardrails'

disabled, metrics, and enforce are the three documented modes; request-level mesh_guardrails flags on an individual API call cannot upgrade a server that started disabled, so the server-side posture is the one that matters for a production gate.⁴

7. Set production config: telemetry and the owner-control plane¶

~/.mesh-llm/config.toml is where a production node's non-default behavior should live, rather than accumulating as ad hoc CLI flags. Two sections matter specifically for production operation:¹¹

# ~/.mesh-llm/config.toml
version = 1

[telemetry]
enabled  = true
endpoint = "http://localhost:4317"       # your OTLP collector, not a Mesh LLM-hosted service

[owner_control]
bind           = "0.0.0.0:7447"          # owner-control QUIC listen address
advertise_addr = "203.0.113.10:18443"    # address advertised to authorized operator clients

Config precedence, highest to lowest: an explicit request field, then a per-model [[models]] entry, then [defaults.*], then family/topology policy, then built-in runtime defaults. Request-level defaults are merged only at the OpenAI frontend boundary when a field is absent or null; they never reach StageConfig, protobuf, or lower runtime layers. Editing this file affects future starts or reloads, not an already-running session.¹¹

Owner-control is a separate protocol lane (mesh-llm-control/1) from the public mesh plane used for join, gossip, routing, and inference; it exists specifically for config and inventory mutation by an authorized operator, and it is not required for ordinary mesh participation.¹¹ A new client needs an explicit bootstrap endpoint token before it can use it:

# Read the local bootstrap policy
mesh-llm runtime bootstrap --port 3131 --json
curl -s localhost:3131/api/runtime/control-bootstrap | jq .

# Read and apply config through owner-control, with optimistic-concurrency revisioning
mesh-llm runtime get-config --port 3131 --endpoint '<control-endpoint>' --json
mesh-llm runtime apply-config \
  --port 3131 \
  --endpoint '<control-endpoint>' \
  --expected-revision 7 \
  --config /absolute/path/to/config.toml \
  --json

A client with no explicit endpoint gets ControlEndpointRequired, by design, there is no silent fallback to a legacy config path.¹¹ The other failure codes worth knowing before an incident, not during one:

Error	Meaning
`ControlUnsupported`	Target does not speak `mesh-llm-control/1` at that endpoint
`ControlUnavailable`	Endpoint token, listener, network path, or local owner-key unlock failed
`Unauthorized`	The same-owner handshake failed, both sides must share an owner identity
`RevisionConflict`	`apply-config` used a stale `--expected-revision`; re-read and retry
`LegacyJsonUnsupported`	An old mesh-plane config frame hit the new owner-control lane

If the local bootstrap payload reports enabled=false, that means a signed same-owner keystore is required before owner-control requests can succeed on that machine, a prerequisite report, not a silent pass.¹¹

Sizing note. Before pinning fit_target_mib/gpu_layers/tensor_split values in [[models]] entries, understand that Mesh LLM distinguishes rated VRAM (a user-facing capacity class like 32 GB), system-reported VRAM (raw driver-reported bytes, the internal source of truth), reserved VRAM (true driver/runtime-reserved bytes where the platform reports them, live used-memory counters are not this), and allocatable VRAM (system-reported minus reserved). Internal fit decisions use allocatable bytes; only the rated class is meant for user-facing labels.¹² No documented minimum RAM or disk footprint exists anywhere in the project's docs; size hosts from the model/layer-package weight size plus KV budget, not from a published floor.

8. Certify the deployment before calling it production-ready¶

models certify gives a repeatable confidence check distinct from "the process is running." Package-only mode checks artifact integrity and local stage materialization; runtime mode additionally proves the live OpenAI-compatible endpoint by requiring real text-bearing responses from both Chat Completions and the Responses API, not just an HTTP 200:⁴

mesh-llm models certify hf://meshllm/Qwen3-8B-Q4_K_M-layers \
  --package-only --report-out cert-package.json

mesh-llm models certify unsloth/Qwen3-8B-GGUF:Q4_K_M \
  --api-base http://127.0.0.1:9337 --json

Run the runtime form after every deploy and after every upgrade (step 11), not just once at initial bring-up.

9. Monitor continuously¶

The most reliable per-node health signal without extra infrastructure is /api/status itself, polled on a schedule:

curl -s http://localhost:3131/api/status | jq '{
  state: .node_state,
  guardrails: .runtime.openai_guardrails,
  reputation: .routing_affinity.target_reputation
}'

routing_affinity.target_reputation exposes penalized_targets (how many local model+target entries currently carry a health penalty) and routes_penalized (how many route orderings moved a penalized candidate behind cleaner ones). This is process-local routing health, not a mesh-wide trust or correctness signal, do not treat a clean reputation counter as proof a peer is honest; it only means that peer has not recently timed out or returned unavailable to this node.⁷

For real telemetry beyond point-in-time polling, point [telemetry] (step 7) at your own OTLP collector, or use the project's own reproducible reference implementation, metrics-server (OTLP/gRPC ingest, DuckDB storage), demonstrated in the Docker WAN lab (step 10) on ports :18080 (HTTP) and :14317 (OTLP/gRPC):⁸

curl -s http://127.0.0.1:18080/v1/runs/<run-id>/status | jq
curl -s http://127.0.0.1:18080/v1/runs/<run-id>/report.json | jq

Gap to plan around: [telemetry] sends OTLP, there is no documented built-in Prometheus /metrics exporter on the serve path itself. If your production stack standardizes on Prometheus, run an OTLP-to-Prometheus bridge (e.g. an OpenTelemetry Collector with a Prometheus exporter) in front of [telemetry].endpoint, or scrape /api/status with a small custom exporter, rather than assuming a native /metrics endpoint exists.

10. Reproduce WAN conditions in a test lab before a real cross-region rollout¶

Before trusting a topology on an actual multi-region link, the project ships a Docker Compose lab that runs a four-stage CPU-backed chain with real Linux tc netem traffic shaping between stages and metrics-server wired in:⁸

cp docker/skippy-wan-lab/.env.example docker/skippy-wan-lab/.env

# Calibrate shaping parameters from a real target's measured RTT/bandwidth
scripts/skippy-wan-calibrate.sh <target-ip> docker/skippy-wan-lab/.env.link

# Bring the lab up (enforces the model-package invariant; do not use raw `docker compose`)
docker/skippy-wan-lab/up.sh

# Verify and exercise it
curl -s http://127.0.0.1:9337/v1/models | jq
docker/skippy-wan-lab/prompt.sh --max-new-tokens 64

Use WAN_ENABLE=0 to disable shaping without changing the compose topology, useful for isolating whether a problem is the network model or the serving stack. This is the closest thing to a pre-production integration test this project ships; there is no separate CI-integrated load-test harness documented for multi-node WAN behavior beyond this lab.

11. Upgrade and roll back¶

mesh-llm update            # updates the binary and exits

Before rolling an upgrade to every node, certify a canary node first (step 8, runtime mode) and compare its /api/status guardrail and reputation fields against the pre-upgrade baseline. Gap to plan around: no dedicated rollback command was found in the CLI reference or the setup spec. The practical rollback path is reinstalling the previous release's binary at the same path the installer manages and restarting the supervised service; script this yourself and keep the previous release archive on hand rather than assuming an in-place downgrade path exists.

Uninstall / decommission a node¶

mesh-llm uninstall --dry-run     # print the cleanup plan first, always
mesh-llm uninstall --yes         # stops tracked processes, removes the service, keeps ~/.mesh-llm config
mesh-llm uninstall --yes --purge-config   # also removes ~/.mesh-llm identity/config data

Default uninstall preserves ~/.mesh-llm (including node identity) unless you pass --purge-config; decide deliberately whether a decommissioned machine should keep or lose its identity before it might return to service.⁴

How to maintain it¶

Certificate lifetime. The project's identity design recommends node certificates that live for hours to days, not indefinitely, with automatic renewal in the final 24 to 36 hours of a 7-day default lifetime; shorten it further for more sensitive deployments.⁹ Certificate renewal itself is meant to be automatic; node-key and owner-key rotation are meant to stay manual, explicit operator actions.
Have both incident-response playbooks ready before you need them, not after. A single compromised node: revoke that node's certificate (and endpoint ID as an emergency block if certificate tracking is incomplete), then rotate its node key on return to service. A compromised owner keystore: revoke the owner everywhere, generate a new owner identity, and re-enroll every trusted machine under it, treating all machines that ever held the old keystore file as suspect.⁹ Confirm the specific revoke/rotate commands your installed version actually supports first (step 2's caveat).
Watch guardrail mode after every upgrade or config change. A mode can silently reset to disabled; re-check /api/status rather than assuming a prior runtime guardrails --mode call persisted.
Watch node churn and reputation counters, not just uptime. A rising penalized_targets/routes_penalized count on a node that "looks up" is an earlier signal than a user-visible latency complaint.
A stage failure replans the whole topology; there is no mid-request resume. In-flight requests on a failed stage fail outright, the mesh withdraws the routable target and rebuilds from remaining eligible peers.¹⁰ Plan client-side retry for this, do not expect the mesh to resume a request transparently.

Running it in production: posture checklist¶

Deployment	Minimum posture
Public mesh (`--publish`, `--auto`)	Non-sensitive prompts only; do not run this recipe's identity/trust steps as a substitute for treating it as a demo/community service.
Private mesh, machines you own	The realistic production floor: owner identity (`auth init`), a non-`off` `--trust-policy`, invite tokens kept off public channels, guardrails at `metrics` or `enforce`, supervised processes, readiness gating before routing traffic, and `models certify` before and after every change.
Cross-datacentre, machines you own	All of the above, plus: validate the topology in the Docker WAN lab against your actual measured inter-region RTT before rollout, and size stage count from the network-floor arithmetic in the conceptual page, not from what merely "works" in a low-latency test.

Peer artifact transfer (fetching missing layer-package fragments from the coordinating peer) is disabled by default on public meshes; enable it only inside a trust boundary via MESH_LLM_ARTIFACT_TRANSFER=trusted.⁵

Failure modes¶

Process dies when the SSH session closes. Caused by & or a bare tmux session with no supervisor. Fix: supervisorctl/mesh-llm setup --service/tmux new -d, never a bare backgrounded foreground process for anything you plan to leave running.¹
--headless mistaken for "run quietly in the background." It only disables the embedded web console; it does nothing for process supervision and is a recurring operator mistake.¹
Docker/multi-NIC bridge address collision. iroh advertises a bridge address like 172.17.0.1; if every host shares it, peers race the wrong local path. Pin --bind-ip on every node.³
Trusting /v1/models the instant the port opens. Model load after a mesh join can take minutes; poll GPU VRAM usage and the stage-readiness endpoint, do not conclude "not serving" from an early check, and do not route production traffic before readiness confirms.¹⁵
Assuming release attestation proves the remote process is safe to trust. It proves the binary was signed by a trusted release signer, nothing about the running process, host, or hardware.³
Treating design-stage identity commands as shipped. auth trust add/list/remove and auth revoke-node/auth revoke-owner appear in mesh-workflow and incident-response documentation but not in the CLI command reference; verify with --help before an incident is the first time you try them.⁴⁹
Leaving guardrails at the disabled default and assuming reliability enforcement is on. It is an explicit opt-in at both serve time and via runtime guardrails; a request-level flag cannot upgrade a server that started disabled.⁴
Confusing package certification with runtime certification. models certify --package-only proves artifact integrity, not that a live endpoint actually serves correct completions; use --api-base for a real production readiness gate.⁴
Assuming a Prometheus /metrics endpoint exists on the production serve path. It does not appear in this project's documented surface; metrics-server/OTLP is the shipped telemetry path, and it is lab/benchmark infrastructure you run alongside the deployment, not a flag on serve.

References¶

Mesh LLM (Apache-2.0), as of commit 2c2e808be11f (2026-07-15): https://github.com/Mesh-LLM/mesh-llm
CLI User Guide (all serve/auth/models/runtime guardrails/uninstall/update commands and switches): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/CLI.md
Mesh Workflows (mesh types, release attestation, bootstrap tokens, owner/trust flags, multi-interface binding): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/MESHES.md
Skippy stage-split operator guide (readiness, immutable layer-package refs): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/SKIPPY_SPLITS.md
Benchmarks and guardrail evidence scaffolding (--mesh-guardrails, runtime guardrails, corpus runner): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/BENCHMARKS.md
Local Node Reputation (process-local routing health, explicit non-goals): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/NODE_REP.md
Mesh setup installer spec (install.sh/mesh-llm setup, service setup boundary, checksum verification): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/specs/mesh-setup-installer.md
deploy-linux-gpu operator skill (install, supervisor config, systemd, verification, troubleshooting table): https://github.com/Mesh-LLM/mesh-llm/blob/main/.skills/deploy-linux-gpu/SKILL.md
Skippy WAN Docker Lab (tc netem shaping, metrics-server, reproducible pre-production test harness): https://github.com/Mesh-LLM/mesh-llm/blob/main/docker/skippy-wan-lab/README.md
Node Owner Identity design proposal (owner/node identity split, certificate lifecycle, status: proposal at time of review): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/design/NODE_OWNER_IDENTITY.md
Identity Incident Response draft runbook (node-compromise and owner-compromise response flows): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/design/IDENTITY_INCIDENT_RESPONSE.md
Usage Guide (config.toml reference, owner-control plane bootstrap/get-config/apply-config, failure-mode table): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/USAGE.md
VRAM accounting spec (rated/system-reported/reserved/allocatable definitions): https://github.com/Mesh-LLM/mesh-llm/blob/main/docs/specs/vram-accounting.md

deploy-linux-gpu SKILL.md: installer auto-detects GPU/CUDA; binary at ~/.local/bin/mesh-llm; supervisor config example; mesh-llm setup --service installs a systemd user unit at ~/.config/systemd/user/mesh-llm.service; sudo loginctl enable-linger "$USER" for reboot survival; model load can take minutes, poll before concluding failure; --headless only disables the web console. ↩↩↩↩↩↩
docs/specs/mesh-setup-installer.md: two-stage install.sh then mesh-llm setup flow; checksum verification is best-effort by default, MESH_LLM_REQUIRE_CHECKSUM=1 fails closed; --service/--no-service/--skip-runtime/--verbose setup switches; service setup supported for Linux systemd user units and macOS launchd, a hard error on Windows. ↩↩
docs/MESHES.md: unrestricted vs. private-invite-token vs. requirement-aware mesh types; changing mesh requirements derives a new mesh id; release attestation is build provenance, not runtime attestation; --bind-ip/--bind-port for multi-interface/Docker hosts; owner-key/trust-policy/trust-owner private-mesh example; auth trust add/list/remove listed as related commands; peer artifact transfer disabled by default on public meshes, MESH_LLM_ARTIFACT_TRANSFER=trusted to enable inside a trust boundary. ↩↩↩↩↩↩
docs/CLI.md: full auth, models, runtime guardrails, uninstall, update, release-attestation inspect command and switch reference; --mesh-guardrails/runtime guardrails --mode values disabled/metrics/enforce; models certify --package-only vs. --api-base runtime gates; release-attestation inspect requires a Cargo/xtask invocation, not a plain mesh-llm subcommand. ↩↩↩↩↩↩↩↩↩↩↩
docs/SKIPPY_SPLITS.md: stage 0 not routable until all stages ready; prefer immutable hf://namespace/repo@revision package refs for production split runs; /api/status and mesh-llm doctor split readiness checks; peer artifact transfer is disabled by default on public meshes, enabled via MESH_LLM_ARTIFACT_TRANSFER=trusted for trusted or lab deployments. ↩↩↩↩
docs/BENCHMARKS.md: guardrail evidence scaffolding, --mesh-guardrails metrics/runtime guardrails --mode metrics and verifying via /api/status. ↩
docs/NODE_REP.md: routing_affinity.target_reputation is process-local routing health only, explicitly not gossiped, not persisted, not proof of peer identity or model honesty. ↩
docker/skippy-wan-lab/README.md: four-stage CPU-backed Docker lab with tc netem shaping and metrics-server on :18080/:14317; up.sh enforces the model-package invariant; WAN_ENABLE=0 disables shaping without changing topology. ↩↩
docs/design/IDENTITY_INCIDENT_RESPONSE.md and docs/design/NODE_OWNER_IDENTITY.md (status: proposal at time of review): node-compromise vs. owner-compromise response flows; recommended 7-day certificate lifetime with renewal in the final 24-36 hours; auth revoke-node/auth revoke-owner documented here but not in the CLI reference. ↩↩↩↩
Mesh LLM and Skippy stage splits: a stage failure replans the whole topology with no mid-request resume; peer artifact transfer disabled by default on public meshes. ↩
docs/USAGE.md: [telemetry]/[owner_control] config.toml sections; config precedence order and the request-defaults-merge-only-at-the-frontend rule; owner-control as a separate mesh-llm-control/1 protocol lane distinct from the public mesh plane; runtime bootstrap/get-config/apply-config commands and REST equivalents; the ControlEndpointRequired/ControlUnsupported/ControlUnavailable/Unauthorized/RevisionConflict/LegacyJsonUnsupported failure table; enabled=false bootstrap payload as an explicit prerequisite report. ↩↩↩↩↩
docs/specs/vram-accounting.md: rated vs. system-reported vs. reserved vs. allocatable VRAM definitions; internal fit decisions use system_reported_bytes - reserved_bytes where a true reserved value is available; user-facing labels show the rated capacity class. ↩