Skip to content
Markdown

Privacy-aware split inference over a WAN

Scope: the two-party split where a trusted local GPU holds the token-to-vector mapping and an untrusted cloud GPU holds the bulk of the transformer, connected by a public-internet link. This page covers what the embedding boundary structurally guarantees and what it demonstrably does not, why autoregressive decoding turns every token into a network round trip, how speculative decoding is repurposed to amortize that round trip, the measured token-recovery attack against the transmitted activations, and the sizing arithmetic that decides whether the pattern is worth building at all. The throughput-first cousins are cross-WAN model-parallel inference and non-colocated inference patterns; the hardware-attested alternative is GPU confidential computing; the decoding mechanism is speculative decoding.

Primary source: Michael Cunningham, "Privacy-Aware Split Inference with Speculative Decoding for Large Language Models over Wide-Area Networks", arXiv:2602.16760v1 [cs.CR], submitted 2026-02-18. Single-author preprint, no venue stated. Every measurement quoted below is the paper's; no model was run and no attack was reproduced here.

What this page adds. The Python block is executed and asserted (Python 3.11, numpy 2.x). It refits the paper's own published tables to the paper's own stated model and finds that the lookahead projection column cannot be produced by that model with the reported acceptance rate. It also recomputes the inversion-attack table at its stated sample size, and derives the prefill transport cost the paper does not report. Results labelled "derived" are this page's; results labelled with a section number are the paper's.

Read the throughput numbers as single-stream, consumer-hardware measurements. One client, one session, greedy decoding, two models from one architecture family, an RTX 3090 local and a rented A100 or 4090 remote. Nothing here has been shown to hold under batching, tensor parallelism, or sampling.

What it is

The system partitions a decoder-only transformer across a trust boundary rather than a performance boundary. Three components stay on hardware the operator controls:

  • The token embedding, which maps token IDs to vectors.
  • A small number of leading and trailing transformer layers.
  • The final norm and the LM head, which map hidden states back to token logits.

Everything in between runs on rented GPU capacity. The wire carries only intermediate hidden states: for Mistral 7B, one float16 vector of 4,096 dimensions per token position, about 8 KB.

The evaluated configurations put layers 0-1 and 30-31 locally for Mistral 7B (32 layers) and layers 0-1 and 38-39 locally for Mistral NeMo 12B (40 layers), leaving 28 and 36 layers respectively on the cloud side.1 Local VRAM is 2.0 GB for the 7B and 4.9 GB for the 12B, both measured. The 12B needs more not because it is larger in depth but because its vocabulary is 131K against 32K, which quadruples the embedding and LM head.

The KV cache for the cloud-resident layers stays on the cloud, keyed by session ID and expired after five minutes of inactivity. That is the decision that makes the pattern affordable: KV state grows linearly with sequence length and would dominate the link if it were shipped each step. Per generation step, only the new token's hidden state crosses.

Transport is a persistent WebSocket carrying [4-byte header length][JSON header][raw float16 tensor bytes]. The JSON header carries tensor shapes, a prompt-versus-generation flag, the KV-cache crop position, and an optional attention-mask shape.

Why use it

The motivation is regulatory, not economic. An organisation under HIPAA, ITAR, attorney-client privilege, or a financial-services data-residency rule faces a binary choice: send plaintext to a cloud API, or run a smaller model on hardware it owns. The cryptographic middle grounds are priced out of reach for transformer inference, and the paper's own comparison table puts the orders of magnitude plainly.2

Approach Guarantee type Stated overhead
Homomorphic encryption Cryptographic 100-1000x slower
Secure MPC Cryptographic 8-50x slower
TEE-based Hardware-attested 2-3x slower
Split inference Architectural 3-5x slower
Cloud API None 1x

"Architectural privacy" is the paper's own term for what the split provides, chosen deliberately to avoid conflation with formal security notions. The structural property is narrow and real: the cloud never receives a representation with a direct algebraic relationship to the vocabulary, because it holds neither the embedding matrix nor the LM head.

The second reason is capacity. Local layers scale with vocabulary and depth-at-the-edges, not with total parameters. The paper's estimate puts a 405B split at roughly 20-36 GB of local VRAM, which a workstation card can hold, while the 405B itself cannot fit on any single local device.

When to use it (and when not)

The pattern only makes sense for models you cannot run locally at all. The paper's own baseline table settles this: an RTX 3090 runs the whole 7B at about 39 tok/s, while the split configuration reaches 13-17 tok/s, or 33-44% of local-only. For any model that fits, splitting is strictly worse on every axis, including privacy, since local-only leaks nothing. The paper states the framing correctly ("the comparison is not split-vs-local-native") but the abstract and title do not carry that qualification, and the entire empirical section is run on models that fit.

The regime where it pays is exactly the regime that is unmeasured. The paper defines an RTT-to-compute ratio, network RTT divided by cloud compute per token, and observes it falling with model size: 4.9x at 7B and 3.8x at 12B measured, 1.6-2.7x at 70B and 0.4-0.8x at 405B estimated. Split overhead approaches its theoretical minimum only when cloud compute rivals RTT, which happens at 405B. The 7B and 12B rows are measured; the 70B and 405B rows are estimates with no run behind them.

Do not use it if the local layers are the published weights. The privacy analysis assumes the cloud lacks the local layer weights. For a public checkpoint used as-is, that assumption is false by download. The paper is explicit that production deployments must fine-tune or LoRA-adapt the local layers per organisation, which turns "run Mistral 7B privately" into "run a private variant of Mistral 7B", with all the evaluation and drift consequences that implies.

Do not use it for long-context work without sizing prefill first. Decode is latency-bound and its bandwidth is trivial. Prefill ships the entire prompt's hidden states in one payload, and that payload grows linearly with prompt length and with hidden dimension. The paper measures only 100-token prompts.

Do not treat it as a defence against an active adversary. The threat model is semi-honest. A malicious cloud can corrupt activations to poison outputs, deny service, or fingerprint by timing, and the paper places all of those out of scope.

Architecture

flowchart LR
  subgraph LOCAL["Trusted local device (RTX 3090, 2.0-4.9 GB)"]
    EMB["Token embedding<br/>IDs to vectors"] --> L01["Layers 0-1"]
    LNN["Layers N-2, N-1"] --> HEAD["RMS norm + LM head<br/>logits to token"]
    HEAD -->|"argmax"| EMB
  end
  subgraph NET["Public internet, SSH tunnel + WebSocket"]
    UP["hidden state 8-10 KB<br/>+ JSON shape header<br/>+ explicit attention mask"]
    DOWN["hidden state 8-10 KB"]
  end
  subgraph CLOUD["Untrusted cloud GPU (A100 / 4090)"]
    MID["Layers 2..N-3<br/>bulk compute"] --- KV["Per-session KV cache<br/>DynamicCache, 5 min TTL<br/>crop + relocate on reject"]
  end
  L01 --> UP --> MID
  MID --> DOWN --> LNN

The load-bearing detail is the boundary's asymmetry. The cloud observes layer-1 output on the input side and layer-29 output on the output side, both two layers from a vocabulary-facing matrix. Deepening the split has to be done on both ends or one direction stays exposed.

How to use it

Four decisions determine whether a deployment works, in the order they bite.

1. Pick the provider by network path, not by GPU or geography. This is the paper's most transferable finding and it is an infrastructure result, not a machine-learning one.3

Provider GPU SSH path Measured RTT Throughput
RunPod (US-TX-3) RTX 4090 Direct 80-100 ms 11-17 tok/s
HyperStack A100 80GB PCIe Direct 78-85 ms 8.7-10.9 tok/s
VAST.ai RTX 4090 (Texas) Proxied via Virginia 200-400 ms 1.2 tok/s

Two providers with 4090s in the same state differ by 2-4x in RTT and by more than 10x in throughput, because one routes SSH through proxy servers regardless of pod location. Geographic GPU selection cannot fix a proxied control path. Measure RTT on the actual tunnel before committing to a provider.

2. Budget the round trip honestly. At 78 ms RTT on the 7B, the paper's instrumented per-step breakdown is 77.4 ms network (64%), 26.2 ms local GPU (22%), 15.9 ms cloud GPU (13%), 1.0 ms serialization (1%), for 120.2 ms per step. The 12B is 78.5 / 24.4 / 21.1 / 1.0 for 124.7 ms. Note what dominates second: local compute on a consumer card costs more than cloud compute on an A100, so the "small" local half is not free.

3. Construct attention masks explicitly for every multi-token step. PyTorch's scaled_dot_product_attention with is_causal=True builds a relative causal mask from query and key lengths. With KV-cache entries present from committed tokens, that relative mask lets speculative position i attend to entries it must not see. The paper transmits an explicit mask alongside the hidden states, about 2 KB for typical block sizes:

# Reference template from arXiv:2602.16760 Section 4.4. Not executed here.
attn_mask = torch.full(
    (1, 1, k, kv_len), float('-inf'),
    device='cuda', dtype=torch.float16)
for i in range(k):
    attn_mask[0, 0, i, :committed_len + i + 1] = 0.0

4. Choose the speculation setting from the per-step overhead, not from the acceptance rate. This is where the paper's own ablation and its own projection table disagree, and the executed block below resolves the disagreement.

How to develop with it

Transport engineering carried more of the total speedup than the decoding algorithm did. The paper's optimisation progression runs 1.4 tok/s (HTTP with torch.save and base64), 3.9 (numpy with base64), 5.3 (relaying through the 3090 instead of a Mac), 7.8 (WebSocket with binary float16), 8.7 (adding lookahead), 10.9 (switching to a lower-RTT provider), 13-17 (both). Four of the seven steps are serialization and placement; base64 alone costs a 33% payload expansion plus encode and decode CPU.

The tunnel is not incidental. The local machine sits behind NAT, cloud pods get ephemeral IPs and a fresh SSH port on each start, and SSH keys must be distributed across machines. The paper documents this as "essential systems engineering that is universally required for split inference over public cloud infrastructure and is largely absent from the academic literature", which is a fair characterisation.

Three cloud-side cache operations are required by multi-token verification and are easy to miss when adapting a single-token server: cache cropping to roll back to a checkpoint when a speculative block is rejected, cache relocation when the n-gram window shifts, and custom attention masks as above.

Executed audit of the published tables

import numpy as np

# Section 6.3, "Projected throughput at target RTTs".
RTT = np.array([20.0, 40.0, 60.0, 80.0, 100.0, 120.0, 150.0])
PUB = {"7B seq":  np.array([15.9, 12.1, 9.7, 8.1, 7.0, 6.1, 5.2]),
       "7B LA":   np.array([18.6, 13.6, 10.7, 8.8, 7.5, 6.5, 5.4]),
       "12B seq": np.array([15.1, 11.6, 9.4, 7.9, 6.8, 6.0, 5.1]),
       "12B LA":  np.array([15.2, 11.6, 9.4, 8.0, 6.9, 6.1, 5.2])}

def fit(rtt, tps):
    """Paper's model: tok/s = 1000*a/(rtt+f). Linear in 1/tps."""
    slope, intercept = np.polyfit(rtt, 1.0 / tps, 1)
    return 1.0 / (1000.0 * slope), intercept / slope

a_seq, f_seq_col = fit(RTT, PUB["7B seq"])
a_la, f_la_col = fit(RTT, PUB["7B LA"])
assert abs(f_seq_col - 42.9) < 1.0          # matches the measured 42.9 ms overhead
assert abs(a_seq - 1.0) < 0.02              # sequential accepts exactly 1 token/step
assert f_la_col < f_seq_col - 9.0           # the lookahead column implies LESS overhead
assert abs(a_la - 1.0) < 0.05               # and an acceptance rate of 1.0, not 1.17

# The 12B pair, where the paper says it DID adjust for batch overhead, behaves correctly.
_, f12_seq = fit(RTT, PUB["12B seq"])
_, f12_la = fit(RTT, PUB["12B LA"])
assert abs(f12_seq - 46.3) < 0.5            # matches the measured 46.3 ms overhead
assert f12_la > f12_seq                     # lookahead costs MORE per step, as it should

# Section 6.4 ablation, 7B on the A100 at ~78 ms. Invert the same model for f.
RTT_ABL, NGRAM = 78.0, np.array([3, 4, 5, 6, 7])
TPS = np.array([9.3, 9.2, 8.9, 9.1, 8.3])
ACC = np.array([1.21, 1.23, 1.23, 1.25, 1.25])
overhead = lambda tps, acc: 1000.0 * acc / tps - RTT_ABL
f_seq, f_la = overhead(8.3, 1.00), overhead(TPS, ACC)
assert f_la.min() > f_seq                    # every lookahead setting costs MORE per step
assert f_la[0] < f_la[-1]                    # and bigger n-grams cost more still

# Break-even acceptance a* = (RTT + f_LA) / (RTT + f_seq).
star = lambda r, f: (r + f) / (r + f_seq)
assert 1.14 < star(RTT_ABL, f_la[2]) < 1.16  # n=5 reproduces the paper's stated 1.15-1.20
assert star(20.0, f_la[0]) > star(RTT_ABL, f_la[0])   # a* RISES as the link gets faster

# Re-project the lookahead column with its own measured per-step overhead.
seq = 1000.0 / (RTT + f_seq)
la_honest = 1000.0 * ACC[0] / (RTT + f_la[0])
assert la_honest[0] < PUB["7B LA"][0]        # 16.8 tok/s at 20 ms, not 18.6
gain_pub, gain_honest = PUB["7B LA"] / seq, la_honest / seq
assert gain_pub[0] > gain_pub[-1]            # published: speculation helps MORE on a fast link
assert gain_honest[0] < gain_honest[-1]      # corrected: it helps LESS. The trend reverses.

# Section 5.2 inversion table, 221 test samples.
N = 221
acc_by_depth = {2: 0.588, 4: 0.443, 6: 0.448, 8: 0.348}
counts = {k: round(v * N) for k, v in acc_by_depth.items()}
assert counts == {2: 130, 4: 98, 6: 99, 8: 77}        # every cell is an exact k/221
assert counts[6] - counts[4] == 1                     # the "non-monotonic" cell is ONE sample

def z(p1, p2, n=N):
    return (p1 - p2) / np.sqrt(p1 * (1 - p1) / n + p2 * (1 - p2) / n)

assert z(0.588, 0.348) > 5.0                 # the 59 -> 35 headline is ~5.2 sigma
assert abs(z(0.443, 0.448)) < 0.2            # the 4-vs-6 layer ordering is noise
assert 1.9 < z(0.443, 0.348) < 2.2           # 4 -> 8 is marginal on its own

# Transport. Decode is latency-bound; prefill is bandwidth-bound and linear in prompt length.
for d, kb in ((4096, 8), (5120, 10), (8192, 16), (16384, 32)):
    assert d * 2 / 1024 == kb                # float16 hidden state, matches Section 7.1
decode_mbit = 2 * 4096 * 2 * 8.8 * 8 / 1e6
assert decode_mbit < 1.5                     # ~1.15 Mbit/s at 8.8 tok/s, both directions
prefill_gb = 131072 * 16384 * 2 / 1e9
assert prefill_gb > 4.0                      # 4.29 GB for a 128k prompt on a 405B split
assert 8 * prefill_gb / 1.0 > 30             # 34.4 s of pure transfer at 1 Gbit/s

print("all assertions passed")

Six results come out of that block.

The 7B lookahead projection column is not the paper's own model. Refitting tok/s = 1000a/(RTT+f) to the published sequential column recovers f = 43.0 ms and a = 1.000, matching the independently instrumented 42.9 ms fixed overhead exactly. Refitting the lookahead column recovers f = 32.8 ms and a = 0.991. The lookahead column therefore implies a per-step overhead 10.2 ms lower than sequential and an acceptance rate of 1.0, neither of which the paper reports or claims. The 12B pair is the control: its sequential column recovers 46.3 ms, matching its own instrumented figure, and its lookahead column recovers 48.6 ms, correctly higher. That is the same pair of columns produced by a method the paper footnotes as batch-overhead-adjusted, so the anomaly is confined to the 7B row.

The paper's own ablation says the opposite. Inverting the same model against the Section 6.4 measurements gives sequential 42.5 ms and lookahead 52.1, 55.7, 60.2, 59.4, and 72.6 ms at n = 3 through 7. Every speculative setting costs more per step than sequential, by 9.6 to 30.1 ms, and the cost grows with n-gram size. That is the mechanism the paper itself names ("larger speculative batches increase cloud compute and transfer overhead") and it is why n=3 wins on throughput despite having the lowest acceptance rate.

The break-even acceptance rate rises as the link gets faster. With a* = (RTT + f_LA)/(RTT + f_seq), the n=3 threshold is 1.080 at 78 ms, 1.154 at 20 ms, and 1.203 at 5 ms. The paper's stated 1.15-1.20 break-even at 80 ms actually corresponds to n=5 (1.147), not to the n=3 configuration it recommends. The consequence is a design rule the paper does not state: speculation is a high-latency technique, and its margin shrinks as you succeed at reducing RTT.

The 19 tok/s headline does not survive. Re-projecting the lookahead column with the overhead implied by its own best measured point gives 16.8 tok/s at 20 ms against the published 18.6, a 10.8% overstatement. The published speedup over sequential falls from 1.162x at 20 ms to 1.039x at 150 ms; the corrected speedup rises from 1.048x to 1.152x. The trend reverses. The abstract's "15-19 tok/s at 20ms RTT" should read roughly 15-17. Note that the paper already applies this correction to the 12B column, footnoting that its lookahead projections are "scaled from measured throughput to account for batch overhead", and does not apply it to the 7B column beside it.

The inversion table has one real result and two that are noise. Every cell is an exact multiple of 1/221: 130, 98, 99, and 77 successes. The 58.8% to 34.8% drop from 2 to 8 local layers is about 5.2 sigma and is the finding. The 44.3% to 44.8% "non-monotonicity" at 4 versus 6 local layers is a single test sample, which is what the paper says and is worth restating because the table's four rows read as a trend curve. The 4-to-8 step alone is 2.05 sigma, marginal.

Prefill is the transport constraint nobody sized. Decode moves about 1.15 Mbit/s in both directions at 8.8 tok/s on the 7B, which is nothing. Prefill ships prompt_length x hidden_dim x 2 bytes in one payload: 0.8 MB for the paper's 100-token prompt, 268 MB for a 32k prompt on the 7B, and 4.29 GB for a 128k prompt on the 405B configuration the scaling section projects, or 34.4 seconds of pure transfer on a 1 Gbit/s link before any compute. Time-to-first-token in this architecture is bandwidth-bound and linear in prompt length while decode is latency-bound and constant, and the two want opposite network properties.

On the attack numbers themselves

The attack is a 3-layer MLP (4096 to 2048 to 2048 to 32000) trained on roughly 880 activation-token pairs and tested on 221, recovering the top-1 token from a single activation. Two framing points matter more than the headline.

First, the reported random baseline of 0.003% is uniform guessing over a 32K vocabulary, which nothing resembles. The paper says so and estimates a frequency-prior baseline at 3-5%. Under a Zipf(s=1) proxy over 32,000 types, always predicting the single most common token scores about 9%, so the attack's lift over a trivial baseline is real but smaller than the 20,000x the uniform baseline implies.

Second, 880 training samples cannot cover a 32K vocabulary. The same Zipf proxy puts about 517 distinct types in an 880-token draw and a type-coverage ceiling near 50% for fresh test tokens, below the 58.8% measured. That gap is what an 80/20 split of one 1,100-pair pool produces: train and test come from the same documents, so the measurement is in-domain. The paper's characterisation of the result as a lower bound on vulnerability is right for a different reason than stated, and a defender should not read 58.8% as a portable number.

The load-bearing assumption is weaker than the depth knob. The defence the paper recommends first, deepening the split, moves the attack from 58.8% to 34.8% at roughly 3 ms of added local latency per layer. The defence it treats as a prerequisite, keeping local layers private, is what the whole architecture rests on, and it holds only while the adversary cannot obtain labelled (activation, token) pairs from your local layers. An adversary who can act as a client of the deployment (a public-facing application, a shared tenant, any chosen-plaintext path) collects exactly those pairs against the fine-tuned weights and trains the same MLP. The paper anticipates half of this, hypothesising that random LoRA perturbation would not survive an adaptive attacker who can retrain on perturbed activations, and flags validation as future work. Treat "the cloud lacks our local weights" as an assumption to be enforced operationally, not as a property of the architecture.

How to run it in production

Split symmetrically and deeper than the default. The paper's own recommendation is layers 0-4 and 27-31 local for a 32-layer model, ten local layers protecting five layers of depth on each side, at an estimated 10-15% throughput cost. The measured per-layer cost on a 3090 is about 3 ms against a ~100 ms round trip, so depth is cheap relative to the link. There is no reason to run the 2-layer default in production.

Enforce weight divergence and treat it as a release artifact. If the local layers must differ from the published checkpoint for the privacy story to hold, then the fine-tuned local variant is a versioned model with its own evaluation, its own drift, and its own rollback path. Pin it, hash it, and never let a deployment fall back to stock weights on a cache miss.

Pin the split point in the session handshake and randomise it across sessions if you rely on that. All layers share a hidden dimension, so tensor shape does not reveal the split depth. Activation magnitude distributions may still shift by layer, which the paper flags as an uninvestigated caveat, so randomisation raises attack cost rather than providing a guarantee.

Size the session cache. The cloud holds per-session KV state for the resident layers with a five-minute idle expiry. That is user-derived state on untrusted hardware for the session's lifetime plus the TTL, and it is also the capacity limit: concurrent sessions multiply cloud KV footprint exactly as they would in a normal server. The KV-cache sizing in KV cache management applies unchanged to the cloud half.

Price it before you build it. Deriving from the paper's own figures, an A100 at $1.35/hr delivering 8.8 tok/s single-stream costs about $42.61 per million output tokens in rental alone, before the local GPU, its power, or the operator. A 4090 at $0.59/hr and 15 tok/s costs about $10.93 per million. Both are single-stream numbers and both fall with batching, which the paper does not measure. The comparison that matters is against the compliance alternative, not against a cloud API list price, but the number should be on the page before anyone commits.

Instrument RTT and fixed overhead separately. The decomposition per_step_time = RTT + fixed_overhead is the useful part of the paper's model and it survives the audit above. Track both terms as separate SLIs: RTT regressions mean the network path changed, fixed-overhead regressions mean local or cloud compute changed, and only the second is actionable by engineering. The inference SLOs page covers the alerting shape.

Do not let sampling in without re-measuring. All published acceptance rates are under greedy argmax, which the paper flags as an upper bound for stochastic decoding. Given that the corrected break-even at 20 ms RTT is already 1.154 against a measured 1.21 acceptance, a modest drop under temperature sampling puts lookahead below break-even entirely.

How to maintain it

The output-equivalence result is the one property that makes the system safe to change: under greedy argmax, lookahead produces token-identical output to sequential decoding, verified on 4 prompts per model across both models. That is a structural guarantee, not an empirical one, since both paths compute the same forward pass and take the same argmax. Use it as the regression test: any transport, mask, or cache change that alters the token stream under greedy decoding is a bug, and the check is cheap.

The paper's reported "self-perplexity" difference between sequential and lookahead (1.21 versus 8.48 on the 7B, 1.18 versus 13.29 on the 12B, with max logit differences of 29.0 and 38.3) is float16 accumulation order in batched versus incremental attention, not a behavioural change. Do not gate on logit equality; gate on token equality.

Re-verify the provider network path on every pod restart. Ephemeral IPs, fresh SSH ports, and provider routing changes all move RTT, and RTT is 63-64% of per-step time.

Failure modes

  • Relative causal masking corrupts speculative attention. Using is_causal=True for a multi-token step with a populated KV cache lets speculative positions attend to entries they must not. The failure is silent: output degrades rather than crashing. Build explicit masks.
  • Proxied provider networking silently caps throughput. 200-400 ms RTT from a proxy hop reduces the system to 1.2 tok/s regardless of GPU. Nothing in the provider's documentation surfaces it; only measurement does.
  • Speculation below break-even makes things slower. At ~80 ms RTT the n=5 threshold is 1.147 accepted tokens per step. Creative-text prompts measured 1.06-1.12, below it, and the paper's own category table shows lookahead at 0.91-0.99x sequential on structured and creative content. Speculation is not free to leave on.
  • Public local weights void the privacy claim. If the cloud can download identical copies of the local layers, an attacker reconstructs (activation, token) pairs at will. This is the paper's stated critical assumption and it is an operational obligation, not an architectural property.
  • Shallow splits leak on the output side too. The default protects input tokens with two layers and output tokens with two layers. Deepening only the input half leaves generated text exposed at the same rate.
  • Prefill blows the time-to-first-token budget on long prompts. 268 MB for a 32k prompt on a 7B, 4.29 GB for 128k on a 405B, all in one payload with no chunking. Chunked prefill and prefill-decode disaggregation are named by the paper as unimplemented mitigations; see disaggregated inference.
  • Session KV expiry drops long-running conversations. A five-minute idle TTL on the cloud silently invalidates a session; the client must detect and re-prefill, which is a full-prompt retransmission.
  • Single-stream results do not extrapolate to a fleet. Every number here is one client on one link. Concurrency changes cloud compute, cloud KV footprint, and the fixed-overhead term the projection model treats as constant.

Open questions and validation

  • Nothing above 12B has been run. The RTT-to-compute argument says the pattern improves with scale, and that is the regime where every figure is an estimate. A 70B measurement would either confirm or kill the thesis.
  • One architecture family. Mistral 7B and NeMo 12B are both MistralForCausalLM. Acceptance-rate consistency across those two is weak evidence for architecture-independence.
  • No batched measurement. Both the throughput and the economics change under concurrency, and the fixed-overhead term in the projection model is not constant across batch sizes.
  • The fine-tuning defence is untested. The paper hypothesises that random LoRA perturbation would not stop an adaptive attacker and does not test it. Whether genuinely private local weights resist a chosen-plaintext adversary is the question the whole privacy claim rests on.
  • A sequence-aware attacker was not built. The MLP treats positions independently. An attacker with a language-model prior over token sequences would do better, by an unmeasured margin.

References

Related: Cross-WAN model-parallel inference · Non-colocated inference patterns · Speculative decoding · Speculative decoding economics · GPU confidential computing · KV cache management · Disaggregated inference · P2P transport for decentralized inference · Mesh LLM decentralized serving · Inference SLOs · Security and multitenancy · Overlay mesh networking · Glossary

  1. arXiv:2602.16760v1, Section 3.1. Mistral 7B Instruct v0.3, 32 layers, hidden 4096: embedding, layers 0-1, layers 30-31, RMS norm and LM head local; layers 2-29 cloud. Mistral NeMo 12B Instruct, 40 layers, hidden 5120, 8 KV heads against 32 attention heads, 131K vocabulary: layers 0-1 and 38-39 local, layers 2-37 cloud. Local VRAM measured at 2.0 GB and 4.9 GB respectively, both on a 24 GB RTX 3090. ↩

  2. arXiv:2602.16760v1, Section 5.2, "Comparison of privacy approaches". The overhead figures are the paper's own summary of the cited literature, not its own measurements, and the split-inference row's "3-5x slower" is relative to a cloud API baseline rather than to local-only execution. Against the paper's own local-only 7B baseline of about 39 tok/s, the measured split configuration at 13-17 tok/s is 2.3-3.0x slower. ↩

  3. arXiv:2602.16760v1, Sections 3.4 and 6.6. "VAST.ai routes all SSH traffic through proxy servers in Virginia, regardless of GPU location." The paper adds that this "is not documented in VAST.ai's literature and was discovered only through empirical measurement." Provider behaviour is a 2026-02 observation and should be re-measured before it is relied on. ↩