Skip to content
Markdown

Privacy-aware split inference over a WAN

Scope: the two-party split where a trusted local GPU holds the token-to-vector mapping and an untrusted cloud GPU holds the bulk of the transformer, connected by a public-internet link. This page covers what the embedding boundary structurally guarantees and what it demonstrably does not, why autoregressive decoding turns every token into a network round trip, how speculative decoding is repurposed to amortize that round trip, the measured token-recovery attack against the transmitted activations, and the sizing arithmetic that decides whether the pattern is worth building at all. The throughput-first cousins are cross-WAN model-parallel inference and non-colocated inference patterns; the hardware-attested alternative is GPU confidential computing; the decoding mechanism is speculative decoding.

Primary source: Michael Cunningham, "Privacy-Aware Split Inference with Speculative Decoding for Large Language Models over Wide-Area Networks", arXiv:2602.16760v1 [cs.CR], submitted 2026-02-18. Single-author preprint, no venue stated. Every measurement quoted below is the paper's; no model was run and no attack was reproduced here.

What this page adds. The Python block is executed and asserted (Python 3.11, numpy 2.x). It refits the paper's published tables to its stated model and finds that the 7B lookahead projection cannot be produced with the reported acceptance rate. It reconstructs the inversion-table counts while showing why paired significance cannot be recovered from the published marginals, and derives transport limits from the implementation. Results labelled "derived" are this page's; results labelled with a section number are the paper's.

Read the throughput numbers as single-stream, consumer-hardware measurements. One client, one session, greedy decoding, two models from one architecture family, an RTX 3090 local and a rented A100 or 4090 remote. Nothing here has been shown to hold under batching, tensor parallelism, or sampling.

What it is

The system partitions a decoder-only transformer across a trust boundary rather than a performance boundary. Three components stay on hardware the operator controls:

  • The token embedding, which maps token IDs to vectors.
  • A small number of leading and trailing transformer layers.
  • The final norm and the LM head, which map hidden states back to token logits.

Everything in between runs on rented GPU capacity. The wire does not carry raw token IDs, but it carries more than hidden states: each request contains hidden states, rotary-position tensors, framing metadata, and an optional dense attention mask. One Mistral 7B float16 hidden vector has 4,096 dimensions and occupies about 8 KB.

The evaluated configurations put layers 0-1 and 30-31 locally for Mistral 7B (32 layers) and layers 0-1 and 38-39 locally for Mistral NeMo 12B (40 layers), leaving 28 and 36 layers respectively on the cloud side.1 Local VRAM is 2.0 GB for the 7B and 4.9 GB for the 12B, both measured. The 12B needs more not because it is larger in depth but because its vocabulary is 131K against 32K, which quadruples the embedding and LM head.

The KV cache for the cloud-resident layers stays on the cloud, keyed by session ID and expired after five minutes of inactivity. Shipping that state on every step would be prohibitive because it grows with sequence length. Sequential decoding sends one new hidden state in each direction; lookahead sends a multi-token candidate batch and a dense mask whose width grows with the committed context.

Transport is a persistent WebSocket carrying [4-byte header length][JSON header][raw float16 tensor bytes]. The JSON header carries tensor shapes, a prompt-versus-generation flag, the KV-cache crop position, and an optional attention-mask shape.

Why use it

The motivation is regulatory rather than rental cost. An organisation handling data covered by HIPAA, ITAR, attorney-client privilege, or a financial-services residency rule may reject plaintext cloud APIs yet lack local capacity for the target model. The paper summarises cryptographic and hardware-attested alternatives with the following overheads; these are its literature-derived figures, not measurements from this implementation.2

Approach Guarantee type Stated overhead
Homomorphic encryption Cryptographic 100-1000x slower
Secure MPC Cryptographic 8-50x slower
TEE-based Hardware-attested 2-3x slower
Split inference Architectural 3-5x slower
Cloud API None 1x

"Architectural privacy" is the paper's term for the split, chosen to avoid conflation with formal security. The structural property is narrow: raw token IDs and embedding vectors do not cross the boundary. Hidden states remain input-dependent and demonstrably leak tokens; keeping vocabulary-facing matrices local is not a confidentiality guarantee.

The second reason is capacity. Local layers scale with vocabulary and depth-at-the-edges, not with total parameters. The paper's estimate puts a 405B split at roughly 20-36 GB of local VRAM, which a workstation card can hold, while the full model is far beyond the evaluated RTX 3090's capacity at the paper's assumed precision.

When to use it (and when not)

The strongest fit is a model that cannot run locally. In the paper's non-controlled baseline table, an RTX 3090 runs the whole 7B at about 39 tok/s while the split configuration reaches 13-17 tok/s, or 33-44% of that rate. Local-only execution also has the stronger privacy boundary. Splitting a model that fits can still reduce local VRAM or compute demand, but the page's privacy-and-capacity case is strongest when local-only execution is unavailable. The 7B and 12B experiments do not measure that target regime.

The regime where it pays is exactly the regime that is unmeasured. The paper defines an RTT-to-compute ratio, network RTT divided by cloud compute per token, and observes it falling with model size: 4.9x at 7B and 3.8x at 12B measured, 1.6-2.7x at 70B and 0.4-0.8x at 405B estimated. Split overhead approaches its theoretical minimum only when cloud compute rivals RTT, which happens at 405B. The 7B and 12B rows are measured; the 70B and 405B rows are estimates with no run behind them.

Do not use it if the local layers are the published weights. The privacy analysis assumes the cloud lacks the local layer weights. For a public checkpoint used as-is, that assumption is false by download. The paper is explicit that production deployments must fine-tune or LoRA-adapt the local layers per organisation, which turns "run Mistral 7B privately" into "run a private variant of Mistral 7B", with all the evaluation and drift consequences that implies.

Do not use it for long-context work without sizing both prefill and masks. Prefill sends and returns the prompt-length hidden tensor, while speculative decode transmits a dense mask that grows with committed context. The paper measures 100-token prompts. At the inspected source commit, both WebSocket endpoints cap one message at 20 MiB and prefill is not chunked.

Do not treat it as a defence against an active adversary. The threat model is semi-honest. A malicious cloud can corrupt activations to poison outputs, deny service, or fingerprint by timing, and the paper places all of those out of scope.

Architecture

flowchart LR
  subgraph LOCAL["Trusted local device (RTX 3090, 2.0-4.9 GB)"]
    EMB["Token embedding<br/>IDs to vectors"] --> L01["Layers 0-1"]
    LNN["Layers N-2, N-1"] --> HEAD["RMS norm + LM head<br/>logits to token"]
    HEAD -->|"argmax"| EMB
  end
  subgraph NET["Public internet, SSH tunnel + WebSocket"]
    UP["hidden state 8-10 KB<br/>+ rotary tensors and JSON header<br/>+ optional explicit attention mask"]
    DOWN["hidden state 8-10 KB"]
  end
  subgraph CLOUD["Untrusted cloud GPU (A100 / 4090)"]
    MID["Layers 2..N-3<br/>bulk compute"] --- KV["Per-session KV cache<br/>DynamicCache, 5 min TTL<br/>crop + relocate on reject"]
  end
  L01 --> UP --> MID
  MID --> DOWN --> LNN

The load-bearing detail is the boundary's asymmetry. The cloud observes layer-1 output on the input side and layer-29 output on the output side, both two layers from a vocabulary-facing matrix. Deepening the split has to be done on both ends or one direction stays exposed.

How to use it

Four decisions determine whether a deployment works, in the order they bite.

1. Pick the provider by network path, not by GPU or geography. This is the paper's most transferable finding and it is an infrastructure result, not a machine-learning one.3

Provider GPU SSH path Measured RTT Throughput
RunPod (US-TX-3) RTX 4090 Direct 80-100 ms 11-17 tok/s
HyperStack A100 80GB PCIe Direct 78-85 ms 8.7-10.9 tok/s
VAST.ai RTX 4090 (Texas) Proxied via Virginia 200-400 ms 1.2 tok/s

Two providers with 4090s in the same state differ by 2-4x in RTT and by more than 10x in throughput, because one routes SSH through proxy servers regardless of pod location. Geographic GPU selection cannot fix a proxied control path. Measure RTT on the actual tunnel before committing to a provider.

2. Budget the round trip honestly. At 78 ms RTT on the 7B, the paper's instrumented per-step breakdown is 77.4 ms network (64%), 26.2 ms local GPU (22%), 15.9 ms cloud GPU (13%), 1.0 ms serialization (1%), for 120.2 ms per step. The 12B is 78.5 / 24.4 / 21.1 / 1.0 for 124.7 ms. Note what dominates second: local compute on a consumer card costs more than cloud compute on an A100, so the "small" local half is not free.

3. Construct attention masks explicitly for every multi-token step. PyTorch's scaled_dot_product_attention with is_causal=True builds a relative causal mask from query and key lengths. With KV-cache entries present from committed tokens, that relative mask lets speculative position i attend to entries it must not see. The reference implementation transmits a dense float16 mask of shape [1, 1, query_tokens, total_KV_tokens]; it is small in the paper's short-prompt tests but grows linearly with context:

# Reference template from arXiv:2602.16760 Section 4.4. Not executed here.
attn_mask = torch.full(
    (1, 1, k, kv_len), float('-inf'),
    device='cuda', dtype=torch.float16)
for i in range(k):
    attn_mask[0, 0, i, :committed_len + i + 1] = 0.0

4. Choose the speculation setting from the per-step overhead, not from the acceptance rate. This is where the paper's own ablation and its own projection table disagree, and the executed block below resolves the disagreement.

Do not reach for plain Jacobi decoding. The obvious cheaper alternative to lookahead is vanilla Jacobi decoding: initialise a block of k future positions and iterate the whole block through one forward pass until it converges, committing k tokens for however many round trips convergence takes. The paper measured it and it lost badly, at roughly 1.5 tok/s against about 8 tok/s sequential.5 Two effects compound. Convergence is poor, so a block "rarely commits more than 1-2 tokens per iteration for language modeling", and every iteration still costs a full round trip while shipping a larger tensor than a single hidden state. Lookahead wins because it harvests n-gram candidates from the Jacobi iteration trajectories rather than waiting for the block to converge. This is a useful negative result to inherit: over a WAN the naive parallel-decoding method is about five times slower than doing nothing clever, so the n-gram machinery is not optional complexity.

How to develop with it

Transport engineering carried more of the total speedup than the decoding algorithm did. The paper's optimisation progression runs 1.4 tok/s (HTTP with torch.save and base64), 3.9 (numpy with base64), 5.3 (relaying through the 3090 instead of a Mac), 7.8 (WebSocket with binary float16), 8.7 (adding lookahead), 10.9 (switching to a lower-RTT provider), 13-17 (both). Four of the seven steps are serialization and placement; base64 alone costs a 33% payload expansion plus encode and decode CPU.

The paper's reverse SSH tunnel is an implementation choice, not a structural requirement of split inference. In its deployment the local machine sits behind NAT, cloud pods get ephemeral IPs and a fresh SSH port on each start, and SSH keys are distributed across machines. Other deployments can use authenticated ingress, a VPN, or a different bidirectional transport. The transferable requirement is to measure and secure the actual network path rather than to copy the paper's tunnel topology unchanged.

Three cloud-side cache operations are required by multi-token verification and are easy to miss when adapting a single-token server: cache cropping to roll back to a checkpoint when a speculative block is rejected, cache relocation when the n-gram window shifts, and custom attention masks as above.

Executed audit of the published tables

import numpy as np

# Section 6.3, "Projected throughput at target RTTs".
RTT = np.array([20.0, 40.0, 60.0, 80.0, 100.0, 120.0, 150.0])
PUB = {"7B seq":  np.array([15.9, 12.1, 9.7, 8.1, 7.0, 6.1, 5.2]),
       "7B LA":   np.array([18.6, 13.6, 10.7, 8.8, 7.5, 6.5, 5.4]),
       "12B seq": np.array([15.1, 11.6, 9.4, 7.9, 6.8, 6.0, 5.1]),
       "12B LA":  np.array([15.2, 11.6, 9.4, 8.0, 6.9, 6.1, 5.2])}

def fit(rtt, tps):
    """Paper's model: tok/s = 1000*a/(rtt+f). Linear in 1/tps."""
    slope, intercept = np.polyfit(rtt, 1.0 / tps, 1)
    return 1.0 / (1000.0 * slope), intercept / slope

a_seq, f_seq_col = fit(RTT, PUB["7B seq"])
a_la, f_la_col = fit(RTT, PUB["7B LA"])
assert abs(f_seq_col - 42.9) < 1.0          # matches the measured 42.9 ms overhead
assert abs(a_seq - 1.0) < 0.02              # sequential accepts exactly 1 token/step
assert f_la_col < f_seq_col - 9.0           # the lookahead column implies LESS overhead
assert abs(a_la - 1.0) < 0.05               # and an acceptance rate of 1.0, not 1.17

# The 12B pair, where the paper says it DID adjust for batch overhead, behaves correctly.
_, f12_seq = fit(RTT, PUB["12B seq"])
_, f12_la = fit(RTT, PUB["12B LA"])
assert abs(f12_seq - 46.3) < 0.5            # matches the measured 46.3 ms overhead
assert f12_la > f12_seq                     # lookahead costs MORE per step, as it should

# Section 6.4 ablation, 7B on the A100 at ~78 ms. Invert the same model for f.
RTT_ABL, NGRAM = 78.0, np.array([3, 4, 5, 6, 7])
TPS = np.array([9.3, 9.2, 8.9, 9.1, 8.3])
ACC = np.array([1.21, 1.23, 1.23, 1.25, 1.25])
overhead = lambda tps, acc: 1000.0 * acc / tps - RTT_ABL
f_seq, f_la = overhead(8.3, 1.00), overhead(TPS, ACC)
assert f_la.min() > f_seq                    # every lookahead setting costs MORE per step
assert f_la[0] < f_la[-1]                    # and bigger n-grams cost more still

# Break-even acceptance a* = (RTT + f_LA) / (RTT + f_seq).
star = lambda r, f: (r + f) / (r + f_seq)
assert 1.14 < star(RTT_ABL, f_la[2]) < 1.16  # n=5 reproduces the paper's stated 1.15-1.20
assert star(20.0, f_la[0]) > star(RTT_ABL, f_la[0])   # a* RISES as the link gets faster

# Re-project the lookahead column with its own measured per-step overhead.
seq = 1000.0 / (RTT + f_seq)
la_honest = 1000.0 * ACC[0] / (RTT + f_la[0])
assert la_honest[0] < PUB["7B LA"][0]        # 16.8 tok/s at 20 ms, not 18.6
gain_pub, gain_honest = PUB["7B LA"] / seq, la_honest / seq
assert gain_pub[0] > gain_pub[-1]            # published: speculation helps MORE on a fast link
assert gain_honest[0] < gain_honest[-1]      # corrected: it helps LESS. The trend reverses.

# Section 5.2 inversion table, 221 test samples.
N = 221
acc_by_depth = {2: 0.588, 4: 0.443, 6: 0.448, 8: 0.348}
counts = {k: round(v * N) for k, v in acc_by_depth.items()}
assert counts == {2: 130, 4: 98, 6: 99, 8: 77}
assert all(round(100 * counts[k] / N, 1) == round(100 * v, 1)
           for k, v in acc_by_depth.items())
assert counts[6] - counts[4] == 1             # the reported non-monotonicity is one sample

# A paired test needs per-token outcomes at both depths, which the paper does not publish.
# Marginal counts alone cannot supply the discordant-pair counts required by McNemar's test.

# Zipf(s=1) proxy used below: expected observed types and covered probability mass.
vocab, train_n = 32_000, 880
p = 1.0 / np.arange(1, vocab + 1, dtype=float)
p /= p.sum()
seen_probability = 1.0 - (1.0 - p) ** train_n
assert 510 < seen_probability.sum() < 520
assert 0.50 < np.sum(p * seen_probability) < 0.51
assert 0.09 < p[0] < 0.10

# Transport. Hidden-state decode traffic is small, but dense masks grow with context.
for d, kb in ((4096, 8), (5120, 10), (8192, 16), (16384, 32)):
    assert d * 2 / 1024 == kb                # float16 hidden state, matches Section 7.1
hidden_decode_mbit = 2 * 4096 * 2 * 8.8 * 8 / 1e6
assert hidden_decode_mbit < 1.5              # ~1.15 Mbit/s, hidden states in both directions
steps_per_second = 8.8 / 1.17
mask_mbit_128k = 9 * 131072 * 2 * steps_per_second * 8 / 1e6
assert mask_mbit_128k > 140                  # lower-end 9-token mask at 128k context

prefill_one_way_gb = 131072 * 16384 * 2 / 1e9
prefill_roundtrip_gb = 2 * prefill_one_way_gb
assert prefill_roundtrip_gb > 8.5            # hidden tensors only, before rotary data and framing
assert 8 * prefill_roundtrip_gb > 68         # pure transfer seconds at 1 Gbit/s

max_message = 20 * 1024**2
hidden_only_prompt_limit = max_message // (4096 * 2)
assert hidden_only_prompt_limit == 2560       # actual limit is lower after rotary tensors/header

print("all assertions passed")

Six results come out of that block.

The 7B lookahead projection column is not the paper's own model. Refitting tok/s = 1000a/(RTT+f) to the published sequential column recovers f = 43.0 ms and a = 1.000, matching the independently instrumented 42.9 ms fixed overhead exactly. Refitting the lookahead column recovers f = 32.8 ms and a = 0.991. The lookahead column therefore implies a per-step overhead 10.2 ms lower than sequential and an acceptance rate of 1.0, neither of which the paper reports or claims. The 12B pair is the control: its sequential column recovers 46.3 ms, matching its own instrumented figure, and its lookahead column recovers 48.6 ms, correctly higher. That is the same pair of columns produced by a method the paper footnotes as batch-overhead-adjusted, so the anomaly is confined to the 7B row.

The paper's own ablation says the opposite. Inverting the same model against the Section 6.4 measurements gives sequential 42.5 ms and lookahead 52.1, 55.7, 60.2, 59.4, and 72.6 ms at n = 3 through 7. Every speculative setting costs more per step than sequential, by 9.6 to 30.1 ms. Overhead generally rises with n-gram size, with a small dip at n = 6. That is the mechanism the paper itself names ("larger speculative batches increase cloud compute and transfer overhead") and it is why n=3 wins on throughput despite having the lowest acceptance rate.

The break-even acceptance rate rises as the link gets faster. With a* = (RTT + f_LA)/(RTT + f_seq), the n=3 threshold is 1.080 at 78 ms, 1.154 at 20 ms, and 1.203 at 5 ms. The paper's stated 1.15-1.20 break-even at 80 ms actually corresponds to n=5 (1.147), not to the n=3 configuration it recommends. The consequence is a design rule the paper does not state: speculation is a high-latency technique, and its margin shrinks as you succeed at reducing RTT.

The 19 tok/s headline does not survive. Re-projecting the lookahead column with the overhead implied by its own best measured point gives 16.8 tok/s at 20 ms against the published 18.6, a 10.8% overstatement. The published speedup over sequential falls from 1.162x at 20 ms to 1.039x at 150 ms; the corrected speedup rises from 1.048x to 1.152x. The trend reverses. The abstract's "15-19 tok/s at 20ms RTT" should read roughly 15-17. Note that the paper already applies this correction to the 12B column, footnoting that its lookahead projections are "scaled from measured throughput to account for batch overhead", and does not apply it to the 7B column beside it.

The inversion marginals show a large depth effect but do not support a significance test. The one-decimal rates reconstruct to 130, 98, 99, and 77 successes out of 221. The 4-versus-6-layer reversal is one sample. The same tokens appear to be evaluated at every depth, so the outcomes are paired; a valid McNemar test needs the per-token discordant pairs, which neither the paper nor the repository publishes. An independent-proportions z-test would manufacture precision from the marginals and is not valid here.

The current transport cannot carry a long prefill. Hidden-state decode traffic alone is about 1.15 Mbit/s across both directions at 8.8 tok/s on the 7B. Prefill sends the prompt tensor to the cloud and returns a tensor of the same shape: hidden states alone total 0.8 MB for the paper's 100-token prompt, 537 MB for 32k tokens on the 7B, and 8.59 GB for 128k tokens at hidden size 16,384. The last case has a 68.7-second transfer floor at 1 Gbit/s before rotary tensors, framing, or compute. More immediately, the reference client and server both set max_size=20 * 1024 * 1024; a 7B request reaches 20 MiB at 2,560 tokens using hidden states alone, and rotary tensors lower the actual ceiling. Chunking is required before the larger examples can run.

Speculative decode is not constant-bandwidth at long context in this implementation. A dense float16 attention mask is sent on every multi-token step. At 128k context, even a 9-query-token mask consumes about 2.36 MB per step and roughly 142 Mbit/s at the measured 8.8 tok/s and 1.17 accepted tokens per step. The 17-token end of the paper's batch range roughly doubles that traffic. A compact mask representation or cloud-side reconstruction is required for long contexts.

On the attack numbers themselves

The attack is a 3-layer MLP (4096 to 2048 to 2048 to 32000) trained on roughly 880 activation-token pairs and tested on 221, recovering the top-1 token from a single activation. Two framing points matter more than the headline.

First, the reported random baseline of 0.003% is uniform guessing over a 32K vocabulary, which nothing resembles. The paper says so and estimates a frequency-prior baseline at 3-5%. Under a Zipf(s=1) proxy over 32,000 types, always predicting the single most common token scores about 9%, so the attack's lift over a trivial baseline is real but smaller than the 20,000x the uniform baseline implies.

Second, 880 training samples cannot cover a 32K vocabulary. Under the same Zipf proxy, an 880-token draw contains about 515 expected distinct types and those observed types carry about 50.6% of fresh-token probability mass. The published 58.8% result is not directly comparable to that proxy: the implementation randomly splits token positions from the same 16 fixed passages, not held-out documents, and hidden representations can carry more structure than token frequency. The paper is right to call the small MLP a lower-bound attacker, but 58.8% is an in-collection measurement, not a portable recovery rate.

The load-bearing assumption is weaker than the depth knob. The defence the paper recommends first, deepening the split, moves the attack from 58.8% to 34.8% at roughly 3 ms of added local latency per layer. The defence it treats as a prerequisite, keeping local layers private, is what the whole architecture rests on, and it holds only while the adversary cannot obtain labelled (activation, token) pairs from your local layers. An adversary who can act as a client of the deployment (a public-facing application, a shared tenant, any chosen-plaintext path) collects exactly those pairs against the fine-tuned weights and trains the same MLP. The paper anticipates half of this, hypothesising that random LoRA perturbation would not survive an adaptive attacker who can retrain on perturbed activations, and flags validation as future work. Treat "the cloud lacks our local weights" as an assumption to be enforced operationally, not as a property of the architecture.

How to run it in production

Evaluate a symmetric, deeper split than the default. The paper recommends layers 0-4 and 27-31 local for a 32-layer model, ten local layers protecting five layers of depth on each side, at an estimated 10-15% throughput cost. The attack table favours more local depth, but no configuration provides a formal privacy guarantee. Choose the split from an application-specific attack target, local compute budget, and measured throughput rather than treating five layers per side as a universal production default.

Enforce weight divergence and treat it as a release artifact. If the local layers must differ from the published checkpoint for the privacy story to hold, then the fine-tuned local variant is a versioned model with its own evaluation, its own drift, and its own rollback path. Pin it, hash it, and never let a deployment fall back to stock weights on a cache miss.

Pin the split point in the session handshake and randomise it across sessions if you rely on that. All layers share a hidden dimension, so tensor shape does not reveal the split depth. Activation magnitude distributions may still shift by layer, which the paper flags as an uninvestigated caveat, so randomisation raises attack cost rather than providing a guarantee.

Size the session cache. The cloud holds per-session KV state for the resident layers with a five-minute idle expiry. That is user-derived state on untrusted hardware for the session's lifetime plus the TTL, and it is also the capacity limit: concurrent sessions multiply cloud KV footprint exactly as they would in a normal server. The KV-cache sizing in KV cache management applies unchanged to the cloud half.

Price it before you build it. Deriving from the paper's own figures, an A100 at $1.35/hr delivering 8.8 tok/s single-stream costs about $42.61 per million output tokens in rental alone, before the local GPU, its power, or the operator. A 4090 at $0.59/hr and 15 tok/s costs about $10.93 per million. Both are single-stream numbers and both fall with batching, which the paper does not measure. The comparison that matters is against the compliance alternative, not against a cloud API list price, but the number should be on the page before anyone commits.

Instrument RTT and fixed overhead separately. The decomposition per_step_time = RTT + fixed_overhead is the useful part of the paper's model and it survives the audit above. Track both terms as separate SLIs: RTT regressions point to placement, tunnelling, routing, or provider changes; fixed-overhead regressions point to local compute, cloud compute, serialization, or batch shape. Both terms are actionable through different owners. The inference SLOs page covers the alerting shape.

Know the ceiling before you optimise the network. The fixed overhead is a floor that no amount of RTT reduction can cross. The paper's own same-rack projection puts per-step time at about 48 ms and throughput at roughly 21-26 tok/s at 5 ms RTT, at which point 90% of the step is local compute, cloud compute, and serialization rather than network.4 Combined with the corrected break-even above, that sets the order of work: chase RTT while it is 60% of the step, then stop and attack the 43-46 ms fixed term, because past a certain link quality the network is no longer the problem and speculation is worth less, not more.

Do not let sampling in without re-measuring. All published acceptance rates are under greedy argmax, which the paper flags as an upper bound for stochastic decoding. Given that the corrected break-even at 20 ms RTT is already 1.154 against a measured 1.21 acceptance, a modest drop under temperature sampling puts lookahead below break-even entirely.

How to maintain it

The paper observes token-identical greedy output on four prompts per model, 200 generated tokens per prompt. Treat that as a regression corpus, not a structural guarantee. Sequential and batched paths report different logits, so a prompt near an argmax tie can diverge even under greedy decoding. Expand the corpus with tie-margin cases, long contexts, cache relocation, rejected speculation, and both model sizes.

The paper attributes its large "self-perplexity" and logit differences to float16 accumulation order: 1.21 versus 8.48 on the 7B, 1.18 versus 13.29 on the 12B, with maximum absolute logit differences of 29.0 and 38.3. Eight matching token streams do not establish that attribution or distributional equivalence. Gate greedy mode on token identity and minimum argmax margin; gate sampling modes on output-distribution tests, since changed logits can change sampling even when the greedy token matches.

Re-verify the provider network path on every pod restart. Ephemeral IPs, fresh SSH ports, and provider routing changes all move RTT, and RTT is 63-64% of per-step time.

Failure modes

  • Relative causal masking corrupts speculative attention. Using is_causal=True for a multi-token step with a populated KV cache lets speculative positions attend to entries they must not. The failure is silent: output degrades rather than crashing. Build explicit masks.
  • Proxied provider networking silently caps throughput. The paper measured 1.2 tok/s on a 200-400 ms proxied path despite using a 4090. Nothing in the provider's documentation surfaced that route; only measurement did.
  • Speculation below break-even makes things slower. At ~80 ms RTT the n=5 threshold is 1.147 accepted tokens per step. Creative-text prompts measured 1.06-1.12, below it, and the paper's own category table shows lookahead at 0.91-0.99x sequential on structured and creative content. Speculation is not free to leave on.
  • Public local weights void the privacy claim. If the cloud can download identical copies of the local layers, an attacker reconstructs (activation, token) pairs at will. This is the paper's stated critical assumption and it is an operational obligation, not an architectural property.
  • Shallow splits leak on the output side too. The default protects input tokens with two layers and output tokens with two layers. Deepening only the input half leaves generated text exposed at the same rate.
  • Prefill exceeds the transport's hard message cap. Hidden states alone reach 20 MiB by 2,560 tokens on the 7B, before rotary tensors and framing, while both WebSocket endpoints cap messages at 20 MiB. A 32k round trip moves at least 537 MB of hidden states. Chunked prefill is not implemented; see disaggregated inference.
  • Dense masks turn long-context decode into a bandwidth load. The transmitted float16 mask grows as query_tokens x total_KV_tokens. Reconstruct it on the cloud from compact branch metadata instead of shipping the matrix.
  • Session KV expiry drops long-running conversations. A five-minute idle TTL on the cloud silently invalidates a session; the client must detect and re-prefill, which is a full-prompt retransmission.
  • Single-stream results do not extrapolate to a fleet. Every number here is one client on one link. Concurrency changes cloud compute, cloud KV footprint, and the fixed-overhead term the projection model treats as constant.

Open questions and validation

  • Nothing above 12B has been run. The RTT-to-compute argument says the pattern improves with scale, and that is the regime where every figure is an estimate. A 70B measurement would either confirm or kill the thesis.
  • One architecture family. Mistral 7B and NeMo 12B are both MistralForCausalLM. Acceptance-rate consistency across those two is weak evidence for architecture-independence.
  • No batched measurement. Both the throughput and the economics change under concurrency, and the fixed-overhead term in the projection model is not constant across batch sizes.
  • The fine-tuning defence is untested. The paper hypothesises that random LoRA perturbation would not stop an adaptive attacker and does not test it. Whether genuinely private local weights resist a chosen-plaintext adversary is the question the whole privacy claim rests on.
  • A sequence-aware attacker was not built. The MLP treats positions independently. An attacker with a language-model prior over token sequences would do better, by an unmeasured margin.
  • The inversion transcript is absent from the cited repository. At commit ce6dfba811ff3058da134e92a128deb6e0027b69, the repository contains the attack script but not the experiment_data/inversion_results.json file it writes. The table can be checked against the paper, but its run cannot be independently audited from the published artifacts.

References

Related: Cross-WAN model-parallel inference · Non-colocated inference patterns · Speculative decoding · Speculative decoding economics · GPU confidential computing · KV cache management · Disaggregated inference · P2P transport for decentralized inference · Mesh LLM decentralized serving · Inference SLOs · Security and multitenancy · Gradient leakage in training · Prima.cpp heterogeneous home-cluster inference · Overlay mesh networking · Glossary · Activation obfuscation on untrusted accelerators · Covariant obfuscation for private inference

  1. arXiv:2602.16760v1, Section 3.1. Mistral 7B Instruct v0.3, 32 layers, hidden 4096: embedding, layers 0-1, layers 30-31, RMS norm and LM head local; layers 2-29 cloud. Mistral NeMo 12B Instruct, 40 layers, hidden 5120, 8 KV heads against 32 attention heads, 131K vocabulary: layers 0-1 and 38-39 local, layers 2-37 cloud. Local VRAM measured at 2.0 GB and 4.9 GB respectively, both on a 24 GB RTX 3090. ↩

  2. arXiv:2602.16760v1, Section 5.2, "Comparison of privacy approaches". The overhead figures are the paper's own summary of the cited literature, not its own measurements, and the split-inference row's "3-5x slower" is relative to a cloud API baseline rather than to local-only execution. Against the paper's own local-only 7B baseline of about 39 tok/s, the measured split configuration at 13-17 tok/s is 2.3-3.0x slower. ↩

  3. arXiv:2602.16760v1, Sections 3.4 and 6.6. "VAST.ai routes all SSH traffic through proxy servers in Virginia, regardless of GPU location." The paper adds that this "is not documented in VAST.ai's literature and was discovered only through empirical measurement." Provider behaviour is a 2026-02 observation and should be re-measured before it is relied on. ↩

  4. arXiv:2602.16760v1, Section 8.4, "Partnership Model with Inference Providers". The paper sketches a deployment pairing a local SDK with a partnered inference provider and projects throughput at four partner-infrastructure tiers: about 80 ms RTT (standard GPU cloud) giving 8.1 sequential and 8.8 lookahead on the 7B, about 40 ms (regional cloud) giving 12.1 and 13.6, about 20 ms (co-located partner) giving 15.9 and 18.6, and about 5 ms (same rack) giving 20.9 and 23.9. "The fixed overhead (43-46ms) represents a floor that cannot be reduced by lowering RTT alone, it includes local GPU compute, cloud GPU compute, and serialization. At 5ms RTT (same-rack), per-step time is dominated by this fixed overhead (~48ms), yielding ~21-26 tok/s." These rows share the projection model audited above, so the 7B lookahead column carries the same overstatement; the sequential column and the floor argument are unaffected. ↩

  5. arXiv:2602.16760v1, Sections 4.2 and 4.3. Jacobi decoding initialises a block of k future positions and iterates to convergence, giving tok/s = k / (iterations x RTT). "In practice over WAN, we observed poor convergence, vanilla Jacobi rarely commits more than 1-2 tokens per iteration for language modeling, yielding ~1.5 tok/s (worse than sequential due to the overhead of transmitting larger tensors for the full block)." Against the paper's measured sequential baseline of about 8.3 tok/s at 78 ms RTT, that is roughly 5.5x slower. Lookahead recovers the idea by "collecting n-gram candidates from the Jacobi iteration trajectories themselves". The paper argues the split setting amplifies lookahead's value because a round trip costs about 100 ms against about 3 ms locally, so verifying five candidates costs the same single round trip as generating one token; that argument is about the size of the prize, and the executed audit above shows the realised gain is smaller than the projection table claims. ↩