Skip to content
Markdown

Activation obfuscation on untrusted accelerators

Scope: the mixed-fleet pattern where a small number of confidential GPUs hold a model's activations and a larger pool of ordinary accelerators executes its matrix multiplications on obfuscated data. This page covers why a secret transform applied on the token axis passes through a weight multiplication unchanged, what an adversary reading device memory still learns, the batching-time defences the scheme depends on, the cost accounting that decides whether offloading is worth doing at all, and how to tell a tested obfuscation scheme from an untested one. The hardware-attested alternative that needs no obfuscation is GPU confidential computing; the approach that moves the secret into the weights instead of the activations is covariant obfuscation; the layer-split cousin is privacy-aware split inference.

Primary source: Anatoly Belikov and Ilya Fedotov, "Good-Enough LLM Obfuscation (GELO)", arXiv:2603.05035v2 [cs.CR], 6 March 2026, SingularityNET Foundation and Singularity Compute. No venue stated, no code released. The historical predecessor discussed under a learned obfuscator is Xu et al., arXiv:1912.09859v3, IEEE 2020.

What this page adds. Both Python blocks are executed and asserted (Python 3.12.3, numpy 2.4.6). The first confirms the correctness identity, then measures the covariance leak the paper concedes, verifies that shield vectors shift the observed Gram by exactly the shield covariance, shows that the paper's non-orthogonal mitigation masks far less than it appears to once an attacker removes scale, and quantifies its low-precision cost, which the paper describes only qualitatively. The second reproduces the paper's own Appendix A arithmetic and then audits its headline offload fraction against published model configs. Figures labelled with a table or section number are the paper's; everything else is this page's derivation.

Read the latency numbers as a two-process microbenchmark, not as serving results. The paper states plainly that it did not integrate with an inference engine: its measurements move random batches between two processes on one machine over a socket, with no KV cache, no scheduler, and no real model in the loop.

What it is

A transformer layer's projections are right-multiplications. If the batch of hidden states is H with one row per token and the projection weight is W, the layer computes H W. Multiply H on the left by any invertible matrix A and the product survives:

A^-1 ((A H) W) = (A^-1 A) H W = H W

A acts on the token axis, so it commutes with anything acting on the feature axis. That gives a division of labour. The confidential GPU samples a fresh secret A, forms U = A H, and ships U plus the weights to a non-confidential accelerator. That accelerator computes Y = U W and returns it. The confidential GPU applies A^-1 and recovers the true projection exactly. The untrusted device sees U, W, and Y, and never sees H.

The scheme is applied to the query, key, value, and output projections of attention. It cannot be applied across a non-linearity: softmax, SwiGLU, and RMSNorm all act per token or per row, so the mixing must be undone before any of them run. Each offloaded GEMM therefore ends with a return trip to the trusted side.

Security rests on A being fresh for every batch. Given only U = A H, the pair (A R^-1, R H) produces the same observation for any invertible R, so H is determined only up to an unknown invertible transform. The attacker faces a single-batch blind source separation problem, and because A never repeats, statistics from different batches sit in different coordinate systems and cannot be accumulated. The paper is explicit that this is an identifiability argument and not a cryptographic reduction: "We do not offer a reduction-based or complexity-theoretic proof of security."

Why use it

The target deployment is a fleet with a few confidential GPUs and many ordinary ones. Confidential computing is available on Hopper and Blackwell class parts; an L40S or an older A100 has no such mode. If policy forbids plaintext activations on the non-confidential pool, that pool is idle for private work, and cluster throughput is capped by the small confidential partition.

The alternatives each fail a different constraint. Multi-party computation and homomorphic encryption carry overheads the paper puts at over 100x, and MPC additionally assumes non-colluding parties, which a single cloud tenant cannot arrange. The cheaper prior schemes fail differently, and the failure is specific to open-weight models. STIP and KV-Shield both hide data behind a static secret permutation applied to the weights. When the weights are public, the adversary observes the permuted weights being loaded onto the untrusted GPU and solves W' = W R for the secret R directly. Any scheme whose secret is recoverable from the weights alone is broken by publishing the checkpoint.

Per-batch mixing avoids that because the secret is never applied to the weights. W is shipped in the clear; only the activations are transformed, and the transform changes every batch.

When to use it (and when not)

The security cost is several times the work being offloaded, and that ratio survives any transport. Table 3 at batch 512 splits GELO's own overhead into 2.322 ms to generate A, 0.199 ms to mix, and 0.272 ms to un-mix: 2.793 ms of trusted-side work to move a GEMM that costs 0.441 ms. The scheme spends 6.3 times the offloaded compute to offload it, and generating A alone is 5.3 times it. No faster interconnect changes that. Read this before the transport numbers, because it is the part that does not depend on the prototype.

The transport measurement is worse still, but it is a prototype artifact. In the same table the data copy is 14.186 ms, 81.4% of the 17.42 ms total. The baseline is insecure offload, not local execution, so both columns pay that copy; computing the GEMM on the trusted side would cost about 0.44 ms and no copy at all. As measured, shipping the work costs roughly thirty times doing it locally.1 That ratio is a same-machine socket, not PCIe. The durable conclusion is narrower: the paper shows obfuscated offload is about 20% slower than unobfuscated offload, and never shows that offloading beats not offloading.

The offloadable share is smaller than advertised. The paper claims the scheme "offloads a large fraction of linear-algebra cost (e.g., ≈ 76% in a typical Llama 2 7B configuration)". Its own Appendix A puts the four attention projections at about 67M multiply-adds per token against about 135M for the feed-forward network. The executed audit below reproduces those figures and finds the offloadable share is 33.2% of per-token linear algebra: the FFN, which the scheme does not touch, is twice the size of everything it does touch. That 33.2% is the supremum over all context lengths, so no reading of the denominator makes what the scheme offloads reach 76%.

Grouped-query attention shrinks what gets offloaded, but attention width decides the final number. GQA makes the key and value projections a fraction of their multi-head size: 44% smaller on both Llama-3-70B and Qwen3-32B. The offloadable share lands at 17.0% for Llama-3-70B and 18.1% for Qwen3-32B. Qwen3-32B is the higher of the two despite the same GQA ratio, because it runs 64 heads of width 128 over a 5120-wide residual stream, so its attention block is 8192 wide and the projections are larger relative to the model. Read head_dim from the config rather than dividing hidden size by head count; for Qwen3 those differ by 60%.

Round trips scale with depth. Query, key, and value can be fused into one offloaded GEMM, but the output projection sits behind softmax and needs a second. That is two crossings per layer, so 64 for Llama-2-7B and 160 for Llama-3-70B per forward pass, each carrying a full batch-by-hidden tensor in both directions. This is the page's derivation, not the paper's; the paper benchmarks a single projection.

Do not deploy the bare protocol. Fresh orthogonal mixing on its own is not sufficient against independent component analysis. At low padding energy the paper reports that "ICA p95 is high (≈ 0.97–0.99 for n ≥ 256), even when medians are modest", meaning the best-recovered 5% of tokens in a batch come back at near-perfect cosine similarity. Only high-energy shield vectors bring p95 below 0.28. The defence, not the mixing, is doing the work.

Do not expect low precision to be free. Table 1 reports float32 top-1 equality of 1.000000 but bfloat16 at 0.988045, which the paper describes as introducing "no meaningful numerical error". Slightly over one token in a hundred changes at the top of the distribution, and the mean per-token logit L2 distance is 6.18. Float16 does better than bfloat16 (0.998450), which is the expected ordering given the mantissa widths.

Do not use it where the threat is an active adversary. The model is honest-but-curious with read access to device memory. Tampering, timing and cache side channels, and availability attacks are all out of scope.

Architecture

flowchart LR
  subgraph TEE["Confidential GPU (H100/H200 class)"]
    HS["Hidden states H<br/>plaintext, never leaves"]
    GEN["Sample fresh A per batch<br/>QR if orthogonal, else check cond(A)"]
    MIX["U = A H<br/>+ k shield rows"]
    UNMIX["A^-1 Y<br/>then softmax, RoPE, norms"]
  end
  subgraph WIRE["PCIe / fabric, 2 crossings per layer"]
    UP["U (batch x hidden)"]
    DOWN["Y (batch x proj)"]
  end
  subgraph GPU["Untrusted accelerator (L40S class)"]
    GEMM["Y = U W<br/>weights W are public"]
    ADV["Adversary reads VRAM:<br/>sees U, W, Y"]
  end
  HS --> GEN --> MIX --> UP --> GEMM --> DOWN --> UNMIX
  UNMIX -.-> HS
  GEMM -.- ADV

The load-bearing detail is that A is n x n on the token axis, not d x d on the feature axis. That is why generation cost grows as the cube of batch size and why the overhead curve turns back upward for large batches.

How to use it

1. Exclude the layers where the mapping to tokens is shortest. The paper's own batching-time defence list says not to apply the scheme to the first few layers or the final layer, and to compute those entirely inside the trusted device. Early hidden states sit close to the embedding table and are the easiest to match against known token vectors.

2. Build batches across users. A batch drawn from one user's session is a batch whose rows are correlated and whose content the adversary may partly control. Mixing requests from independent users reduces both.

3. Detect and disrupt flooding. An adversary who can inject many copies of a known token gains anchors. The paper's countermeasure is to track token-frequency statistics and inject random tokens when they diverge from baseline. Note this interacts badly with prefix caching, which exists precisely to exploit repeated prefixes.

4. Pad with shield vectors, and scale them. Append k secret Gaussian rows to the batch, with k around 5% of the batch and row norms 4 to 10 times the mean data row norm. The shields displace real tokens, so the throughput cost is the padding fraction.

5. Enforce conditioning when using a non-orthogonal transform. Orthogonal A is fast and stable but leaks the Gram matrix exactly. A general invertible A masks it, at the cost of an O(n^3) inverse per batch and amplified rounding. The paper's rule is cond(A) < 100; the block below shows what that buys and what violating it costs.

How to develop with it

The three claims worth checking before building anything are that the offload is exact, that an orthogonal transform leaks the covariance in full, and that the shield trick does what it says. All three are one-screen numpy checks.

# Runnable on system python3 (numpy only). The core claim of per-batch left-mixing
# (GELO, arXiv:2603.05035v2 Sec 3.2): a secret invertible A applied on the TOKEN axis
# commutes with right-multiplication by a weight matrix, so A^-1((A H) W) == H W.
# We assert that, then measure exactly what the untrusted device still learns.

import numpy as np

rng = np.random.default_rng(0)
n, d, p = 64, 128, 128  # n tokens in the batch, model dim d, projection width p
H = rng.standard_normal((n, d))
W = rng.standard_normal((d, p))


def haar_orthogonal(m, rng):
    """Uniform orthogonal matrix. The QR sign fix matters: without it Q is not Haar."""
    Q, R = np.linalg.qr(rng.standard_normal((m, m)))
    return Q * np.sign(np.diag(R))


def conditioned(m, kappa, rng):
    """Invertible matrix with condition number exactly kappa. Singular values are
    geometric-mean 1, so |det| is 1, but the energy ||A H|| still grows with kappa."""
    U, V = haar_orthogonal(m, rng), haar_orthogonal(m, rng)
    s = np.geomspace(np.sqrt(kappa), 1.0 / np.sqrt(kappa), m)
    return (U * s) @ V.T


# 1. Correctness. The offload is output-preserving in exact arithmetic.
A = haar_orthogonal(n, rng)
Y = (A @ H) @ W  # the untrusted accelerator computes this and sees U = A H
rec = np.linalg.solve(A, Y)  # the TEE un-mixes
assert np.allclose(rec, H @ W, rtol=0, atol=1e-9), "left-mixing must be output-preserving"
print(f"1. exact offload            max abs err {np.abs(rec - H @ W).max():.2e}")

# 2. Adversarial: an orthogonal A leaks the feature-side Gram matrix in full.
#    U^T U = H^T A^T A H = H^T H whenever A^T A = I. The attacker gets the d x d
#    covariance of the hidden states for free, without solving anything.
U = A @ H
assert np.allclose(U.T @ U, H.T @ H, rtol=0, atol=1e-9), "orthogonal A leaks H^T H exactly"
gram_err = np.linalg.norm(U.T @ U - H.T @ H) / np.linalg.norm(H.T @ H)
print(f"2. orthogonal A, H^T H leak rel err {gram_err:.2e}  (0 means fully leaked)")

# 3. The token-side spectrum leaks too: U U^T = A (H H^T) A^T is similar to H H^T,
#    so every eigenvalue of the token-token similarity matrix survives the mixing.
ev_true = np.sort(np.linalg.eigvalsh(H @ H.T))
ev_obs = np.sort(np.linalg.eigvalsh(U @ U.T))
assert np.allclose(ev_true, ev_obs, rtol=0, atol=1e-8), "orthogonal mixing preserves eigenvalues"
print(f"3. similarity spectrum      max eigval drift {np.abs(ev_true - ev_obs).max():.2e}")

# 4. Mitigation 1 (non-orthogonal A) actually masks the Gram matrix.
A_gen = conditioned(n, 50.0, rng)
assert abs(np.linalg.cond(A_gen) - 50.0) < 1e-6, "condition number must be as requested"
U_gen = A_gen @ H
G_obs, G_true = U_gen.T @ U_gen, H.T @ H
masked = np.linalg.norm(G_obs - G_true) / np.linalg.norm(G_true)
assert masked > 1.0, "a general invertible A must destroy the H^T H identity"
# Most of that error is a change of scale, and scale is free to remove: the attacker
# matches traces without knowing anything secret. Report what survives that step.
G_scaled = G_obs * (np.trace(G_true) / np.trace(G_obs))
scaled = np.linalg.norm(G_scaled - G_true) / np.linalg.norm(G_true)
assert scaled < masked / 5, "trace matching must remove most of the apparent masking"
# And the principal subspace partly survives: compare the top-10 eigenvector span of
# the masked Gram against the truth, versus a random-subspace baseline.
def top_span(M, k=10):
    return np.linalg.eigh(M)[1][:, -k:]
align = lambda A, B: np.linalg.norm(A.T @ B) ** 2 / A.shape[1]
true_sp = top_span(G_true)
leak = align(top_span(G_obs), true_sp)
base = np.mean([align(np.linalg.qr(rng.standard_normal((d, 10)))[0], true_sp)
                for _ in range(50)])
assert leak > 3 * base, "the non-orthogonal mitigation still leaks principal directions"
print(f"4. non-orthogonal A (k=50)  H^T H rel err {masked:.2f} raw, "
      f"{scaled:.2f} after free trace matching")
print(f"   top-10 subspace alignment {leak:.3f} vs {base:.3f} random "
      f"({leak/base:.1f}x chance): masking is partial")

# 5. Mitigation 2 (shield rows). Padding with k secret high-energy rows S shifts the
#    observed Gram by exactly S^T S, so the attacker sees H^T H + S^T S and cannot
#    isolate either term. Assert the identity holds exactly, not approximately.
k = max(1, int(round(0.05 * n)))  # paper's k ~ 5% of n
row_norm = np.linalg.norm(H, axis=1).mean()
S = rng.standard_normal((k, d))
S *= (10.0 * row_norm) / np.linalg.norm(S, axis=1, keepdims=True)  # scale 10x, per paper
H_full = np.vstack([H, S])
A_full = haar_orthogonal(n + k, rng)
U_full = A_full @ H_full
resid = np.linalg.norm(U_full.T @ U_full - (H.T @ H + S.T @ S))
assert resid < 1e-8, "shielded Gram must equal H^T H + S^T S exactly"
energy = np.linalg.norm(S) ** 2 / np.linalg.norm(H) ** 2
print(f"5. {k} shield rows ({100*k/n:.0f}% of n)  Gram residual {resid:.2e}, "
      f"shield/data energy {energy:.2f}")

# 6. Conditioning is load-bearing, and the paper's kappa < 100 rule is the reason.
#    The accelerator returns Y in low precision; the TEE then applies A^-1, which
#    amplifies that rounding by roughly kappa(A). Sweep kappa and measure.
def bf16(x):
    """Round to bfloat16: add half an ulp of the low 16 bits, then drop them.
    This rounds, it does not truncate; truncating instead doubles the error."""
    v = np.asarray(x, dtype=np.float32).view(np.uint32)
    return ((v + 0x8000) & 0xFFFF0000).view(np.float32).astype(np.float64)


print("6. un-mix error vs condition number (accelerator returns bf16):")
exact = H @ W
prev = None
for kappa in [1.0, 10.0, 100.0, 1e3, 1e4]:
    Ak = haar_orthogonal(n, rng) if kappa == 1.0 else conditioned(n, kappa, rng)
    err = np.linalg.norm(np.linalg.solve(Ak, bf16(( Ak @ H) @ W)) - exact) / np.linalg.norm(exact)
    print(f"     kappa={kappa:>8.0f}   relative error {err:.3e}")
    if prev is not None:
        assert err > prev, "error must grow with the condition number"
    prev = err
assert prev > 1e-2, "kappa=1e4 must visibly corrupt the un-mixed result"
print("   all assertions passed")

Executed output:

1. exact offload            max abs err 8.97e-14
2. orthogonal A, H^T H leak rel err 6.67e-16  (0 means fully leaked)
3. similarity spectrum      max eigval drift 9.09e-13
4. non-orthogonal A (k=50)  H^T H rel err 11.33 raw, 1.38 after free trace matching
   top-10 subspace alignment 0.403 vs 0.079 random (5.1x chance): masking is partial
5. 3 shield rows (5% of n)  Gram residual 1.06e-11, shield/data energy 4.67
6. un-mix error vs condition number (accelerator returns bf16):
     kappa=       1   relative error 1.655e-03
     kappa=      10   relative error 3.633e-03
     kappa=     100   relative error 1.835e-02
     kappa=    1000   relative error 1.203e-01
     kappa=   10000   relative error 9.972e-01
   all assertions passed

Four results are worth carrying forward. The covariance leak under orthogonal mixing is total, not partial: the adversary reconstructs H^T H to machine precision without solving anything, and with it the principal subspace. The paper measures a participation ratio of 123 on 4096-dimensional Llama-2 states, so that leak reduces the attacker's search from a 4096-dimensional problem to roughly 123 dimensions.

Second, the non-orthogonal mitigation masks much less than a first look suggests. Its raw Gram error of 11.33 falls to 1.38 as soon as the attacker rescales the observed Gram to match the true trace, which costs nothing and needs no secret, so most of the apparent masking was a change of scale rather than a change of structure. What remains still leaks direction: the top-10 eigenvector span of the masked Gram aligns with the truth at 5.1 times the random-subspace baseline. Mitigation 1 is a partial obscuring of the covariance, not a removal of it, and the page's earlier framing of the two mitigations as a clean choice between a leak and a numerical budget was too generous to the non-orthogonal branch.

Third, shield rows work by exact construction rather than by statistical luck, but at 5% padding and 10x norm scaling the shields carry 4.67 times the total energy of the real data, which is what the paper means when it says such vectors "pollute higher-order statistics". Note that the paper's stated range is 4 to 10 times the mean row norm and only the top of that range was shown to drive p95 recovery below 0.28; at 4x the shields carry less energy than the data they are meant to hide.

Fourth, the price of the non-orthogonal mitigation is a tenfold increase in low-precision error at the paper's own cond(A) < 100 limit, and near-total corruption two orders of magnitude beyond it. The growth is not linear in the condition number: measured against the orthogonal case it is about 2x at cond(A) = 10 and about 11x at 100, so treat the rule as an empirical ceiling rather than a proportionality.

Auditing the offload economics

# Runnable on system python3 (numpy only). Audit of the claim that TEE-anchored
# projection offload moves "about 76% of linear-algebra cost" for Llama 2 7B
# (arXiv:2603.05035v2, contributions list and Appendix A).
# Only Q/K/V/O are offloaded; the FFN and the attention core stay in the TEE.
# Shapes come from the published config.json of each model, not from the paper.
# head_dim is read as its own field: Qwen3 sets head_dim=128 while d/n_heads=80,
# so deriving it by division silently understates that model by 60%.

import numpy as np

MODELS = {  # d, ffn_intermediate, n_heads, n_kv_heads, head_dim
    "Llama-2-7B  (MHA)": (4096, 11008, 32, 32, 128),
    "Llama-3-70B (GQA)": (8192, 28672, 64, 8, 128),
    "Qwen3-32B   (GQA)": (5120, 25600, 64, 8, 128),
}


def per_token_madds(d, ffn, n_heads, n_kv, head_dim, ctx):
    """Multiply-adds per token in one decoder layer, split by where they run."""
    q_out, kv_out = n_heads * head_dim, n_kv * head_dim
    qkvo = d * q_out + 2 * (d * kv_out) + q_out * d  # W_q, W_k, W_v, W_o
    mlp = 3 * d * ffn  # gate, up, down
    core = 2 * q_out * ctx  # scores QK^T and the weighted sum over V
    return qkvo, mlp, core


# Guard the trap: attention width is not always the residual width.
for name, (d, _, nh, _, hd) in MODELS.items():
    if nh * hd != d:
        print(f"note: {name.split()[0]} attention width {nh*hd} != hidden size {d}")
assert MODELS["Qwen3-32B   (GQA)"][2] * MODELS["Qwen3-32B   (GQA)"][4] == 8192

print(f"\n{'model':<20} {'Q/K/V/O':>10} {'FFN':>10} {'attn core':>10} "
      f"{'offloadable':>12}")
print("-" * 66)
CTX = 2048
for name, (d, ffn, h, kv, hd) in MODELS.items():
    qkvo, mlp, core = per_token_madds(d, ffn, h, kv, hd, CTX)
    share = qkvo / (qkvo + mlp + core)
    print(f"{name:<20} {qkvo/1e6:>9.1f}M {mlp/1e6:>9.1f}M {core/1e6:>9.1f}M "
          f"{100*share:>11.1f}%")

# Reproduce the paper's own Appendix A figures for Llama 2 7B before drawing any
# conclusion, so we know we are counting the same way it does.
d, ffn, h, kv, hd = MODELS["Llama-2-7B  (MHA)"]
qkvo, mlp, _ = per_token_madds(d, ffn, h, kv, hd, 0)
assert abs(qkvo / 1e6 - 67.1) < 0.1, "Appendix A states ~67M MAdds for Q/K/V/O"
assert abs(mlp / 1e6 - 135.3) < 0.2, "Appendix A states ~135M MAdds for the FFN"
assert abs((qkvo + mlp) / 1e6 - 202.4) < 0.3, "Appendix A states ~202M in total"
print(f"\nAppendix A reproduced: Q/K/V/O {qkvo/1e6:.1f}M + FFN {mlp/1e6:.1f}M "
      f"= {(qkvo+mlp)/1e6:.1f}M MAdds/token")

# The claim under test, stated two ways so the denominator is never ambiguous.
share_lin = qkvo / (qkvo + mlp)
share_all = qkvo / (qkvo + mlp + 2 * h * hd * CTX)
print(f"offloaded / (projections + FFN)              = {100*share_lin:.1f}%")
print(f"offloaded / (projections + FFN + attn core)  = {100*share_all:.1f}% at ctx {CTX}")
assert 0.30 < share_lin < 0.35, "Q/K/V/O is about a third of per-token linear algebra"

# Can any denominator reach 76%? Q/K/V/O over the full Appendix A cost peaks at
# ctx=0 and only falls from there, so 76% is unreachable for what is offloaded.
peak = qkvo / (qkvo + mlp)
assert peak < 0.34, "even at zero context the offloaded share cannot reach 76%"
print(f"supremum of the offloaded share over all contexts = {100*peak:.1f}%")

# Two readings do produce 76%. Neither describes what the scheme offloads.
ctx_attn = qkvo * (1 - 0.76) / 0.76 / (2 * h * hd)  # Q/K/V/O vs attention block only
assert abs(qkvo / (qkvo + 2 * h * hd * ctx_attn) - 0.76) < 1e-9
ctx_lin = (qkvo + mlp) * (1 - 0.76) / 0.76 / (2 * h * hd)  # all linear vs total
assert abs((qkvo + mlp) / (qkvo + mlp + 2 * h * hd * ctx_lin) - 0.76) < 1e-9
print(f"76% is reached by  Q/K/V/O vs attention block   at ctx {ctx_attn:,.0f}")
print(f"76% is reached by  (proj+FFN) vs total          at ctx {ctx_lin:,.0f} "
      f"(Llama-2-7B max_position_embeddings is 4096)")

# Grouped-query attention shrinks K and V, which is most of what gets offloaded,
# but a model can widen attention past its residual stream and claw some back.
print("\nEffect of GQA, holding each model's own shapes:")
for name in ("Llama-3-70B (GQA)", "Qwen3-32B   (GQA)"):
    d, ffn, h, kv, hd = MODELS[name]
    gqa, mlp_i, core_i = per_token_madds(d, ffn, h, kv, hd, CTX)
    mha, _, _ = per_token_madds(d, ffn, h, h, hd, CTX)  # same model as if MHA
    print(f"  {name}  Q/K/V/O {gqa/1e6:.1f}M vs MHA {mha/1e6:.1f}M "
          f"({100*(1-gqa/mha):.0f}% smaller), share {100*gqa/(gqa+mlp_i+core_i):.1f}%")
    assert gqa < mha, "GQA must shrink the offloadable tensors"

# Round trips are the real cost. Q/K/V can go in one fused GEMM, O needs a second,
# and everything between them is non-linear, so it must come back to the TEE.
layers = {"Llama-2-7B  (MHA)": 32, "Llama-3-70B (GQA)": 80, "Qwen3-32B   (GQA)": 64}
print("\nTEE <-> accelerator round trips per forward pass (2 per layer):")
for name, L in layers.items():
    print(f"  {name:<20} {2*L:>4} round trips")
print("all assertions passed")

Executed output:

note: Qwen3-32B attention width 8192 != hidden size 5120

model                   Q/K/V/O        FFN  attn core  offloadable
------------------------------------------------------------------
Llama-2-7B  (MHA)         67.1M     135.3M      16.8M        30.6%
Llama-3-70B (GQA)        151.0M     704.6M      33.6M        17.0%
Qwen3-32B   (GQA)         94.4M     393.2M      33.6M        18.1%

Appendix A reproduced: Q/K/V/O 67.1M + FFN 135.3M = 202.4M MAdds/token
offloaded / (projections + FFN)              = 33.2%
offloaded / (projections + FFN + attn core)  = 30.6% at ctx 2048
supremum of the offloaded share over all contexts = 33.2%
76% is reached by  Q/K/V/O vs attention block   at ctx 2,587
76% is reached by  (proj+FFN) vs total          at ctx 7,801 (Llama-2-7B max_position_embeddings is 4096)

Effect of GQA, holding each model's own shapes:
  Llama-3-70B (GQA)  Q/K/V/O 151.0M vs MHA 268.4M (44% smaller), share 17.0%
  Qwen3-32B   (GQA)  Q/K/V/O 94.4M vs MHA 167.8M (44% smaller), share 18.1%

TEE <-> accelerator round trips per forward pass (2 per layer):
  Llama-2-7B  (MHA)      64 round trips
  Llama-3-70B (GQA)     160 round trips
  Qwen3-32B   (GQA)     128 round trips
all assertions passed

The Appendix A figures reproduce exactly, so the disagreement is about the denominator rather than about the arithmetic. Against all per-token linear algebra the offloadable share is 33.2%, and because that ratio peaks at zero context and only falls as context grows, 76% is out of reach for what the scheme offloads under every denominator and every context length.

Two readings do produce 76%, and neither describes the offload. Q/K/V/O reaches 76% of the attention block alone at about 2,587 tokens of context. Projections plus FFN reach 76% of total per-token compute at about 7,801 tokens, which is past Llama-2-7B's 4,096-token limit, but that quantity includes the feed-forward network, which the scheme does not offload. The second reading is the more natural fit to Appendix A's own framing, and under it the paper's error is not the arithmetic but the step from "linear algebra is most of the cost" to "GELO offloads most of the cost".2 Extending the scheme to the FFN, which the paper lists as future work, is what would close that gap, and it is exactly the part that has not been evaluated.

The predecessor: a learned obfuscator

The same problem was attacked in 2019 with a learned transform rather than an algebraic one. ObfNet puts a small neural network on the edge device, trained by concatenating it with the frozen backend model and updating only the small network's weights, so that the backend accepts obfuscated and plaintext inputs interchangeably. The paper summarises its accuracy cost as drops "generally within 1%" from a baseline near 99%; per case study the reported drops are 0.10% to 1.15% on spoken digits, 0.46% to 1.43% on MNIST, and 0.12% to 2.81% on sign language. Per-sample cost on a Coral TPU is 0.2 to 11 ms.

It is worth reading for one reason: its privacy evaluation is a demonstration of what not to accept. For the audio task the assessment is that ten student volunteers could not recognise the obfuscated samples, at a 4% to 7% recognition rate against a 10% chance baseline. For the two image tasks there is no volunteer study at all; the evidence is the authors' own statement that they cannot interpret the obfuscated images.3 No inversion attack is attempted anywhere. The paper's security argument is that ReLU is many-to-one, which makes exact inversion "virtually impossible". Non-injectivity does not by itself defeat an optimisation-based inversion, and the argument sits awkwardly with the paper's own threat model, because the honest-but-curious backend generates, trains, and distributes every ObfNet itself and therefore holds the weights; the only residual secret is which member of a small set the device selected. The durable objection is simpler than any of that: no attack was ever run. Compare GELO, which runs FastICA, JADE, joint diagonalisation, anchor-based recovery, and multi-view ICA against its own defences and publishes where they weaken, or covariant obfuscation, which runs seven published attacks against its own construction. A scheme that reports no attack has not been shown to resist one.

How to run it in production

Batch size is the primary tuning knob and the overhead curve is not monotone. Table 2 gives 28.9% overhead at batch 64, a minimum near 19.9% at 256, and 50.1% at 8192. Small batches pay because generating A dominates a fast GEMM; large batches pay because generating an n x n orthogonal matrix costs O(n^3). Batches in the 256 to 512 range sit at the bottom of the curve. Note that this interacts directly with serving throughput, which normally wants the largest batch the memory budget allows.

Budget the trusted side for matrix generation, not for mixing. At batch 512 the split is 2.322 ms to generate A, 0.199 ms to mix, and 0.272 ms to un-mix. Generation is 83% of the 2.793 ms security cost, and it is pure overhead that produces no model output.

How often that 2.322 ms is paid is unresolved in the source, and the two readings differ by two orders of magnitude. Section 3.2 says the protocol "is executed for each attention block", while its step 2 samples a fresh A "for this batch" and Section 5.6 asks only that mixing be refreshed "per batch (or more frequently)". If A is fresh per offloaded projection, Llama-2-7B pays 64 generations per forward pass, about 0.15 s of QR alone, and the economics collapse. If one A is reused across a whole forward pass, the adversary observes 64 to 160 matrices under a fixed mixing, which is precisely the accumulation setting Section 5.4 says the design avoids. Resolve this before sizing anything: the paper benchmarks one projection and never states which regime it measured.

The offload relieves compute, not memory, and that decides the fleet ratio. The feed-forward network, every non-linearity, softmax, the norms, and the un-mixed keys and values all run on the trusted side, so the confidential GPU still holds most of the weights and the whole KV cache. What the untrusted pool absorbs is a third of the per-token multiply-adds under multi-head attention and under a fifth under grouped-query attention. Size the confidential partition from memory and the untrusted pool from that residual FLOP share, and compare the result against simply buying more confidential capacity before committing: the scheme buys throughput per confidential GPU, not capacity to hold a larger model.

Instrument the two failures that are otherwise silent. Bind each generated A to a batch identifier and count generations per forward pass, so reuse shows up as a counter that stops incrementing rather than as a privacy loss nobody sees. Reject and count ill-conditioned matrices at generation time instead of clamping them. Neither failure produces an error on its own.

Treat the following as unbuilt rather than as configuration. There is no inference-engine integration; the paper defers vLLM and KV-cache work to future work and does not evaluate throughput. The KV cache is the specific gap: keys and values are what the cited LeftoverLocals and cache-sharing attacks actually read, and a scheme that protects the projection GEMM while leaving the resulting cache in plaintext on the untrusted device has moved the problem rather than solved it. Confirm where the cache lives before believing any deployment claim.

How to maintain it

Never reuse A. Reuse is the single failure that collapses the security argument from single-batch blind source separation to the classical fixed-mixing ICA problem, which is solvable with enough samples. A cache of generated matrices, a seeded generator that restarts deterministically, or a retry path that re-sends a failed batch with the same transform all reintroduce reuse.

Re-check the sensitive-layer exclusion list whenever the model changes, since it is expressed in layer indices. Re-measure the batch-size overhead curve after any change to the transport path, because its shape is set by the ratio of matrix generation to copy cost and both are hardware-specific. Verify cond(A) at generation time and reject rather than clamp: the block above shows the un-mixed result is destroyed at cond(A) = 10^4, and a silently ill-conditioned transform produces plausible but wrong logits.

Failure modes

  • Reused or predictable A. Cross-batch statistics become alignable and standard ICA applies. Symptom: none at runtime; the failure is silent.
  • Ill-conditioned A with low-precision returns. Logits drift without any error being raised. At cond(A) = 10^3 the relative error is already 13%.
  • Orthogonal A with weak or absent shields. The covariance and the token-similarity spectrum leak exactly. At low padding energy the paper measures ICA recovering the best 5% of tokens at 0.97 to 0.99 cosine similarity; it reports no p95 for the zero-padding case in text.
  • Anchor accumulation. If an adversary learns enough in-batch rows, A = U H^+ is solvable algebraically and the whole batch is de-mixed. The paper's own data shows anchor attacks improving sharply once anchors exceed roughly 70% of rows under shielding. The starting count is not zero: the paper measures 16.4% exactly duplicated embeddings over 10M vectors, falling to 0.148% once BOS and EOS are removed, and those special tokens sit at known positions in every request, so a batch begins with a handful of rows whose identity the adversary can guess for free.
  • Prefix caching re-enabled. Cache reuse defeats cross-user batch mixing and supplies exactly the repeated known tokens the flooding detector exists to suppress.
  • Batch size raised for throughput. Overhead climbs to 50% at 8192 as matrix generation dominates.
  • Applied to the first or last layer. Hidden states near the embedding or the LM head are closest to recoverable token identities.

Open questions and validation

  • No end-to-end serving measurement exists, and the obvious hope that a real interconnect rescues the transport number does not survive contact with the hardware. On Hopper the GPU TEE is confined to one GPU across PCIe and every byte between the confidential VM and the device is encrypted through AES-GCM bounce buffers, with cross-GPU peer paths unavailable in confidential mode (see GPU confidential computing). Each of the 64 to 160 crossings per forward pass therefore pays that path twice. Blackwell TEE-I/O removes the bottleneck, which makes this scheme a Blackwell-era design whose stated motivation, a fleet too poor in confidential silicon to serve privately, is the fleet least likely to have Blackwell.
  • The security evaluation is empirical negative evidence, not a proof. The authors say so. Absence of a successful off-the-shelf ICA attack is weaker than a bound.
  • The anchor-attack tables span two batch sizes and one of the three does not state which. Tables 7 and 8 declare n = 512 and n = 256; Table 6 declares no n, yet its accompanying text says recovery improves "when more than 90% of rows are anchors" at k = 240, which is consistent only with n = 256.4
  • Extension to the FFN is untested, and it is the extension that would make the offload fraction worth the round trips.
  • Whether the KV cache can be kept obfuscated at rest on the untrusted device, rather than just the projection that produces it, is unaddressed and is the difference between mitigating and solving the cited attacks.

References

Related: GPU confidential computing · Covariant obfuscation for private inference · Privacy-aware split inference · Membership inference against fine-tuned LLMs · Security, isolation and multi-tenancy · Gradient leakage in distributed training · Remote GPU verification · KV cache management · Disaggregated inference · Glossary

  1. arXiv:2603.05035v2, Table 3, batch 512. GELO column: A-generation 2.322 ms, mix 0.199 ms, GEMM 0.441 ms, un-mix 0.272 ms, copy 14.186 ms, total 17.420 ms. Baseline column: GEMM 0.443 ms, copy 14.123 ms, total 14.566 ms. The paper's reading is that overhead is 19.6% and that "The majority of time (∼ 81%) in both GELO and the baseline is spent on Copy (socket+I/O), indicating the experiment is bottlenecked by inter-process communication rather than GELO's computations." That is correct as stated. The observation this page adds is that the same table shows the offloaded work is 0.441 ms against 14.186 ms of transport, so within this microbenchmark the offload itself is a net loss regardless of obfuscation. Table 2 reports a slightly different pair at the same batch size (16.11 ms against 13.41 ms, 20.1%), so the two tables are separate runs. ↩

  2. arXiv:2603.05035v2, contributions list in Section 1: "showing that GELO offloads a large fraction of linear-algebra cost (e.g., ≈ 76% in a typical Llama 2 7B configuration) with modest overhead on the trusted side." The paper does not define the denominator. Appendix A, which is the only place the model's cost is broken down, gives "≈ 67 M" multiply-adds per token for Q/K/V/O and "≈ 135 M" for the FFN; the executed block above reproduces both, and the sharper 67.1M and 135.3M quoted elsewhere on this page are its arithmetic rather than the paper's figures. The Appendix A crossover of L = 24,658 for Llama-2-7B also reproduces from those figures. Its companion figure of "roughly L ≈ 49,000" for Llama-3-70B does not reproduce from that model's published config (hidden 8192, FFN 28672, 64 query heads over 8 key-value heads), which gives 52.2k accounting for grouped-query attention or 59.4k without it; the paper states neither the FFN width nor the attention layout it assumed. The stated value is almost exactly twice its own Llama-2-7B crossover (24,658 x 2 = 49,316), which suggests the figure was scaled by hidden size rather than recomputed. ↩

  3. arXiv:1912.09859v3, Section IV. "We invited ten student volunteers (five males and five females)"; for the spoken-digit case the volunteers' recognition rate on obfuscated samples is reported as 5%, 7%, 7%, and 4% for four ObfNets, against a 10% chance baseline. The generation procedure is in Section III-B: "the backend generates multiple sets of ObfNets ... and then transmits a unique set to each of the edge devices." The paper's stated collusion protection is against colluding edge devices that reveal which ObfNet they use, not against the backend, which trained every ObfNet and therefore holds all of the weights. For MNIST and sign language there is no volunteer study: Sections IV-B and IV-C report only the authors' own reading, "we cannot interpret the obfuscation results". The cross-task accuracy summary quoted on this page is from the abstract, "drops of generally within 1%". ↩

  4. arXiv:2603.05035v2, Section 4.3.3. Table 7 is captioned "median Gram error; n = 512" and Table 8 "median Gram error; n = 256". Table 6 carries no batch size, and its text reads "recovery quality for the remaining unknown tokens generally decreases markedly up to the point when more than 90% of rows are anchors." Its largest anchor count is k = 240, which is 94% of 256 but only 47% of 512, so Table 6 is consistent with n = 256 and not with the n = 512 of the table it is discussed alongside. The paper does not resolve this. ↩