Markdown

Tenant cache isolation and noisy-neighbor protection for shared inference¶

Scope: isolating and fair-sharing the KV/prefix cache across tenants in a shared LLM inference deployment, the two problems bundled under "cache multi-tenancy": confidentiality (can one tenant detect or recover another tenant's cached prompt via timing) and fair-share (can one tenant's cache traffic starve another tenant's cache regardless of compute priority). Distinct from inference QoS and admission control (the request-queue and compute-scheduling half of fairness) and from security and multi-tenancy (GPU-hardware isolation via MIG/vGPU); this page is what sits between them, at the cache layer.

Vendor flags, issue/PR status, and paper results below are quoted or paraphrased from the cited sources at research time; verify current status against the linked repository or paper before relying on it. The Python blocks are runnable and self-checking.

What it is¶

Two distinct problems get bundled under "cache multi-tenancy" in a shared inference deployment (multi-LoRA / multi-tenant serving, continuous batching):

Confidentiality. Automatic prefix caching and SGLang's RadixAttention share KV blocks across every request hitting an engine, keyed only by token content. If tenant B's request shares a prefix with tenant A's (a common system prompt, a leaked secret, another user's message), B's request can hit blocks A warmed. Even when the served content never crosses the tenant boundary, the fact of a cache hit is visible as a latency drop, and that is a timing side channel with demonstrated, not just theorized, exploits.
Fair-share (noisy neighbor). Even with confidentiality solved, the cache is one finite pool. A tenant issuing many large, unique-prefix requests can evict everyone else's warm cache or dominate scheduling, degrading others' latency with no security breach at all: a resource-fairness problem that request-priority scheduling does not automatically fix.

Why it matters¶

The confidentiality risk is proven, not hypothetical. "The Early Bird Catches the Leak" demonstrates timing side channels from shared KV and semantic caches in real LLM serving systems: an incremental search algorithm recovers a victim's system prompt token by token, averaging "112.72 queries per recovered token" against black-box production services, for "an average recovery accuracy of 89.0%."¹ The failure mode is not confined to a single engine's internals either: CacheProbe audits whether gateway architectures quietly collapse a provider's own per-account cache isolation, using OpenRouter (a shared-credential API gateway) as its case study for whether routing through one account can create cache sharing across all of a gateway's users, exactly the trap a multi-provider aggregator must not fall into.² The transport layer is not a free pass either: Whisper Leak shows that even TLS-encrypted, non-cached streaming responses leak conversation topic from packet-size and timing metadata alone, with "near-perfect classification (often >98% AUPRC)" across 28 LLMs from major providers, a reminder that cache isolation is one layer of a defense-in-depth problem, not the whole of it.³

The fairness risk is separate and just as real. Sheng et al. show that naive rate limiting either lets one client dominate or leaves the fleet underutilized, and propose VTC, a scheduler that tracks a virtual token counter per client and proves "a 2x tight upper bound on the service difference between two backlogged clients" while staying work-conserving.⁴ That is a compute-scheduling fairness result; it says nothing about whether a client's cached blocks survive contention. TokenCake measures the gap directly in multi-agent serving, where "spatial contention leads to the eviction of critical agents' caches," and its Spatial Scheduler (a reserved-plus-shared memory partition sized by a hybrid priority and usage metric) recovers "47.06% latency reduction and 16.9% GPU memory utilization improvement compared to vLLM."⁵ Resident KV Claims names the underlying gap precisely: when resident (already-cached) and active (in-flight) KV both need the same blocks, most serving stacks have no portable contract for which one survives, so resident data can be silently evicted with no accounting for what was lost or why.⁶

When to use it (and when not)¶

Use dedicated cache isolation and fair-share controls when:

Multiple distinct trust domains, separate customers, business units, or untrusted third-party agents, share one inference deployment's cache.
System prompts, few-shot exemplars, or retrieved context contain secrets or proprietary IP that must not be inferable by another tenant, even indirectly via timing.
SLA differentiation across tenants must hold under one tenant's traffic spike or thrash, not just under average load.

Do not over-build it when:

The deployment is genuinely single-tenant; there is no cache boundary to defend.
All "tenants" share one trust domain already (internal teams under one blanket security boundary) and the shared-cache efficiency win is worth the explicitly-accepted, negligible internal risk. That is a risk decision to make deliberately, not a default to fall into silently.
Tenant count is small enough that full physical isolation (separate deployments, or MIG partitions per tenant) is simpler and cheap enough; cache-level controls exist for when you need a shared cache's efficiency and multiple trust domains at once, and full isolation gives up the sharing benefit entirely.

Architecture¶

flowchart TB
  A["Tenant A request"] --> HASH["Block hash: parent + tokens + salt/tenant_id"]
  B["Tenant B request"] --> HASH
  N["Tenant N: flood of<br/>unique-prefix requests"] --> HASH
  HASH --> POOL["Shared KV / radix pool"]
  subgraph POOL2["Cache pool (partitioned)"]
    RES["Reserved region<br/>(critical tenant, protected)"]
    SHARE["Shared region<br/>(LRU/LFU, priority-evicted)"]
  end
  POOL --> RES
  POOL --> SHARE
  N -.->|"cannot evict"| RES
  N -->|"evicts under pressure"| SHARE
  RES --> OBS["Per-tenant hit-rate,<br/>eviction-count observability"]
  SHARE --> OBS

Requests are hashed into cache blocks with an isolation key (a salt or a tenant ID) before they ever touch the shared pool. The pool itself is split so a noisy tenant's flood can only contend for the shared region, never the reserved one. Both halves feed per-tenant observability, because a silent regression in either isolation or fairness is invisible without it.

How to close the confidentiality gap: salted or tenant-scoped cache keys¶

vLLM ships this today. Its RFC proposed injecting a cache_salt into the hash of a request's first block "so that only requests with the same salt can reuse cached KV blocks," explicitly to prevent "timing-based attacks where an adversary could infer cached content by observing latency differences."⁷ It shipped as PR #17045, "Prevent side-channel attacks via cache salting," merged April 30, 2025.⁸ The salt is folded into the block hash alongside the block's tokens, any LoRA ID, and any multimodal input hash⁹: a base64-encoded 256-bit string, never passed to the model itself.⁷ Two sessions can deliberately share a salt (a "trust group") to keep the caching win for co-located work, while anyone without that salt is isolated from it:

# REFERENCE TEMPLATE (needs vllm >= the cache_salt release) - not run here.
# Pin the vLLM version and verify the field name against your installed release.
import requests

# Two different tenants hitting the same shared system prompt: each supplies its
# own salt, so neither can hit blocks the other warmed, and neither can time-probe
# whether the other's prefix is resident.
requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "system", "content": SYSTEM_PROMPT},
                 {"role": "user", "content": "..."}],
    "cache_salt": "tenant-9f2c-256bit-random",   # unique per tenant, or per trust group
})

SGLang ships this too, via a different path than its own tracked feature request. The request that named this need explicitly, "Support Cache Salting for secure prefix caching," cited the same vLLM RFC and was closed inactive with no PR linked to it.¹⁰ But a separate, untracked PR shipped cache_salt and extra_key fields on request models well before that closure, computing a combined extra-cache-key the same way vLLM's salt isolates requests.¹¹ A closed-inactive feature-request issue does not reliably reflect what actually shipped; verify the field name and behavior against your installed SGLang version rather than assuming the tracked issue is the last word. NVIDIA Dynamo's KV router offers a complementary, engine-independent isolation point at the routing layer regardless of which engine sits behind it: it "maintains one radix tree per (model_name, tenant_id) pair," with tenant_id optional and defaulting to "default", so "queries against one model/tenant never return scores from another."¹² Useful for isolating routing and cache-locality scoring per tenant when you cannot verify or trust an engine's own salt implementation, not a workaround for an engine that lacks one.

Do not salt indiscriminately. Blanket per-request isolation throws away the cross-request sharing that justified a shared deployment in the first place, which is why SafeKV and PrefixWall exist: SafeKV classifies content sensitivity asynchronously and applies selective, radix-tree-scoped isolation only where needed, holding "up to 40.58% lower TTFT overhead and up to 2.66x higher throughput compared to complete cache isolation."¹³ PrefixWall takes the same selective-isolation approach inside the serving engine itself, extending vLLM's prefix-cache internals rather than sitting at a gateway.¹⁴ Salt what must not leak; let the rest share.

The mechanism itself is simple enough to verify directly. This runnable block reproduces vLLM's block-hash scheme, confirms two unsalted tenants collide (the leak), confirms salted tenants do not, confirms a deliberate trust-group salt still shares, and confirms an attacker cannot brute-force a victim's salt from a small guess set:

# cache_salt_isolation.py -- validated: per-tenant salting isolates prefix-cache hits
# and collapses the hit/miss timing side channel across tenants; hashlib only.
import hashlib

def block_hash(parent_hash: bytes, tokens: tuple, salt: bytes = b"") -> bytes:
    """vLLM-style block hash: parent hash + block tokens + extra hash (salt)."""
    payload = parent_hash + repr(tokens).encode() + salt
    return hashlib.sha256(payload).digest()

def hash_chain(token_blocks, salt: bytes = b"") -> list:
    parent = b""
    chain = []
    for block in token_blocks:
        h = block_hash(parent, block, salt)
        chain.append(h)
        parent = h
    return chain

PREFIX = [(1, 2, 3, 4), (5, 6, 7, 8)]   # shared system-prompt prefix, two blocks

# Two tenants, no isolation: identical prefix -> identical hash chain -> cache HIT.
chain_no_salt_A = hash_chain(PREFIX)
chain_no_salt_B = hash_chain(PREFIX)
assert chain_no_salt_A == chain_no_salt_B   # unsalted: cross-tenant reuse (the leak)

# Two tenants, salted per-tenant: identical prefix, different salt -> no collision.
salt_A, salt_B = b"tenant-A-256bit", b"tenant-B-256bit"
chain_A = hash_chain(PREFIX, salt_A)
chain_B = hash_chain(PREFIX, salt_B)
assert chain_A != chain_B                     # salted: isolated, no cross-tenant hit
assert chain_A == hash_chain(PREFIX, salt_A)  # same tenant still hits its own cache

# Trust group: two sessions sharing one deliberate group salt still reuse each other's cache.
group_salt = b"team-shared-prompt-cache"
assert hash_chain(PREFIX, group_salt) == hash_chain(PREFIX, group_salt)

# Adversarial: an attacker who does not know tenant B's salt cannot construct a
# colliding chain by brute-force guessing a 256-bit salt from a small dictionary.
guesses = [b"guess-1", b"guess-2", b"tenant-A-256bit", b"admin", b"default"]
assert not any(hash_chain(PREFIX, g) == chain_B for g in guesses if g != salt_B)

# Timing side channel model: distinguishing "is this prefix cached" degenerates to
# random guessing once every tenant salts, because the attacker's own (wrong-salt)
# probe never collides with the victim's resident chain.
def probe_is_hit(probe_chain, resident_chains) -> bool:
    return probe_chain in resident_chains

resident = {tuple(chain_B)}
attacker_probe = hash_chain(PREFIX, b"attacker-guess")
assert not probe_is_hit(tuple(attacker_probe), resident)   # attacker sees only misses
victim_probe = hash_chain(PREFIX, salt_B)
assert probe_is_hit(tuple(victim_probe), resident)         # tenant B still hits its own

print("OK: unsalted cross-tenant collision reproduced; salting isolates hits; "
      "trust-group sharing preserved; brute-force guesses miss; attacker probe blind")

Run output:

OK: unsalted cross-tenant collision reproduced; salting isolates hits; trust-group sharing preserved; brute-force guesses miss; attacker probe blind

How to close the fairness gap: priority is not cache protection¶

Inference QoS and admission control already covers the compute half: priority classes, admission control, and latency-aware routing at the request-queue level, including vLLM's --scheduling-policy priority and SGLang's --enable-priority-scheduling. Neither, by default, protects a tenant's cached blocks. SGLang's eviction policy is set independently via --radix-eviction-policy (lru or lfu); a request's priority can save it from being preempted out of the running batch, but the priority-scheduling flags do not make cache eviction priority-aware.¹⁶ That is exactly the gap Resident KV Claims names: no portable contract for which cached data survives contention, so a high-priority tenant's warm cache can still be silently evicted by a low-priority neighbor's flood.⁶

Two complementary fixes exist, at different scopes:

Token-cost fairness across the request queue. VTC's virtual-token-counter scheduler is the reference algorithm if you need a provable bound (2x service-difference between backlogged clients) rather than a static integer priority.⁴ It is not, to date, a built-in vLLM or SGLang scheduling policy; treat it as the design to adopt at your gateway if fairness must be provably bounded rather than merely configured.
Reserved cache-memory partitions. TokenCake's Spatial Scheduler splits the KV pool into a shared region and a reserved region sized by a hybrid priority-and-usage metric, so a noisy tenant's flood can only contend for the shared region.⁵ The block below models the same shape: a plain shared LRU pool where a flood evicts a "protected" tenant entirely despite that tenant's priority, against a reserved partition where the protected tenant's blocks are untouchable by any flood size.

# kv_noisy_neighbor.py -- validated: request priority alone does not protect resident
# KV blocks from a noisy neighbor's eviction pressure under plain LRU; a reserved
# partition (TokenCake-style) bounds it. Standard library only (OrderedDict as LRU).
from collections import OrderedDict

def run_plain_lru(pool_capacity, protected_blocks, flood_blocks):
    """Single shared LRU pool. protected_blocks load once, then flood evicts under LRU."""
    pool = OrderedDict()
    for b in protected_blocks:
        pool[b] = True
    for b in flood_blocks:
        if b in pool:
            pool.move_to_end(b)
            continue
        if len(pool) >= pool_capacity:
            pool.popitem(last=False)         # evict LRU victim, priority-blind
        pool[b] = True
    resident = set(pool.keys())
    return sum(1 for b in protected_blocks if b in resident), len(protected_blocks)

def run_reserved_partition(pool_capacity, reserved_frac, protected_blocks, flood_blocks):
    """Partition the pool: a reserved region sized for the protected set, a shared
    region for everyone else. The noisy flood can only evict within the shared region."""
    reserved_cap = max(len(protected_blocks), int(pool_capacity * reserved_frac))
    shared_cap = pool_capacity - reserved_cap
    reserved = OrderedDict((b, True) for b in protected_blocks)
    shared = OrderedDict()
    for b in flood_blocks:
        if b in shared:
            shared.move_to_end(b)
            continue
        if len(shared) >= shared_cap:
            shared.popitem(last=False)
        shared[b] = True
    hits = sum(1 for b in protected_blocks if b in reserved)
    return hits, len(protected_blocks)

PROTECTED = [f"p{i}" for i in range(10)]         # tenant P's warm system-prompt blocks
FLOOD = [f"n{i}" for i in range(500)]             # tenant N: 500 unique-prefix requests

# Plain LRU, capacity barely larger than P's working set: the flood evicts P entirely.
hit_lru, total = run_plain_lru(pool_capacity=12, protected_blocks=PROTECTED, flood_blocks=FLOOD)
assert hit_lru == 0, f"expected total eviction under plain LRU, got {hit_lru}/{total}"

# Reserved partition, same total capacity: P is fully protected regardless of flood size.
hit_res, total = run_reserved_partition(pool_capacity=12, reserved_frac=0.5,
                                         protected_blocks=PROTECTED, flood_blocks=FLOOD)
assert hit_res == total == 10, f"reserved partition must fully protect P, got {hit_res}/{total}"

# Adversarial: an even larger flood (10x) still cannot touch the reserved region.
hit_res2, _ = run_reserved_partition(pool_capacity=12, reserved_frac=0.5,
                                      protected_blocks=PROTECTED, flood_blocks=FLOOD * 10)
assert hit_res2 == 10

# Boundary: with reserved_frac=0, the reserved-policy code path still seeds the
# protected set directly into "reserved" ahead of any flood, so it is never worse
# than plain LRU (here it remains a strict improvement, not merely non-negative).
hit_zero, _ = run_reserved_partition(pool_capacity=12, reserved_frac=0.0,
                                      protected_blocks=PROTECTED, flood_blocks=FLOOD)
assert hit_zero >= hit_lru

print(f"OK: plain LRU protects {hit_lru}/{total} of P under flood (noisy-neighbor "
      f"eviction reproduced); reserved partition protects {hit_res}/{total}, "
      f"holds under a 10x flood ({hit_res2}/{total})")

Run output:

OK: plain LRU protects 0/10 of P under flood (noisy-neighbor eviction reproduced); reserved partition protects 10/10, holds under a 10x flood (10/10)

Fleet-level placement. Beyond one replica, cache locality and load balancing pull against each other: keep a tenant's traffic on the replica holding its warm prefix, but do not let that concentrate all of one tenant's load onto a single GPU. Preble co-optimizes both with a two-level (global request scheduler plus per-GPU iteration scheduler) architecture, layered on top of vLLM and SGLang rather than replacing them, cutting average latency "1.5X to 14.5X" and P99 latency "2X to 10X" against locality-blind or load-blind baselines.¹⁵

Failure modes¶

Multi-tenant prefix caching with no salt or tenant key. Silent cross-tenant timing leak; Early Bird demonstrated recovering system prompts and other users' prompts this way, at production scale.¹
Trusting a gateway's "per-account caching" claim without testing it. CacheProbe's OpenRouter case study shows a shared-credential gateway can collapse a provider's own isolation guarantee; verify it holds across your own aggregation layer, the same way you would reconcile aggregator billing leaks.²
Salting every request indiscriminately. Throws away the sharing benefit that justified the shared deployment; SafeKV's measured cost of full isolation (up to 40.58% worse latency, up to 2.66x less throughput than selective isolation) is the price of over-isolating.¹³
Confusing request priority with cache protection. A tenant's requests can hold top scheduling priority and still lose their warm cache to a noisy neighbor's eviction pressure, because --scheduling-policy priority / --enable-priority-scheduling govern the request queue, not --radix-eviction-policy or vLLM's block eviction.¹⁶
Ignoring the network side channel. Salting the cache and encrypting the transport still leave packet-size and timing metadata on streaming responses exploitable, per Whisper Leak; pad or batch tokens on the wire if topic confidentiality matters as much as content confidentiality.³

Open questions & validation¶

Whether your inference stack's isolation key (vLLM's or SGLang's cache_salt, or a Dynamo tenant_id at the router) is actually wired end to end from your tenant-auth layer to the engine, not merely available as a flag nobody sets.
Whether a synthetic canary probe (two tenants, one shared secret prefix, a timing measurement) can distinguish a cache hit from a miss on your own deployment, the same methodology CacheProbe and Early Bird used against production services.
Whether your own gateway or aggregator preserves upstream per-account cache isolation, tested the way CacheProbe tested OpenRouter, not merely assumed from the provider's documentation.
The measured cost of full per-tenant salting on your traffic (the throughput and latency delta versus shared caching) against your actual risk tolerance, since blanket isolation is not free and selective isolation (SafeKV, PrefixWall) exists because of that cost.
Whether a reserved cache-memory partition is sized to your actual protected working set, and re-validated after traffic-mix changes, the same way KV-cache management's tiering must be re-sized when context length changes.

References¶

vLLM, RFC #16016, "Cache Salting for Secure and Flexible Prefix Caching in vLLM": https://github.com/vllm-project/vllm/issues/16016
vLLM, PR #17045, "[Core] Prevent side-channel attacks via cache salting" (merged): https://github.com/vllm-project/vllm/pull/17045
vLLM, Automatic Prefix Caching design doc (block hash composition, cache_salt, LoRA/multimodal extra hashes): https://docs.vllm.ai/en/stable/design/prefix_caching/
SGLang, issue #9163, "Support Cache Salting for secure prefix caching" (closed, inactive): https://github.com/sgl-project/sglang/issues/9163
SGLang, PR #10718, "feat: add cache_salt support to request" (merged): https://github.com/sgl-project/sglang/pull/10718
SGLang, Server Arguments (--enable-priority-scheduling, --priority-scheduling-preemption-threshold, --radix-eviction-policy): https://github.com/sgl-project/sglang/blob/main/docs/advanced_features/server_arguments.md
NVIDIA Dynamo, Standalone KV Indexer (tenant_id, per-(model_name, tenant_id) radix trees): https://docs.nvidia.com/dynamo/dev/components/router/standalone-indexer
Song, Pang, Wang et al., "The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems," IEEE TIFS: https://arxiv.org/abs/2409.20002
McDonald and Bar Or, "Whisper Leak: A Side-Channel Attack on Large Language Models": https://arxiv.org/abs/2511.03675
Fahey, "CacheProbe: Auditing Prompt Cache Isolation in Gateway APIs": https://arxiv.org/pdf/2605.30613
Pennas, Papaioannou, Guarnieri, and Doudali, "PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systems": https://arxiv.org/abs/2603.10726
Chu et al., "Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference" (SafeKV): https://arxiv.org/abs/2508.08438
Sheng, Cao, Li, Zhu, Li, Zhuo, Gonzalez, and Stoica, "Fairness in Serving Large Language Models" (VTC), OSDI 2024: https://arxiv.org/abs/2401.00588
Srivatsa, He, Abhyankar, Li, and Zhang, "Preble: Efficient Distributed Prompt Scheduling for LLM Serving": https://arxiv.org/abs/2407.00023
Bian, Wu, Li, Ma, and Zhuo, "TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications": https://arxiv.org/abs/2510.18586
Stepanek, "Resident KV Claims: A Conformance Contract for Future Reuse under Active KV Pressure": https://arxiv.org/abs/2605.24259

Song, Pang, Wang, Wang, Wang, Chen, Song, Jin, Meng, and Hou, "The Early Bird Catches the Leak": timing side channels from shared KV and semantic caches let an attacker recover system prompts and other users' prompts; an incremental token-by-token search averages 112.72 queries per recovered token at 89.0% average recovery accuracy against production black-box services. Accepted, IEEE Transactions on Information Forensics and Security. https://arxiv.org/abs/2409.20002 ↩↩
Fahey, "CacheProbe": audits whether gateway architectures (OpenRouter, a shared-credential API gateway, as the case study) inadvertently collapse a provider's per-account prompt-cache isolation into cache sharing across all of a gateway's users. https://arxiv.org/pdf/2605.30613 ↩↩
McDonald and Bar Or, "Whisper Leak": exploits packet-size and timing metadata on encrypted streaming LLM responses to classify conversation topic, "near-perfect classification (often >98% AUPRC)" across 28 LLMs from major providers, with defenses (random padding, token batching, packet injection) each reducing but not eliminating the leak. https://arxiv.org/abs/2511.03675 ↩↩
Sheng et al., "Fairness in Serving Large Language Models": VTC (Virtual Token Counter), a fair scheduler on top of continuous batching that tracks per-client input and output token cost and prioritizes the lowest counter; proves a 2x tight upper bound on service difference between two backlogged clients while remaining work-conserving. OSDI 2024. https://arxiv.org/abs/2401.00588 ↩↩
Bian, Wu, Li, Ma, and Zhuo, "TokenCake": in multi-agent LLM serving, spatial contention evicts critical agents' caches while temporally idle agents hold memory hostage; the Spatial Scheduler partitions GPU KV-cache memory into a shared pool and a reserved pool via dynamic partitioning guided by a hybrid priority-and-runtime-state metric, yielding 47.06% latency reduction and 16.9% GPU memory utilization improvement over vLLM. https://arxiv.org/abs/2510.18586 ↩↩
Stepanek, "Resident KV Claims": names the missing contract when resident (cached) and active (in-flight) KV compete for the same blocks under memory pressure; without an explicit protected-claim mechanism, resident data is silently evicted with no accounting for what was lost or why. https://arxiv.org/abs/2605.24259 ↩↩
vLLM, RFC #16016 (opened by dr75, April 3 2025): proposes injecting an optional cache_salt into the hash of a request's first KV block so cache reuse is restricted to requests sharing the same salt, preventing timing-based inference of cached content across tenants; specifies the salt as a base64-encoded 256-bit string that is never passed to the model; recommends shipping the single-barrier design first. https://github.com/vllm-project/vllm/issues/16016 ↩↩
vLLM, PR #17045, "[Core] Prevent side-channel attacks via cache salting," merged April 30 2025 (DarkLight1337, comaniac, russellb): implements the single-barrier cache_salt design from RFC #16016. https://github.com/vllm-project/vllm/pull/17045 ↩
vLLM, Automatic Prefix Caching design doc: a block hash is computed from the parent block's hash, the block's tokens, and "extra hashes" (LoRA IDs, multimodal input hashes, and cache salts) needed to make the block unique per tenant. https://docs.vllm.ai/en/stable/design/prefix_caching/ ↩
SGLang, issue #9163 (opened by pyc96, August 13 2025, referencing vLLM's RFC #16016): requests an equivalent single-barrier cache-salting mechanism for SGLang; closed November 10 2025 with an "Inactive" label and no PR linked to this specific issue, seven weeks after the capability had already shipped separately: see ¹¹. https://github.com/sgl-project/sglang/issues/9163 ↩
SGLang, PR #10718, "feat: add cache_salt support to request," merged September 24 2025: introduces cache_salt and extra_key fields on request models and a _compute_extra_key method that concatenates them into a combined cache-isolation key; confirmed present in the current codebase (io_struct.py, serving_base.py) as of this writing. https://github.com/sgl-project/sglang/pull/10718 ↩↩
NVIDIA Dynamo, Standalone KV Indexer docs: the indexer maintains one radix tree per (model_name, tenant_id) pair; tenant_id is optional on /register and /query and defaults to "default"; workers registered under different model/tenant pairs are isolated into separate indexers, so queries against one never score against another. https://docs.nvidia.com/dynamo/dev/components/router/standalone-indexer ↩
Chu et al., "Selective KV-Cache Sharing to Mitigate Timing Side-Channels" (SafeKV): an asynchronous sensitivity-classification pipeline plus a radix-tree memory manager with sensitivity-aware eviction and a Reuse-Diversity-Ratio runtime guard, giving up to 40.58% lower TTFT overhead and up to 2.66x higher throughput than isolating every request, while still constraining leakage. https://arxiv.org/abs/2508.08438 ↩↩
Pennas, Papaioannou, Guarnieri, and Doudali, "PrefixWall": monitors prefix-cache reuse patterns and selectively isolates only the prefixes that need it, mitigating the same timing side channel at lower performance cost than isolating the whole cache. https://arxiv.org/abs/2603.10726 ↩
Srivatsa, He, Abhyankar, Li, and Zhang, "Preble": a distributed prompt-scheduling layer on top of (lightly modified) vLLM and SGLang, with a global request-level scheduler and a per-GPU iteration-level scheduler that co-optimize KV-cache locality against load balancing across replicas; evaluated on Mistral 7B and Llama-3 70B over four-A6000 and eight-H100 clusters, cutting average latency 1.5 to 14.5x and P99 latency 2 to 10x versus baselines that optimize only one objective. https://arxiv.org/abs/2407.00023 ↩
SGLang, Server Arguments doc: --enable-priority-scheduling reorders the request queue by an integer priority with --priority-scheduling-preemption-threshold gating when an arrival preempts a running request; --radix-eviction-policy (lru or lfu) governs which cached blocks are evicted under memory pressure, independently of request priority. https://github.com/sgl-project/sglang/blob/main/docs/advanced_features/server_arguments.md ↩↩