Copy for agents Markdown

Serving hybrid recurrent plus full-attention models

Scope: the serving-engine consequences of interleaving linear or recurrent attention layers with periodic full-attention layers. A hybrid stack forces a cache manager to hold two memory shapes at once, breaks the assumptions ordinary prefix caching is built on, and turns cache retention into an explicit policy decision. This page covers the mechanism, the two retention policies that make it affordable, and how the state crosses a disaggregation boundary. General cache mechanics are in KV cache management and KV cache fundamentals; the transfer layer is in KV cache transfer with NIXL.

The engine behaviour described here is vLLM's, as documented in its Kimi K3 day-0 release post (2026-07-27) and the merged pull requests it cites; every PR number below was checked against the repository on 2026-07-30 and each is merged. Performance figures are vLLM's own published measurements, not reproductions. The Python block is this page's own cache-economics model, executed and asserted; it is a cost model for choosing retention policy, not a simulation of vLLM's allocator.

flowchart TB
  SCHED["Single scheduler + hybrid KV-cache manager"]
  SCHED --> PAGED["Paged KV blocks<br/>(full-attention layers)<br/>grows with sequence length"]
  SCHED --> STATE["Recurrent state blocks<br/>(linear-attention layers)<br/>constant size, updated in place"]
  PAGED --> MATCH["Prefix match: per-token, block-granular"]
  STATE --> SNAP["Snapshot registry: copy-before-extend<br/>at selected checkpoints only"]
  MATCH --> REUSE["Reusable prefix length"]
  SNAP --> REUSE

What it is

A hybrid model replaces most of its full-attention layers with a linear-attention variant that carries a fixed-size recurrent state instead of a KV cache that grows with the sequence, then keeps a small number of full-attention layers at intervals to preserve exact global recall. Kimi Delta Attention (KDA) in Kimi K3 is the current large-scale example, but the same shape appears in Mamba-style hybrids and in sliding-window designs. The economics are what make a 1M-token context window affordable: most layers stop paying per-token memory.

For a serving engine this is not a model detail, it is an allocator change. Two kinds of memory now coexist under one scheduler:

• Paged KV blocks for the full-attention layers, matched per token and reusable across requests exactly as in a conventional engine.
• Recurrent state blocks for the linear layers, one constant-size block per sequence, updated in place on every token.

That last property is the whole problem. A paged KV cache is append-only, so any prefix of it is trivially reusable. A recurrent state is destructively overwritten, so reusing a prefix requires that someone took a snapshot at exactly that boundary before the next token overwrote it. Snapshotting at every position is unaffordable, because one state block is large. The engine therefore has to choose, in advance, which boundaries are worth preserving.

vLLM's answer decouples the large physical state blocks from fine-grained prefix matching: it registers state snapshots within those blocks and copies them before extension, so a long shared prompt can reuse both the recurrent state and the paged KV. The engine's release notes describe this hybrid-cache machinery as new in vLLM core and applicable to every hybrid linear model, not only to the model that motivated it.¹

Why use it

For the model architect the case is context length. For the cluster operator the case is that per-token memory stops being the binding constraint on how many long sessions fit in HBM, which changes capacity planning for agentic workloads more than any single kernel optimization does.

The executed model below quantifies the crossover. A constant-size recurrent state is expensive relative to what it replaces at short context and nearly free at long context. At the modelled 3 MiB per state and 1 KiB per token of compressed full-attention KV, the state costs as much as the KV for about 3,072 tokens; by 1M tokens it costs 0.3% as much. Below the crossover, a hybrid stack is carrying memory it does not need. Far above it, the recurrent layers cost almost nothing and the full-attention layers dominate.

This is also why the retention question is unavoidable rather than an optimization. Caching every recurrent state would exhaust a distributed cache pool quickly, because each checkpoint is far larger than a single token's KV.

When to use it (and when not)

Reach for a hybrid stack when:

• Contexts routinely exceed the crossover by a wide margin. Agentic sessions spanning hundreds of thousands of tokens are the intended workload.
• Sessions are long-lived and reuse their own prefixes. Multi-turn agents that replay the previous turn's prompt get the highest return from prompt-boundary checkpoints.
• HBM, not compute, is what limits concurrency. The saving is memory-shaped.

Do not reach for it when:

• Contexts are short. Below the crossover the constant state is pure overhead.
• Prefix reuse is the dominant win and prefixes are highly varied. Ordinary paged KV reuses any prefix for free; recurrent state only reuses boundaries someone chose to keep.
• You need exact-match rollback semantics. In-place updates preclude rolling back a partial overlap, which is the constraint the Marconi work identifies as forcing exact-match hits in the naive design.²
• Your engine version predates hybrid-cache support. This machinery is recent; see the version notes below.

Architecture: two memory shapes, one scheduler

sequenceDiagram
  participant R as "Request N+1"
  participant S as "Scheduler"
  participant G as "GPU cache"
  participant X as "External store"
  R->>S: "prompt with a long shared prefix"
  S->>G: "match paged KV (per token)"
  S->>X: "look up longer prefix"
  G-->>S: "local reusable = 2,047 tokens (ends mid-block)"
  X-->>S: "remote reusable = 2,048 tokens"
  S->>S: "compare exact token counts, take the longer"
  S->>G: "release block reserved for the shorter local tail"
  S->>S: "reconcile all cache groups to the new prefix length"
  Note over S,G: "recurrent state reused only if a snapshot exists at that boundary"

Three details in that flow are easy to get wrong and each has a corresponding fix upstream:

1. Compare tiers in tokens, not blocks. A local hit can end inside a physical block. Rounding it down to a block boundary before comparing against a remote hit mis-ranks the tiers. The executed model shows a 2,047-token local hit rounding to 1,984 under 64-token blocks, which flips the comparison against a 2,048-token remote hit.
2. Release the reserved tail when the remote tier wins. The block held for the shorter local tail has to be freed and every cache group reconciled to the new prefix length, or the two tiers disagree about where the prefix ends.
3. Keep it in the connector API, not the model path. vLLM implemented multi-tier partial-prefix reuse entirely through existing KV connector semantics, so external stores gain it without model-specific integration.³

How to use it: choose a retention policy

Retention is the one decision a hybrid stack forces on an operator, and there are two complementary policies. Neither is a tuning detail; they answer different questions.

Interval-based retention treats fixed positions as checkpoints, for example one every 32K tokens, and additionally retains prompt boundaries automatically. Prompt ends are the better checkpoints in agentic traffic, because the next turn usually begins by replaying the previous turn's prompt. In vLLM this is VLLM_PREFIX_CACHE_RETENTION_INTERVAL; setting it to 0 disables periodic checkpoints and keeps only prompt-end states, which suits multi-turn conversation. Larger intervals trade recomputation for lower cache usage.¹

Marconi-style selective retention answers a different question: which unanticipated prefixes are worth keeping. A system prompt, repository snapshot, or tool specification may be shared across many requests without aligning to any prompt boundary. The rule is cache on the second hit. The first sighting proves the prefix exists; the second proves it is shared. Only then is cache capacity spent.²

Run both. Interval retention checkpoints structurally important boundaries; selective retention learns which other prefixes deserve capacity.

import numpy as np

# --- 1. Constant recurrent state vs growing KV: where the crossover sits ----
# A full-attention layer stores per-token KV that grows linearly. A recurrent
# (KDA/Mamba-style) layer stores a fixed-size state regardless of length.

MLA_BYTES_PER_TOKEN = 1024.0     # per full-attention layer, compressed latent KV
KDA_STATE_BYTES = 3.0 * 2**20    # per recurrent layer, constant


def crossover_tokens(state_bytes: float, kv_bytes_per_token: float) -> float:
    """Sequence length at which one recurrent state costs as much as the KV."""
    assert state_bytes > 0 and kv_bytes_per_token > 0
    return state_bytes / kv_bytes_per_token


x = crossover_tokens(KDA_STATE_BYTES, MLA_BYTES_PER_TOKEN)
print(f"[1] one recurrent state = {KDA_STATE_BYTES / 2**20:.1f} MiB = KV for {x:,.0f} tokens")
assert 1_000 < x < 10_000, "vendor description: 'the MLA cache for a few thousand tokens'"

# Below the crossover a checkpoint is EXPENSIVE relative to what it saves; far
# above it, the state is nearly free. This is why naive per-token checkpointing
# of recurrent state is unaffordable while per-token KV caching is routine.
for n in (1_000, 3_072, 32_768, 1_000_000):
    ratio = KDA_STATE_BYTES / (n * MLA_BYTES_PER_TOKEN)
    verdict = "state dominates" if ratio > 1 else "KV dominates"
    print(f"[1] at {n:>9,} tokens: state/KV = {ratio:6.3f}  {verdict}")
assert KDA_STATE_BYTES / (1_000_000 * MLA_BYTES_PER_TOKEN) < 0.01

# --- 2. Interval-based retention: budget forces the interval ----------------
# Checkpoint every I tokens over a T-token context. Storage is (T/I) states.
# A request resuming at a uniformly random position recomputes I/2 tokens.


def interval_plan(T: int, I: int, state_bytes: float) -> tuple[float, float]:
    assert T >= I >= 1
    n_ckpt = T // I
    return n_ckpt * state_bytes / 2**30, I / 2.0  # GiB stored, expected recompute tokens


print()
T = 1_000_000
for I in (4_096, 32_768, 131_072, 1_000_000):
    gib, recomp = interval_plan(T, I, KDA_STATE_BYTES)
    print(f"[2] interval {I:>9,} -> {gib:8.2f} GiB of state, "
          f"{recomp:>9,.0f} tokens expected recompute")
gib_dense, _ = interval_plan(T, 4_096, KDA_STATE_BYTES)
gib_sparse, _ = interval_plan(T, 131_072, KDA_STATE_BYTES)
assert gib_dense > 32 * gib_sparse * 0.9, "storage scales as 1/I"

# The tradeoff has an interior optimum once storage carries an opportunity cost.
# Total cost = storage_bytes * alpha + expected_recompute_tokens * beta.
# d/dI = 0 gives I* = sqrt(2 * T * S * alpha / beta).
ALPHA = 1.0e-9   # cost per byte held
BETA = 1.0e-3    # cost per token recomputed


def total_cost(T: int, I: float, S: float) -> float:
    return (T / I) * S * ALPHA + (I / 2.0) * BETA


I_star = np.sqrt(2 * T * KDA_STATE_BYTES * ALPHA / BETA)
grid = np.linspace(1_000, 400_000, 4_000)
I_num = grid[np.argmin([total_cost(T, i, KDA_STATE_BYTES) for i in grid])]
print(f"[2] closed-form optimum I* = {I_star:,.0f} tokens; "
      f"grid search minimum = {I_num:,.0f} tokens")
assert abs(I_star - I_num) / I_star < 0.02, "closed form must match the grid"

# Adversarial: the optimum is a function of the cost ratio, not a constant.
# A 100x cheaper recompute pushes checkpoints 10x further apart (sqrt scaling).
BETA_CHEAP = 1.0e-5
I_star_cheap = np.sqrt(2 * T * KDA_STATE_BYTES * ALPHA / BETA_CHEAP)
print(f"[2] recompute 100x cheaper -> I* = {I_star_cheap:,.0f} tokens "
      f"({I_star_cheap / I_star:.1f}x further apart)")
assert np.isclose(I_star_cheap / I_star, 10.0, rtol=1e-6), "sqrt(1/beta) scaling"

# Setting the interval to 0 (prompt-end checkpoints only) is NOT the same as
# a huge interval: it is a different, content-aligned checkpoint set.
print("[2] interval=0 keeps prompt-end states only; correctness is identical, "
      "reuse depends on whether turns replay prior prompts")

# --- 3. Marconi-style admission: cache on the SECOND sighting ---------------
# Workload: many one-off prefixes plus a few genuinely shared ones.
print()
N_ONEOFF, N_HOT, HOT_REPEATS = 800, 40, 12


def admission(policy: str) -> tuple[int, int]:
    """Return (entries stored, cache hits) for one pass over the workload."""
    assert policy in {"always", "second-hit"}
    stored = hits = 0
    if policy == "always":
        stored = N_ONEOFF + N_HOT
        hits = N_HOT * (HOT_REPEATS - 1)       # miss once, then hit
    else:
        stored = N_HOT                          # one-offs never reach a 2nd sighting
        hits = N_HOT * (HOT_REPEATS - 2)        # miss twice, then hit
    return stored, hits


for pol in ("always", "second-hit"):
    stored, hits = admission(pol)
    gib = stored * KDA_STATE_BYTES / 2**30
    print(f"[3] {pol:11s} -> {stored:4d} entries ({gib:6.2f} GiB), {hits:4d} hits, "
          f"{hits / stored:5.2f} hits per stored entry")

s_always, h_always = admission("always")
s_second, h_second = admission("second-hit")
# The cost of the policy is exactly one lost hit per hot prefix, and nothing else.
assert h_always - h_second == N_HOT, "second-hit forfeits exactly one hit per shared prefix"
# The benefit is that every one-off prefix is kept out of the cache entirely.
assert s_always - s_second == N_ONEOFF, "one-off prefixes are never admitted"
print(f"[3] cost: {h_always - h_second} lost hits; "
      f"benefit: {s_always - s_second} one-off entries never admitted "
      f"({(s_always - s_second) * KDA_STATE_BYTES / 2**30:.2f} GiB saved)")
assert h_second / s_second > h_always / s_always, "second-hit wins on hits per byte"

# Adversarial: the policy LOSES when prefixes are reused exactly twice, because
# it admits on the second sighting and then never gets a third to cash in.
def admission_k(n_oneoff: int, n_hot: int, k: int) -> tuple[float, float]:
    always = n_hot * (k - 1) / (n_oneoff + n_hot)
    second = n_hot * max(k - 2, 0) / max(n_hot, 1)
    return always, second


a2, s2 = admission_k(800, 40, 2)
print(f"[3] adversarial k=2 (each prefix seen twice): always {a2:.3f} hits/entry, "
      f"second-hit {s2:.3f} hits/entry -> second-hit is strictly worse")
assert s2 == 0.0 and a2 > 0.0, "cache-on-second-hit collects nothing when k=2"

# --- 4. Multi-tier partial-hit reconciliation ------------------------------
# A local GPU hit may end mid-block; a remote store may hold a longer prefix.
# The scheduler must compare exact reusable token counts and take the longer.
print()
BLOCK = 64


def reusable(local_tokens: int, remote_tokens: int) -> tuple[str, int]:
    assert local_tokens >= 0 and remote_tokens >= 0
    if remote_tokens > local_tokens:
        return "remote", remote_tokens
    return "local", local_tokens


CASES = [(1_000, 4_096), (4_096, 1_000), (2_048, 2_048), (2_047, 2_048)]
for loc, rem in CASES:
    tier, n = reusable(loc, rem)
    naive = loc  # a local-first policy that never re-checks the remote tier
    print(f"[4] local {loc:>5} vs remote {rem:>5} -> take {tier:6s} ({n:>5} tokens); "
          f"local-first policy would reuse {naive:>5} ({n - naive:>5} tokens lost)")
assert reusable(1_000, 4_096) == ("remote", 4_096)
assert reusable(2_048, 2_048)[0] == "local", "ties stay local: no transfer, same reuse"
# A local hit ending mid-block is why the comparison must be in tokens, not blocks.
print(f"[4] a local hit of 2,047 tokens ends inside block "
      f"{2_047 // BLOCK} of {BLOCK}-token blocks, so block-granular comparison "
      f"would report {2_047 // BLOCK * BLOCK} and mis-rank the tiers")
assert 2_047 // BLOCK * BLOCK < 2_048

print("\nAll assertions passed.")

Executed output:

[1] one recurrent state = 3.0 MiB = KV for 3,072 tokens
[1] at     1,000 tokens: state/KV =  3.072  state dominates
[1] at     3,072 tokens: state/KV =  1.000  KV dominates
[1] at    32,768 tokens: state/KV =  0.094  KV dominates
[1] at 1,000,000 tokens: state/KV =  0.003  KV dominates

[2] interval     4,096 ->     0.71 GiB of state,     2,048 tokens expected recompute
[2] interval    32,768 ->     0.09 GiB of state,    16,384 tokens expected recompute
[2] interval   131,072 ->     0.02 GiB of state,    65,536 tokens expected recompute
[2] interval 1,000,000 ->     0.00 GiB of state,   500,000 tokens expected recompute
[2] closed-form optimum I* = 2,508 tokens; grid search minimum = 2,497 tokens
[2] recompute 100x cheaper -> I* = 25,083 tokens (10.0x further apart)
[2] interval=0 keeps prompt-end states only; correctness is identical, reuse depends on whether turns replay prior prompts

[3] always      ->  840 entries (  2.46 GiB),  440 hits,  0.52 hits per stored entry
[3] second-hit  ->   40 entries (  0.12 GiB),  400 hits, 10.00 hits per stored entry
[3] cost: 40 lost hits; benefit: 800 one-off entries never admitted (2.34 GiB saved)
[3] adversarial k=2 (each prefix seen twice): always 0.048 hits/entry, second-hit 0.000 hits/entry -> second-hit is strictly worse

[4] local  1000 vs remote  4096 -> take remote ( 4096 tokens); local-first policy would reuse  1000 ( 3096 tokens lost)
[4] local  4096 vs remote  1000 -> take local  ( 4096 tokens); local-first policy would reuse  4096 (    0 tokens lost)
[4] local  2048 vs remote  2048 -> take local  ( 2048 tokens); local-first policy would reuse  2048 (    0 tokens lost)
[4] local  2047 vs remote  2048 -> take remote ( 2048 tokens); local-first policy would reuse  2047 (    1 tokens lost)
[4] a local hit of 2,047 tokens ends inside block 31 of 64-token blocks, so block-granular comparison would report 1984 and mis-rank the tiers

All assertions passed.

The section 3 result is the one to carry into a capacity review, and it cuts both ways. On a workload of 800 one-off prefixes and 40 prefixes reused 12 times each, cache-on-second-hit stores 40 entries instead of 840 and lifts hits per stored entry from 0.52 to 10.00. The cost is exactly one forfeited hit per shared prefix, no more. But the adversarial case shows the policy collapsing: when every prefix is seen exactly twice, admission happens on the second sighting and no third sighting ever arrives, so selective retention collects nothing while always-cache still collects something. Selective retention assumes a heavy-tailed reuse distribution. Measure yours before assuming it.

How to develop with it

The invariant to preserve is that a recurrent state snapshot and the paged KV it accompanies describe the same prefix length. Everything that goes wrong in a hybrid cache is a violation of that.

• Copy before extend, never after. The state is overwritten in place by the next forward pass. A snapshot taken lazily is a snapshot of the wrong position.
• Register snapshots inside physical blocks rather than allocating a block per snapshot. Decoupling the large physical state block from fine-grained prefix matching is what keeps the scheme affordable.
• Reconcile every cache group when the chosen prefix length changes. Selecting a remote hit over a local one changes the prefix length for both the paged and the recurrent group.
• Test with a prefix that ends mid-block. Block-aligned prefixes hide the entire class of partial-hit bugs.

The upstream work is a useful reading list because each PR isolates one of these concerns:

Concern	vLLM PR	Merged
Selective prefix-cache retention	#43447	2026-06-04
Retention interval for Mamba/linear attention	#45845	2026-06-23
Marconi-style admission policy	#37898	2026-06-10
Marconi caching with selective retention	#47782	2026-07-13
Partial prefix cache primitives	#45939	2026-06-22
Partial prefix hit for hybrid models	#46384	2026-07-12
Reliable partial-tail KV offload	#49502	2026-07-27

How to run it in production

Turn prefix caching on explicitly. vLLM enables prefix caching by default in general, but disables it for Kimi K3 while the hybrid-cache design evolves. Pass --enable-prefix-caching. A hybrid deployment that silently runs without it looks like a performance problem with no obvious cause.¹

Expect disaggregation to be the strictest test. Splitting prefill from decode means the recurrent state, the paged KV, and the block tables all have to arrive intact. vLLM's NIXL connector treats the shared cache page as two logical views, a token-level attention cache and a request-level recurrent state including convolution state, exchanges both sets of metadata during the handshake, and builds separate transfer descriptors for each.¹ Under heterogeneous tensor parallelism, where prefill and decode use different block sizes, the connector tracks logical-to-physical block mapping and zeroes untransferred tail regions so stale data from a previous request cannot leak through padding or layout gaps. That zeroing step is a correctness requirement, not an optimization.

Set the retention interval from your traffic shape, not from a default. Conversation-dominated traffic where each turn replays the prior prompt is well served by VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0, keeping prompt-end states only. Traffic with long uninterrupted contexts needs periodic checkpoints; use the cost model above with your own recompute and storage prices rather than copying an interval from a blog post.

Watch two metrics that a conventional engine does not have: recurrent-state cache occupancy separately from KV occupancy, and the state hit rate separately from the KV hit rate. The failure signature of a bad retention policy is a healthy KV hit rate alongside a collapsed state hit rate, which manifests as unexplained prefill work on requests that appear to be cache hits.

For context on the scale this was validated at: vLLM reports serving a 2.8T-parameter hybrid MoE on a minimum of 8 B300 or 16 B200 class GPUs, reaching 111 tok/s per user at TP8 and 118 tok/s at TP16 at batch size 1, rising to 331 and 370 tok/s respectively with speculative decoding.¹ Treat those as vendor-published measurements on their hardware.

How to maintain it

• Re-check whether prefix caching is still off by default for your model on each engine upgrade. This is an explicitly temporary state.
• Re-derive the retention interval when traffic shape changes. The optimum scales as the square root of the storage-to-recompute cost ratio, so a change in either price moves it, and the executed model shows a 100x cheaper recompute moving the optimum 10x further out.
• Re-validate the disaggregated path after any block-size change. Heterogeneous TP block mapping and tail zeroing are the fragile part.
• Track the hybrid-cache PRs. This machinery landed between June and late July 2026 and is still moving.

Failure modes

• Silent quality corruption from a stale state tail. Under heterogeneous TP, untransferred tail regions that are not zeroed expose the previous request's data. This produces wrong output with no error, on a subset of requests, and only under disaggregation.
• Snapshot taken after the overwrite. The state no longer corresponds to the claimed prefix boundary. Symptom is degraded output quality on cache hits specifically, which is easy to misattribute to the model.
• Block-granular tier comparison. Rounding a local partial hit down to a block boundary before comparing against a remote hit picks the wrong tier and quietly loses reusable prefix.
• Selective retention on a flat reuse distribution. Cache-on-second-hit collects nothing when prefixes are reused exactly twice, as the executed model shows. It is a heavy-tail policy.
• Interval set for storage without pricing recompute. A very large interval looks efficient in cache-occupancy dashboards while forcing large suffix recomputation that shows up as TTFT regression.
• Prefix caching left off. Costs the entire benefit, reports no error.
• Assuming reuse semantics carry over from paged KV. Any prefix of a paged KV cache is reusable; only checkpointed boundaries of a recurrent state are. Capacity planning that assumes the former will over-predict hit rates.

Open questions and validation

• The crossover figures in the executed model use a representative 3 MiB state and 1 KiB per-token KV. Substitute your model's real per-layer sizes before using them for capacity planning; the shape of the conclusion is robust, the specific token counts are not.
• vLLM's published throughput figures were not reproduced here, and no GPU was available on the documentation host to attempt them.
• Whether the two retention policies interact well at scale, as opposed to each working in isolation, is not established by any published measurement located here.
• Marconi's reported gains were measured against its own baselines in the MLSys '25 paper, not against vLLM's current hybrid allocator.

References

• vLLM, "Kimi K3 Is Here: Efficient Day-0 Support on vLLM" (2026-07-27): https://vllm.ai/blog/2026-07-27-k3
• Pan et al., "Marconi: Prefix Caching for the Era of Hybrid LLMs", MLSys 2025, arXiv 2411.19379: https://arxiv.org/abs/2411.19379
• Marconi artifact repository: https://github.com/ruipeterpan/marconi
• vLLM PR #37898, Marconi-style admission policy for hybrid cache: https://github.com/vllm-project/vllm/pull/37898
• vLLM PR #43447, selective prefix-cache retention: https://github.com/vllm-project/vllm/pull/43447
• vLLM PR #45845, prefix-cache retention interval for Mamba and linear attention: https://github.com/vllm-project/vllm/pull/45845
• vLLM PR #46384, partial prefix cache hit for hybrid models: https://github.com/vllm-project/vllm/pull/46384
• vLLM PR #49502, reliable partial-tail KV offload: https://github.com/vllm-project/vllm/pull/49502

Related: KV cache management · KV cache fundamentals · KV cache transfer with NIXL · Disaggregated inference · Prompt caching · KV cache token eviction · Tenant cache isolation

---

1. vLLM, "Kimi K3 Is Here: Efficient Day-0 Support on vLLM", 2026-07-27. Source of the hybrid cache-manager description, the two retention policies and the VLLM_PREFIX_CACHE_RETENTION_INTERVAL semantics, the NIXL two-logical-view handling and tail zeroing under heterogeneous TP, the statement that prefix caching is not enabled by default for that model, and the per-user decode figures (111 and 118 tok/s at TP8 and TP16, 331 and 370 tok/s with speculative decoding, batch size 1). The post also states the hybrid-cache work is new in vLLM core and benefits every hybrid linear model. ↩↩↩↩↩

2. Pan et al., "Marconi: Prefix Caching for the Era of Hybrid LLMs", MLSys 2025, arXiv 2411.19379. Source of the cache-on-second-hit admission rule and of the observation that in-place recurrent state updates preclude rolling back partial overlaps and so mandate exact-match hits in a naive design. Its reported gains are against the baselines in that paper. ↩↩

3. vLLM PRs #45939, #46384, and #49502, all merged and verified on 2026-07-30. The release post states the multi-tier partial-prefix mechanism was built entirely through existing KV connector APIs, so external store connectors gain it without model-specific integration paths. ↩