Skip to content
Markdown

KV cache fundamentals: what is cached and the transformers Cache API

Scope: the tensor and API layer of the KV cache. What exactly is stored (K and V), why the query is not, the shape and append axis of the cached tensors, how grouped-query attention changes the meaning of the head dimension, and the concrete Cache class surface in Hugging Face transformers v5. Deliberately deferred: the speedup arithmetic and capacity model live in KV cache inference speedup; the serving-side block pool, paging, and prefix reuse in KV cache management; the attention variants that shrink the cache (MQA, GQA, MLA) in FlashAttention and MLA; dropping cached tokens in KV cache token eviction; provider-side billing in prompt caching.

Code blocks come in two kinds. The numpy blocks are runnable, self-checking validations of the core mechanics the page teaches (why Q is not cached, the append axis, the GQA head map); each was executed on a stock python3 with numpy 2.4.6, asserts its result including adversarial cases, and the pasted output is the real output. The transformers snippets are unexecuted reference templates (torch is not installed here) targeting transformers==5.13.1. Source details that exist only after that release are identified as main behavior at commit 01dd2fd (__version__ 5.14.0.dev0, 2026-07-13). Pin the complete environment and re-check before relying on them.

What it is

The KV cache stores, per layer, the key and value vectors of every token the model has already seen. It does not store queries. That asymmetry is the whole idea and it follows from causal masking.

At decode step s the model needs exactly one query row, q_s, the projection of the single new token. The queries of tokens 0..s-1 produced their outputs at the steps where those tokens were current; under a causal mask no later token ever attends from an earlier query, so those query rows are never read again. Keys and values are the opposite: k_j and v_j are read by every query at position j or later, which is every future step for the rest of the sequence. K and V are the reusable half of attention; Q is the disposable half.

The consequence is a shape change, not just a saving. Recomputing the prefix at step s materialises an [s, s] score matrix. Caching collapses that to a [1, s] score vector: the cache grows by one row per step while the query shrinks from s rows to 1. Popular explainers describe the cached path as producing "smaller matrices"; the object that gets smaller is the query and its score matrix, never the cache, which grows monotonically until the sequence ends or a sliding window clips it.5

The block below decodes the same sequence twice on one real multi-head attention layer, once by full recompute and once with a growing cache and a one-row query, and counts the query rows each path materialises. That count is the direct evidence for the asymmetry: the recompute path rebuilds a query row for every token in the prefix at every step and discards all but one, while the cached path builds exactly one. It then asserts the documented append axis and the resulting cache shape. The cached-versus-uncached equivalence experiment, the corrupted-cache case, and the work-reduction arithmetic that all follow from this are not re-derived here; they are the subject of KV cache inference speedup.

# Runnable on system python3 (numpy 2.4.6). Core claim: K and V are the reusable half of
# attention and Q is the disposable half. The block decodes the same sequence twice on one
# real multi-head attention layer: (a) full recompute over the whole prefix every step,
# (b) incremental, with a growing [B, n_kv, t, D] K/V cache and a ONE-row query. It counts
# the QUERY ROWS each path materialises, which is the direct evidence that past queries are
# never re-read and so have nothing to cache, asserts the append axis is seq_len (-2) and not
# head_dim (-1), and checks the cache shape. The work-reduction arithmetic that follows from
# all this is deliberately NOT re-derived here: it is the subject of the speedup page.
import numpy as np

B, H, D = 2, 4, 16                 # batch, kv heads (MHA here: n_q == n_kv), head dim
M = H * D                          # model width
rng = np.random.default_rng(0)
Wq, Wk, Wv, Wo = (rng.standard_normal((M, M)) / np.sqrt(M) for _ in range(4))

QROWS = {"n": 0}                   # query rows materialised


def heads(x: np.ndarray) -> np.ndarray:
    """[B, S, M] -> [B, H, S, D]: the layout DynamicCache documents for K and V."""
    return x.reshape(B, x.shape[1], H, D).transpose(0, 2, 1, 3)


def merge(y: np.ndarray) -> np.ndarray:
    """[B, H, S, D] -> [B, S, M]."""
    return y.transpose(0, 2, 1, 3).reshape(B, y.shape[2], M)


def softmax(s: np.ndarray) -> np.ndarray:
    w = np.exp(s - s.max(axis=-1, keepdims=True))
    return w / w.sum(axis=-1, keepdims=True)


def attend(q: np.ndarray, k: np.ndarray, v: np.ndarray, causal: bool) -> np.ndarray:
    """q [B,H,Sq,D], k/v [B,H,Sk,D] -> [B,Sq,M]. Counts the query rows it materialises."""
    sq, sk = q.shape[2], k.shape[2]
    QROWS["n"] += B * H * sq
    s = q @ k.transpose(0, 1, 3, 2) / np.sqrt(D)
    if causal:                     # only needed when Sq == Sk (the full-recompute path)
        s[..., np.triu(np.ones((sq, sk), dtype=bool), 1 + sk - sq)] = -np.inf
    return merge(softmax(s) @ v) @ Wo


def decode_recompute(prompt: np.ndarray, n: int) -> np.ndarray:
    """use_cache=False: re-project Q, K, V for the WHOLE prefix every step. Every past query
    row is recomputed and then thrown away, which is the waste the cache removes."""
    x = prompt.copy()
    for _ in range(n):
        y = attend(heads(x @ Wq), heads(x @ Wk), heads(x @ Wv), causal=True)
        x = np.concatenate([x, np.tanh(y[:, -1:, :])], axis=1)
    return x


def decode_cached(prompt: np.ndarray, n: int, bad_axis: bool = False) -> tuple[np.ndarray, np.ndarray]:
    """The cache: K and V accumulate on the seq_len axis (-2); Q is one row and is dropped."""
    kc, vc = heads(prompt @ Wk), heads(prompt @ Wv)        # [B, H, P, D]
    y = attend(heads(prompt @ Wq), kc, vc, causal=True)    # prefill
    x = np.concatenate([prompt, np.tanh(y[:, -1:, :])], axis=1)
    for _ in range(n - 1):
        tok = x[:, -1:, :]                                 # the ONE new token
        q1, k1, v1 = heads(tok @ Wq), heads(tok @ Wk), heads(tok @ Wv)
        axis = -1 if bad_axis else -2                      # -2 is seq_len; -1 is head_dim
        kc = np.concatenate([kc, k1], axis=axis)
        vc = np.concatenate([vc, v1], axis=axis)
        assert kc.shape[-1] == D, f"append axis wrong: head_dim became {kc.shape[-1]}"
        x = np.concatenate([x, np.tanh(attend(q1, kc, vc, causal=False))], axis=1)
    return x, kc


P, N = 12, 20
prompt = rng.standard_normal((B, P, M))

QROWS["n"] = 0
ref = decode_recompute(prompt, N)
qr_full = QROWS["n"]
QROWS["n"] = 0
out, cache_k = decode_cached(prompt, N)
qr_cached = QROWS["n"]

# 1. Correctness gate for everything below: dropping the past queries changes nothing, so the
#    two paths emit the same tokens. Asserted here only so the counts below compare a correct
#    path against a correct path. The equivalence experiment itself, the corrupted-cache case,
#    and the work-reduction arithmetic belong to kv-cache-inference-speedup.md.
assert ref.shape == out.shape == (B, P + N, M)
assert np.allclose(ref, out, rtol=0, atol=1e-12), float(np.abs(ref - out).max())

# 2. Why Q is not cached, counted. The recompute path materialises a query row for every token
#    in the prefix at every step and discards all but the last; the cached path materialises
#    exactly one query row per step. Past query rows are never re-read, so there is nothing to
#    cache: the asymmetry is a property of the causal mask, not an optimisation choice.
q_rows_recomputed = B * H * sum(range(P, P + N))
assert qr_full == q_rows_recomputed, (qr_full, q_rows_recomputed)
assert qr_cached == B * H * (P + N - 1), qr_cached      # P (prefill) + one per decode step
assert qr_full > 10 * qr_cached, qr_full / qr_cached
# Every query row past the first of each step is recomputed and thrown away, every step.
assert qr_full - qr_cached == B * H * (sum(range(P, P + N)) - (P + N - 1))

# 3. Shape and the append axis. After the full run the cache holds every token once, on the
#    seq_len axis, and it GREW while the query stayed one row wide. The "smaller matrices" of
#    the popular explainers are the query and its score matrix, never the cache.
assert cache_k.shape == (B, H, P + N - 1, D), cache_k.shape   # last token's K is never used
assert cache_k.shape[-2] > P                                  # grew on seq_len
assert cache_k.shape[-1] == D                                 # head_dim untouched

# 4. Adversarial: appending on the head_dim axis (-1) instead of seq_len (-2) is the classic
#    layout bug (upstream `DynamicLayer.update` concatenates on dim=-2). It must raise.
raised = ""
try:
    decode_cached(prompt, N, bad_axis=True)
except (AssertionError, ValueError) as exc:
    raised = type(exc).__name__
assert raised == "ValueError", raised

print(f"query rows materialised: recompute {qr_full}  cached {qr_cached}  ({qr_full / qr_cached:.1f}x)")
print(f"  -> {qr_full - qr_cached} query rows recomputed and discarded; the cache stores none of them")
print(f"cache K shape {cache_k.shape} = [B, n_kv, seq_len, head_dim]; query stayed 1 row wide")
print(f"appending on axis -1 instead of -2 raises: {raised}")

Real output:

query rows materialised: recompute 3440  cached 248  (13.9x)
  -> 3192 query rows recomputed and discarded; the cache stores none of them
cache K shape (2, 4, 31, 16) = [B, n_kv, seq_len, head_dim]; query stayed 1 row wide
appending on axis -1 instead of -2 raises: ValueError

The counts are the point. The recompute path builds 3,440 query rows to emit the same 20 tokens the cached path emits with 248, and every one of the 3,192 extra rows is used once and thrown away. That is why there is no Q cache to build: not because storing queries would be expensive, but because nothing would ever read them. Meanwhile the cache grew to 31 rows while the query never exceeded one, which is the shape change stated above, measured.

Case 4 is the easy failure, because the shape system catches it. The operationally dangerous one is the cache that is wrong while every tensor has the right shape, the right dtype, and every kernel returns success. That failure surfaces as degraded output quality, not as an exception. Cache-integrity bugs (a mis-indexed rotary position on append, a speculative-decoding rollback that does not truncate, cross-request block aliasing) all live in that blind spot. KV cache inference speedup quantifies it by corrupting a single cached float; the GQA block below is a second instance of the same class, where every shape is right and the head map is not.

Why use it

The reason to know this layer, rather than treat the cache as an opaque engine internal, is that four common tasks all bottom out in the tensor layout and the class surface:

  • Reading a memory number correctly. NVIDIA's widely copied formula, 2 * num_layers * (num_heads * dim_head) * precision_in_bytes bytes per token, has no num_kv_heads term; it is an MHA-only formula.6 Its worked example (Llama-2-7B, 1 * 4096 * 2 * 32 * 4096 * 2 bytes, roughly 2 GB) is right only because Llama-2-7B is MHA. Applied to a GQA model it over-counts by the ratio num_heads / num_kv_heads, which is 8x for Llama-3.1-70B. The general form substitutes num_kv_heads; the sizing arithmetic and the capacity model built on it are in KV cache management and KV cache inference speedup.
  • Passing a Cache object across calls. Multi-turn chat, prefix prefill, and speculative decoding all hand a live cache back into forward or generate. That requires knowing which class you hold and what it will do on update.
  • Making decode compilable. torch.compile and CUDA-graph capture need shape-stable tensors. A cache that grows by one row per step is not shape-stable; StaticCache is the fix, and it is a fundamentally different object from DynamicCache, not a flag on it. See CUDA graphs and torch.compile.
  • Not shipping a grouping bug. The cache is stored with num_kv_heads, so every read has to expand it back to num_heads. The wrong expansion is silent on MHA and MQA models and wrong only on GQA models. The block under Architecture proves that.

When to use it (and when not)

Every Cache class is a point on a memory-versus-speed curve. The Hugging Face support matrix is the first filter:3

Cache class Sliding layers Offloading torch.compile() Memory
DynamicCache Yes Yes No Medium
StaticCache Yes Yes Yes High
QuantizedCache No No No Low
  • DynamicCache is the default for every generative model and the right answer unless a specific constraint says otherwise. It grows on demand, so it allocates only what the sequence actually uses.
  • StaticCache when the decode step must be compiled or graph-captured. It pre-allocates max_cache_len so tensor shapes never change between steps. The honest cost, which the upstream doc states plainly: a large fixed cache means many masked-out tokens still participate in the attention computation, so it is "a trade-off which should be very good if you generate with several sequence of more or less the same lengths, but may be sub-optimal if you have for example 1 very large sequence, and then only short sequences".3 One-long-then-many-short traffic pays for the long one on every short request.
  • QuantizedCache when a memory-constrained GPU is the binding constraint and latency is not. It quantizes older K and V to int2/int4/int8 while keeping a small residual_length window (default 128) in original precision. Upstream warns it "can harm latency if the context length is short and there is enough GPU memory available for generation without enabling cache quantization".3 It also raises on any model with sliding, chunked, or linear-attention layers: on main its constructor rejects every layer type except full_attention.
  • EncoderDecoderCache for encoder-decoder models; it is a wrapper holding one self-attention and one cross-attention cache, and needs no setup.
  • Not this API alone for dynamic serving. generate can process a fixed batch, but its per-call cache does not provide iteration-level admission, paging, sharing, or eviction across independent requests. Those functions belong to the separate transformers continuous-batching API or a dedicated serving engine (continuous batching in transformers, KV cache management, inference serving). Mamba and other state-space models carry recurrent state instead of per-token K and V and use a model-specific cache.3

Architecture

In transformers v5 a Cache is "mostly a list of CacheLayerMixin objects, one per model layer".1 The per-layer object, not the top-level class, decides the behaviour. On commit 01dd2fd, DynamicCache(config=...) dispatches through DYNAMIC_LAYER_TYPE_MAPPING and StaticCache through STATIC_LAYER_TYPE_MAPPING. The v5.13.1 release uses the older names LAYER_TYPE_CACHE_MAPPING and LAYER_TYPE_STATIC_CACHE_MAPPING for the same role. A hybrid model with alternating full and sliding layers therefore needs no special cache class: one DynamicCache holds DynamicLayer objects for full-attention layers and DynamicSlidingWindowLayer objects for sliding layers.

flowchart TB
  subgraph CACHE ["DynamicCache(config=model.config)"]
    L0["layer 0: DynamicLayer<br/>(full_attention)"]
    L1["layer 1: DynamicSlidingWindowLayer<br/>(sliding_attention)"]
    L2["layer 2: DynamicLayer<br/>(full_attention)"]
  end
  CFG["config.layer_types"] -->|"DYNAMIC_LAYER_TYPE_MAPPING"| CACHE
  subgraph TENS ["Per-layer tensors"]
    K["keys: [batch, num_kv_heads, seq_len, head_dim]"]
    V["values: [batch, num_kv_heads, seq_len, head_dim]"]
  end
  L0 --> TENS
  STEP["decode step: one new token"] -->|"k_new, v_new: [batch, num_kv_heads, 1, head_dim]"| UPD
  UPD["DynamicLayer.update()<br/>torch.cat(dim=-2)"] --> TENS
  TENS -->|"expand n_kv -> n_q, then attend"| OUT["[batch, num_heads, 1, head_dim]"]
  Q["q_new: one row, never cached"] --> OUT

Two facts about the tensors, both read from the source:

The shape. The DynamicCache docstring says the expected shape for each tensor is [batch_size, num_heads, seq_len, head_dim].1 That wording is a trap. For any grouped-query model the second axis is num_kv_heads, not the query head count. Llama-3.1-70B has 64 query heads and 8 KV heads; its cache tensors carry 8. Sizing a cache from num_heads on a GQA model over-counts by exactly the group factor.

The append axis. DynamicLayer.update() is self.keys = torch.cat([self.keys, key_states], dim=-2).1 Dimension -2 is seq_len. The pseudocode in the widely read Hugging Face community blog concatenates on dim=1, which is the head axis under this layout; the layouts differ, and copying that line into a [batch, heads, seq, dim] model corrupts the cache.4

Because the cache stores num_kv_heads, every read of it expands n_kv heads back to n_q query heads. The correct map sends query head h to KV head h // rep where rep = n_q // n_kv (contiguous groups, np.repeat). The classic bug is h % n_kv (interleaved, np.tile). The block below asserts the correct map, proves the buggy map changes the answer by order 1, and shows why the bug survives testing: on MHA (n_kv == n_q) and on MQA (n_kv == 1) the two maps are identical.

# Runnable on system python3 (numpy 2.4.6). The cache is stored with num_kv_heads, not
# num_heads, so every read of it has to expand n_kv key/value heads back to n_q query heads.
# The correct map is q-head h -> kv-head h // rep (np.repeat). The classic bug is h % n_kv
# (np.tile). This block asserts the correct map, proves the buggy map produces a different
# answer, and shows why the bug ships: it is INVISIBLE on MHA and on MQA models.
import numpy as np

N_Q, N_KV, D, S = 8, 2, 16, 7          # query heads, kv heads, head dim, cached tokens
rng = np.random.default_rng(3)


def rep_factor(n_q: int, n_kv: int) -> int:
    """Group size. GQA requires n_q to be an exact multiple of n_kv."""
    assert n_kv >= 1 and n_q % n_kv == 0, f"n_q={n_q} is not a multiple of n_kv={n_kv}"
    return n_q // n_kv


def expand_correct(kv: np.ndarray, n_q: int) -> np.ndarray:
    """[B, n_kv, S, D] -> [B, n_q, S, D]. Contiguous groups: heads 0..3 share kv head 0."""
    return np.repeat(kv, rep_factor(n_q, kv.shape[1]), axis=1)


def expand_buggy(kv: np.ndarray, n_q: int) -> np.ndarray:
    """The off-by-grouping bug: interleaves instead of grouping (q-head h -> kv-head h%n_kv)."""
    return np.tile(kv, (1, rep_factor(n_q, kv.shape[1]), 1, 1))


def attend(q: np.ndarray, k: np.ndarray, v: np.ndarray) -> np.ndarray:
    s = q @ k.transpose(0, 1, 3, 2) / np.sqrt(D)
    w = np.exp(s - s.max(axis=-1, keepdims=True))
    return (w / w.sum(axis=-1, keepdims=True)) @ v


kc = rng.standard_normal((1, N_KV, S, D))      # the cache: n_kv heads, NOT n_q
vc = rng.standard_normal((1, N_KV, S, D))
q1 = rng.standard_normal((1, N_Q, 1, D))       # one decode step: a single query row per head

# 1. The repeat factor and the index maps.
rep = rep_factor(N_Q, N_KV)
assert rep == 4
correct_map = [h // rep for h in range(N_Q)]
buggy_map = [h % N_KV for h in range(N_Q)]
assert correct_map == [0, 0, 0, 0, 1, 1, 1, 1], correct_map
assert buggy_map == [0, 1, 0, 1, 0, 1, 0, 1], buggy_map
assert correct_map != buggy_map

# 2. np.repeat implements h // rep; np.tile implements h % n_kv. Prove it head by head.
ok, bug = expand_correct(kc, N_Q), expand_buggy(kc, N_Q)
assert ok.shape == bug.shape == (1, N_Q, S, D)
for h in range(N_Q):
    assert np.array_equal(ok[0, h], kc[0, correct_map[h]]), h
    assert np.array_equal(bug[0, h], kc[0, buggy_map[h]]), h

# 3. The bug is not cosmetic: the attention output is different, by O(1), not by round-off.
out_ok = attend(q1, ok, expand_correct(vc, N_Q))
out_bug = attend(q1, bug, expand_buggy(vc, N_Q))
delta = float(np.abs(out_ok - out_bug).max())
assert not np.allclose(out_ok, out_bug), "grouping bug must change the output"
assert delta > 0.1, delta
# Heads 0 and n_q-1 are the fixed points of both maps only when they coincide; here head 0
# agrees (0 // 4 == 0 % 2) while head 1 does not (1 // 4 = 0, 1 % 2 = 1).
assert np.allclose(out_ok[0, 0], out_bug[0, 0])
assert not np.allclose(out_ok[0, 1], out_bug[0, 1])

# 4. Why the bug ships: on an MHA model (n_kv == n_q) and on an MQA model (n_kv == 1) the
#    two maps are IDENTICAL, so a test suite that only covers those never sees it. It bites
#    exactly in the 1 < n_kv < n_q window used by GQA models.
for n_kv in (1, N_Q):
    kv = rng.standard_normal((1, n_kv, S, D))
    assert np.array_equal(expand_correct(kv, N_Q), expand_buggy(kv, N_Q)), n_kv
    r = rep_factor(N_Q, n_kv)
    assert [h // r for h in range(N_Q)] == [h % n_kv for h in range(N_Q)]

# 5. Adversarial: a kv-head count that does not divide the query heads (a mis-parsed config,
#    or a tensor-parallel shard that split 8 query heads across 3 ranks) must raise, not
#    silently truncate or broadcast.
raised = False
try:
    expand_correct(rng.standard_normal((1, 3, S, D)), N_Q)
except AssertionError:
    raised = True
assert raised, "n_q % n_kv != 0 must raise"

# 6. Boundary: expansion is a VIEW-level trick, not extra cache. The stored bytes stay at
#    n_kv heads; only the read is widened. Asserting that keeps the sizing honest.
assert kc.nbytes * rep == ok.nbytes and kc.shape[1] == N_KV

print(f"rep={rep}; correct map {correct_map}; buggy map {buggy_map}")
print(f"max|out_correct - out_buggy| = {delta:.4f} (head 0 agrees, head 1 does not)")
print(f"stored KV heads {kc.shape[1]}, expanded read heads {ok.shape[1]}, "
      f"bytes stored {kc.nbytes} vs bytes read {ok.nbytes}")

Real output:

rep=4; correct map [0, 0, 0, 0, 1, 1, 1, 1]; buggy map [0, 1, 0, 1, 0, 1, 0, 1]
max|out_correct - out_buggy| = 1.5133 (head 0 agrees, head 1 does not)
stored KV heads 2, expanded read heads 8, bytes stored 1792 vs bytes read 7168

Assertion 4 is why this bug class is worth naming. Any test suite whose fixtures are MHA or MQA passes with the wrong map. MQA and GQA themselves are Shazeer's and Ainslie's constructions; what they trade away and why is in FlashAttention and MLA.78

How to use it

The cache is on by default. use_cache=True is the default on every generative model, and use_cache=False in generate is what re-runs the whole prefix each step. The class is selected either by cache_implementation (a string on GenerationConfig or generate) or by constructing a Cache and passing it as past_key_values.

# UNEXECUTED reference template. API target: transformers==5.13.1. Lock torch to an exact
# version compatible with the deployment CUDA build; a minimum-version constraint is not a
# reproducibility pin. Re-check the constants below on every upgrade.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, DynamicCache

model_id = "Qwen/Qwen3-0.6B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
inputs = tok("The KV cache stores", return_tensors="pt").to(model.device)

# (a) default: DynamicCache, created for you.
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)

# (b) explicit class, so the caller can inspect or manage the cache object.
past = DynamicCache(config=model.config)     # config -> per-layer dispatch (sliding/full)
out = model.generate(**inputs, max_new_tokens=32, past_key_values=past)
# Do not combine this cache with a re-render of the full conversation. Reuse requires only
# uncached suffix tokens plus correct attention-mask and position metadata; see below.

# (c) fixed-size cache; also switches on compilation of the decode step for greedy/sample.
out = model.generate(**inputs, max_new_tokens=32, cache_implementation="static",
                     max_cache_len=1024)

# (d) quantized cache: int4 via hqq. Full-attention models only.
out = model.generate(**inputs, max_new_tokens=32, cache_implementation="quantized",
                     cache_config={"backend": "hqq", "nbits": 4, "axis_key": 1,
                                   "axis_value": 1})

# (e) offloading: keep all layers but one on CPU, prefetch the next asynchronously.
out = model.generate(**inputs, max_new_tokens=32, cache_implementation="offloaded")

The accepted cache_implementation strings, taken from the constants in generation/configuration_utils.py rather than from the docstring:2

Group Values
STATIC_CACHE_IMPLEMENTATIONS "static", "offloaded_static"
DYNAMIC_CACHE_IMPLEMENTATIONS "dynamic", "dynamic_full", "offloaded", "quantized"
DEPRECATED_STATIC_CACHE_IMPLEMENTATIONS "sliding_window", "hybrid", "hybrid_chunked", "offloaded_hybrid", "offloaded_hybrid_chunked"
accepted by the validator but not in that union "paged" (re-routes to continuous batching)

Two mismatches worth knowing before a config change fails in CI. The cache_implementation docstring lists only five values (dynamic, static, offloaded, offloaded_static, quantized): "dynamic_full" is valid in the validator and missing from the docs. The same docstring writes the offloaded constructors as DynamicCache(offloaded=True), but the real keyword is offloading=True; offloaded= appears nowhere in cache_utils.py. Trust the constants, not the prose.

max_cache_len applies to static caches only. Passing it with a dynamic implementation does not error; it logs a warning and is ignored.2

How to develop with it: three stale-API traps

Most KV-cache tutorials and a good share of internal code predate transformers v5. Three specific things that were true and are now false:

1. SlidingWindowCache no longer exists as a class. The final line of cache_utils.py is a back-compat alias:

# Deprecated alias: SlidingWindowCache was removed in transformers v5. StaticCache is the replacement.
SlidingWindowCache = StaticCache

Sliding-window behaviour is not a cache class any more; it is a layer type. DYNAMIC_LAYER_TYPE_MAPPING maps "sliding_attention" and "chunked_attention" to DynamicSlidingWindowLayer, and the layer types are derived from the model config. An import SlidingWindowCache still resolves and silently gives a StaticCache, which is a different object with a different memory profile.1

2. OffloadedCache does not exist. Zero occurrences in cache_utils.py. Offloading is a constructor flag on the base Cache: Cache(..., offloading=True, offload_only_non_sliding=...). The defaults differ by class and the difference is not cosmetic: offload_only_non_sliding defaults to False for DynamicCache and True for StaticCache. Sliding layers are small, so offloading them usually costs more in transfer than it saves in HBM; the StaticCache default reflects that and the DynamicCache default does not.13

3. The legacy tuple format is deleted, not deprecated. to_legacy_cache and from_legacy_cache have zero occurrences in cache_utils.py on both main and the v5.13.1 tag. past_key_values is a Cache object or None. Any code or documentation that says past_key_values may be a nested tuple of (key, value) per layer, or that calls .to_legacy_cache() to unwrap one, is broken on v5, not merely deprecated.1

Multi-turn reuse is an incremental-input contract, not merely a persistent object. Once a cache contains a prefix, the next call must supply only token IDs that are not already cached, while the attention mask and position or cache_position metadata still describe the complete logical sequence. Re-rendering the full conversation and passing it with the old cache duplicates the prefix. generate() also returns the complete input-plus-output token sequence while the live cache may lag the final emitted token by one decode step, so comments that equate the cache length with the returned sequence length are unsafe. Use the pinned version's cache examples or a manual forward() loop, and assert cached-versus-uncached token equivalence before adopting cross-call reuse.

The prefill-a-prefix pattern (build a cache for a shared system prompt once, then copy.deepcopy it per request) is documented upstream and is the single-process ancestor of what a serving engine does with block-level prefix caching.3 Reuse across processes and across requests is KV cache management, not this API.

How to run it in production

StaticCache is the transformers-side instance of the fixed-shape rule. A cache that grows one row per step changes tensor shapes every step, which forces torch.compile to retrace and makes CUDA-graph capture impossible: capture requires fixed shapes at fixed addresses. StaticCache pre-allocates max_cache_len once, so the decode step is shape-stable and capturable. Setting cache_implementation="static" also turns on automatic compilation of the decoding stage for greedy and sample strategies.3 The mechanism and the bucketing that serving engines build on top of it are in CUDA graphs.

The cost is real and is not a rounding error. A large fixed cache means a large number of masked-out positions still take part in the attention computation on every step. The trade is good when sequences are of similar length and poor when one very long sequence sets max_cache_len and the rest of the traffic is short.3 Measure it against the workload's actual length distribution rather than assuming the compile win dominates.

Do not treat generate() as a dynamic serving layer. It accepts a fixed batch, but it has no cross-request admission, block pool, preemption, or iteration-level batching. The published transformers-level speedups are single-stream toy runs and should be read as such: the Hugging Face community blog measures SmolLM2-1.7B on a T4, 300 new tokens, 11.7 s cached versus 1 min 1 s uncached (about 5.2x); the Medium article measures GPT-2 on a T4, 1000 new tokens, 11.885 +/- 0.272 s cached versus 56.197 +/- 1.855 s uncached (about 4.7x).45 Both are functions of sequence length, batch size 1, and one small model. For dynamic serving, use transformers continuous batching or a dedicated engine; their KV subsystems are described in continuous batching in transformers, KV cache management, and KV cache inference speedup.

Where this API does belong in production: offline batch scoring and generation, evaluation harnesses, RL rollout workers that already own the model object, speculative-decoding research, and anything that needs to hold a Cache across calls rather than across requests. In those settings the choices above (dynamic versus static, offloaded, quantized) are the whole tuning surface.

How to maintain it

  • Re-diff cache_utils.py on every upgrade. The facts on this page were read at commit 01dd2fd (__version__ 5.14.0.dev0, 2026-07-13) and compared with the v5.13.1 release tag. The dispatch constants already differ: DYNAMIC_LAYER_TYPE_MAPPING / STATIC_LAYER_TYPE_MAPPING on the development commit versus LAYER_TYPE_CACHE_MAPPING / LAYER_TYPE_STATIC_CACHE_MAPPING in v5.13.1. Pin transformers exactly, then re-check the class list, dispatch mappings, cache_implementation constants, and offload defaults on a bump.
  • Read the constants, not the docstrings. Two live mismatches are recorded above ("dynamic_full" missing from the docstring; offloaded= in the docstring versus offloading= in the code). Both are the sort of thing that a copy-paste turns into a ValueError or, worse, a silently-ignored keyword.
  • Assert the cache shape in your own code. The one-line guard assert past.layers[i].keys.shape == (batch, n_kv, seen, head_dim) catches the append-axis and head-count classes of bug at the point they happen instead of at eval time. The num_heads in the upstream docstring means num_kv_heads for any GQA model; write the assertion against the config's num_key_value_heads.
  • Keep an equivalence test. The cached-versus-uncached equivalence in the first block is a real regression test: run a short greedy generation with use_cache=True and use_cache=False and assert identical token ids. It catches cache-integrity regressions that no shape check sees.
  • Quantized cache needs an accuracy gate, not just a memory check. int2 and int4 K/V change the model's outputs. Gate on an eval, and treat the latency warning in the upstream docs as a measurement to make, not a claim to accept.

Failure modes

  • A wrong cache with right shapes. A corrupted, stale, or mis-indexed cache produces fluent, wrong output and raises nothing: shapes, dtypes, and kernel return codes all stay valid. Shape assertions do not catch it; a cached-versus-uncached equivalence test does. KV cache inference speedup quantifies the drift a single flipped cached float produces.
  • GQA head-grouping bug. Expanding n_kv heads to n_q with h % n_kv (interleave) instead of h // rep (group) changes the output by order 1 and is invisible on MHA and MQA fixtures. It appears only in the 1 < n_kv < n_q window used by GQA models.
  • Sizing a GQA cache with an MHA formula. The widely copied NVIDIA byte-per-token formula carries no num_kv_heads term, so it over-counts every GQA model by the group factor; the numbers are under Why use it above.6 The correct sizing is in KV cache management.
  • SlidingWindowCache silently becoming a StaticCache. The name still imports; it is an alias. Code that expected a window-bounded cache gets a fixed-size one with a different memory profile and no warning at the import site.1
  • Code written against to_legacy_cache / tuple past_key_values. Deleted on v5, not deprecated. Any unwrap-the-tuple path fails outright.1
  • StaticCache on a skewed length distribution. max_cache_len sized for the longest sequence makes every short request pay attention over a mostly-masked cache. The compile win can be smaller than the wasted attention work; measure on the real distribution.3
  • QuantizedCache on a sliding-window model. It raises rather than degrading: on main the constructor rejects any layer type other than full_attention.1
  • Offloading defaults assumed uniform. offload_only_non_sliding is False on DynamicCache and True on StaticCache. Assuming one default while using the other class changes what gets moved across PCIe on every layer, every step.13
  • Treating a transformers speedup number as a production number. The 4.7x and 5.2x figures are batch-1, single-model, single-GPU, and length-dependent. A batched server's economics are set by KV footprint and concurrency, not by this ratio (KV cache inference speedup).

References

  • Hugging Face, "Cache strategies" (transformers KV-cache docs): support matrix, cache_implementation, offloading defaults, quantized backends, iterative generation, prefix prefill: https://huggingface.co/docs/transformers/main/en/kv_cache
  • Hugging Face transformers, src/transformers/cache_utils.py (read at commit 01dd2fd, __version__ 5.14.0.dev0, 2026-07-13; cross-checked against tag v5.13.1): class list, layer-type mappings, torch.cat(dim=-2) append, SlidingWindowCache = StaticCache alias: https://github.com/huggingface/transformers/blob/main/src/transformers/cache_utils.py
  • Hugging Face transformers, src/transformers/generation/configuration_utils.py: STATIC_CACHE_IMPLEMENTATIONS, DYNAMIC_CACHE_IMPLEMENTATIONS, DEPRECATED_STATIC_CACHE_IMPLEMENTATIONS, the "paged" validator case, the max_cache_len warning: https://github.com/huggingface/transformers/blob/main/src/transformers/generation/configuration_utils.py
  • Not Lain, "KV Caching Explained: Optimizing Transformer Inference Efficiency" (Hugging Face community blog, 30 Jan 2025): the SmolLM2-1.7B / T4 benchmark and the dim=1 pseudocode: https://huggingface.co/blog/not-lain/kv-caching
  • João Lages, "Transformers KV Caching Explained" (Medium): the GPT-2 / T4 benchmark, 1000 tokens: https://medium.com/@joaolages/kv-caching-explained-276520203249
  • NVIDIA, "Mastering LLM Techniques: Inference Optimization": the MHA-only KV byte formula and the ~2 GB Llama-2-7B example: https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
  • Vaswani et al., "Attention Is All You Need" (arXiv 1706.03762): causal masking, the property the cache exploits: https://arxiv.org/abs/1706.03762
  • Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need" (arXiv 1911.02150): multi-query attention: https://arxiv.org/abs/1911.02150
  • Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (arXiv 2305.13245): grouped-query attention: https://arxiv.org/abs/2305.13245
  • DeepSeek-AI, "DeepSeek-V2" (arXiv 2405.04434): Multi-head Latent Attention, low-rank KV compression: https://arxiv.org/abs/2405.04434
  • Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (arXiv 2309.06180): the serving-layer cache, for contrast with the single-sequence API here: https://arxiv.org/abs/2309.06180

Related: KV cache inference speedup · KV cache management · KV cache token eviction · FlashAttention and MLA · PyTorch attention APIs · CUDA graphs · torch.compile · continuous batching internals · prompt caching (provider APIs) · LLM inference efficiency · inference serving · Glossary

  1. huggingface/transformers, src/transformers/cache_utils.py, read at commit 01dd2fd (1,866 lines, __version__ 5.14.0.dev0, 2026-07-13) and tag v5.13.1 (1,906 lines). Cache docstring: "A Cache is mostly a list of CacheLayerMixin objects, one per model layer." Classes present include Cache, DynamicCache, StaticCache, QuantizedCache, and EncoderDecoderCache. The development commit dispatches through DYNAMIC_LAYER_TYPE_MAPPING / STATIC_LAYER_TYPE_MAPPING; v5.13.1 names those roles LAYER_TYPE_CACHE_MAPPING / LAYER_TYPE_STATIC_CACHE_MAPPING. DynamicCache gives the tensor layout as [batch_size, num_heads, seq_len, head_dim]; for GQA, that head axis is the KV-head count. DynamicLayer.update concatenates on dim=-2. The file ends with SlidingWindowCache = StaticCache. Neither snapshot contains OffloadedCache, to_legacy_cache, from_legacy_cache, or an offloaded= keyword. Offload defaults and quantized-cache constraints quoted on this page are from the development snapshot and must be rechecked against the installed release. ↩↩↩↩↩↩↩↩↩↩

  2. huggingface/transformers, src/transformers/generation/configuration_utils.py, same commit. STATIC_CACHE_IMPLEMENTATIONS = ("static", "offloaded_static"); DYNAMIC_CACHE_IMPLEMENTATIONS = ("dynamic", "dynamic_full", "offloaded", "quantized"); DEPRECATED_STATIC_CACHE_IMPLEMENTATIONS = ("sliding_window", "hybrid", "hybrid_chunked", "offloaded_hybrid", "offloaded_hybrid_chunked"), commented "All the following are redundant and deprecated, but kept for BC". The validator builds valid_cache_implementations = ALL_CACHE_IMPLEMENTATIONS + ("paged",) with the comment that "paged" re-routes to continuous batching. max_cache_len with a non-static implementation emits logger.warning_once and is ignored. The cache_implementation docstring lists only dynamic, static, offloaded, offloaded_static, quantized, omitting dynamic_full, and writes the offloaded forms as DynamicCache(offloaded=True) where the code's keyword is offloading. ↩↩

  3. Hugging Face, "Cache strategies" https://huggingface.co/docs/transformers/main/en/kv_cache. Support matrix (Dynamic: sliding yes / offloading yes / compile no / memory medium; Static: yes / yes / yes / high; Quantized: no / no / no / low). StaticCache: "having a fixed (usually large) size for the key/value states means that while generating, a lot of tokens will actually be masked as they should not take part in the attention ... a trade-off which should be very good if you generate with several sequence of more or less the same lengths, but may be sub-optimal if you have for example 1 very large sequence, and then only short sequences." cache_implementation="static" "will also turn on automatic compilation of the decoding stage for greedy and sample decoding strategies." Offloading: offloading=True on DynamicCache or StaticCache; "By default, this option [offload_only_non_sliding] is False for DynamicCache, and True for StaticCache." Quantized backends: hqq (int2/int4/int8), quanto (int2/int4, the default); warning that quantization "can harm latency if the context length is short". Mamba-class models use a model-specific cache. ↩↩↩↩↩↩↩↩↩↩

  4. Not Lain, "KV Caching Explained: Optimizing Transformer Inference Efficiency," Hugging Face community blog, 30 Jan 2025 https://huggingface.co/blog/not-lain/kv-caching. Benchmark: SmolLM2-1.7B, max_new_tokens=300, T4 GPU, "with KV Caching 11.7 s / Standard Inference 1min 1s / ~5.21x times faster". The post's PyTorch pseudocode uses torch.cat([...], dim=1). Honest limits of this source, recorded because the page corrects them: it does not explain why Q is uncached, gives no memory formula, states no cache tensor shapes, and names no Cache class (it mentions only the use_cache and cache_implementation parameters). ↩↩

  5. João Lages, "Transformers KV Caching Explained" https://medium.com/@joaolages/kv-caching-explained-276520203249. Benchmark: GPT-2 on a Google Colab Tesla T4, 1000 new tokens: "with KV caching: 11.885 +- 0.272 seconds", "without KV caching: 56.197 +- 1.855 seconds" (about 4.7x). The article describes the cached path as producing matrices that are "way smaller"; that is true of the query and its score matrix and false of the cache itself, which grows monotonically. The URL returns HTTP 403 to non-browser clients; the content above was read through a rendering fetch, not curl. ↩↩

  6. NVIDIA, "Mastering LLM Techniques: Inference Optimization" https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/. "Size of KV cache per token in bytes = 2 * (num_layers) * (num_heads * dim_head) * precision_in_bytes"; "with a Llama 2 7B model in 16-bit precision and a batch size of 1, the size of the KV cache will be 1 * 4096 * 2 * 32 * 4096 * 2 bytes, which is ~2 GB." The formula carries no num_kv_heads term and is therefore MHA-only; the example is correct because Llama-2-7B is MHA. ↩↩

  7. Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need," arXiv:1911.02150 https://arxiv.org/abs/1911.02150. ↩

  8. Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints," arXiv:2305.13245 https://arxiv.org/abs/2305.13245. ↩