Cross-layer KV sharing: CLA and xKV¶

Markdown

Scope: reducing KV cache size along the layer axis, by having several layers share one cached tensor. Two mechanisms are covered: Cross-Layer Attention (CLA), an architecture change that requires pretraining, and xKV, a post-training factorization of grouped layers into a shared low-rank subspace. Also covered: the screening statistic that decides whether a layer group is compressible at all, and the capacity-versus-bandwidth distinction that determines whether this shows up as a throughput win. Dropping cached tokens is token eviction; routing a different compression recipe per layer under one budget is per-layer heterogeneous compression, which treats the layer as a routing axis rather than a sharing axis; sharing KV heads across query heads within a layer is GQA/MQA, covered in FlashAttention and MLA.

The numpy blocks below are runnable, self-checking validations of the mechanisms this page teaches: the CKA-versus-cosine dissociation that motivates xKV, the joint-versus-per-layer rank budget (including an adversarial control where grouping must not help), and the CLA byte arithmetic separating stored bytes from bytes read per decode step. Each was executed on a stock python3 with numpy and asserts its result; the pasted output is the real output. The synthetic caches are built to have the structural property each paper claims about real caches; they validate the mechanism and the screening rule, not the published accuracy or compression numbers, which are quoted from the papers and were not reproduced here.

What it is¶

The KV cache is 2 * n_layers * seq_len * n_kv_heads * head_dim * dtype_bytes per sequence. Most compression work attacks the seq_len factor (eviction), the dtype_bytes factor (quantization), or the n_kv_heads factor (GQA/MQA). Cross-layer sharing attacks n_layers, by storing fewer distinct per-layer caches than the model has layers.

CLA changes the architecture. Only a subset of layers computes K/V projections; the rest reuse a previous layer's KV activations. The sharing factor names the configuration: CLA2 shares each KV projection across a pair of adjacent layers, CLA3 across three, and so on. Because this changes the model, it requires pretraining from scratch.¹
xKV changes nothing about the model. It takes the KV cache of a group of adjacent layers at inference time and jointly factorizes it into a shared token basis plus small per-layer coefficients, exploiting the observation that the dominant singular vectors of different layers' caches are already aligned. It is post-training and plug-and-play.²

The two are alternatives at different points in the lifecycle, not competitors on equal footing: if you are training a model, CLA is available; if you are serving someone else's checkpoint, only xKV is.

Why use it¶

The layer axis is large and mostly untouched by the compression methods already deployed in most stacks. It also composes: CLA is explicitly orthogonal to MQA/GQA/MHA and can be combined with any of them,¹ and xKV reports composing with quantization and head-level eviction.² For a stack that has already quantized to FP8 and enabled GQA, the layer axis is often the only remaining lever that does not touch token content.

Reported results:

CLA2 combined with MQA achieves a 2x KV cache reduction against a plain MQA baseline with minimal perplexity degradation, and this holds at both 1B and 3B parameter scales when every model is compared at its own tuned learning rate.¹
xKV reports up to 8x compression on Llama-3.1, Qwen2.5 and DeepSeek-V2 with at most 3% accuracy loss on RULER and LongBench, and up to 4.23x end-to-end throughput improvement when paired with its Selective Reconstruction decode path.²

When to use it (and when not)¶

Use CLA when you control pretraining and KV memory is your binding constraint at serving time. Use a sharing factor of 2: the authors found CLA2 outperforms larger sharing factors, and that CLA is most robust when combined with MQA.¹

Use xKV when you are serving an existing checkpoint, contexts are long, and you have verified that your model's layers are actually aligned (see the screening rule below). It is a prefill-side transform, so it suits long-context, low-turn workloads better than short-prompt chat.

Do not use either when:

You are trying to reduce decode latency directly. CLA does not do this, by construction. See the next section; it is the single most misread property of the technique.
Your layers are not aligned. xKV's benefit is entirely contingent on cross-layer subspace alignment, and the screening statistic below will tell you before you pay for the factorization.
You are memory-bound on weights rather than cache. At short contexts and small batches the KV cache is not what is running you out of HBM.

CLA buys capacity, not bandwidth¶

CLA reduces how many distinct KV tensors are stored. It does not reduce how many bytes attention reads per decode step, because a layer that reuses an earlier layer's cache must still read that cache when its turn comes. The paper states this directly: CLA "has no direct effect on the memory bandwidth consumed by the attention mechanism in each decoding step" and therefore "no direct effect on the latency of the core attention computation during decoding".¹

The benefit is indirect and real: a smaller cache means larger batches and longer contexts fit, and larger batches raise throughput. But if you deploy CLA expecting per-token decode latency to fall at fixed batch size, you will measure nothing. The block below makes the distinction explicit.

import numpy as np

def cla_bytes(n_layers, share, n_kv_heads, head_dim, seq, dtype=2):
    """Return (bytes stored, bytes read per decode step) under a CLA sharing factor."""
    n_unique = int(np.ceil(n_layers / share))
    per_layer = 2 * n_kv_heads * head_dim * seq * dtype
    return n_unique * per_layer, n_layers * per_layer

base_s, base_r = cla_bytes(32, 1, 8, 128, 8192)
for f in (2, 3, 4):
    s, r = cla_bytes(32, f, 8, 128, 8192)
    print(f"CLA{f}: stored {s/2**20:7.1f} MiB ({base_s/s:.2f}x less) | "
          f"read/step {r/2**20:7.1f} MiB ({base_r/r:.2f}x less)")
    assert r == base_r, "CLA must not change bytes read per decode step"

s3, _ = cla_bytes(32, 3, 8, 128, 8192)
print(f"depth 32 under CLA3: {base_s/s3:.3f}x, not 3.000x (ceil(32/3) = 11 unique caches)")
assert base_s / s3 < 3.0

CLA2: stored   512.0 MiB (2.00x less) | read/step  1024.0 MiB (1.00x less)
CLA3: stored   352.0 MiB (2.91x less) | read/step  1024.0 MiB (1.00x less)
CLA4: stored   256.0 MiB (4.00x less) | read/step  1024.0 MiB (1.00x less)
depth 32 under CLA3: 2.909x, not 3.000x (ceil(32/3) = 11 unique caches)

Two operational notes fall out. The stored-bytes reduction is the sharing factor only when it divides depth evenly, otherwise slightly less. And under pipeline parallelism, layers sharing a cache must either sit in the same pipeline stage or the KV activations must be communicated across the stage boundary.¹ That constraint interacts with placement decisions and is easy to discover late.

Architecture: why grouping works at all¶

xKV rests on a specific empirical claim, and the claim is not the obvious one. Prior post-training work (MiniCache) merged adjacent layers' caches assuming high token-wise cosine similarity. xKV's analysis shows that assumption is weak in practice: adjacent layers show surprisingly low token-level similarity, which is why interpolation-based merging is limited to roughly 1.2x compression and degrades accuracy.²

What is shared is the geometry. Measured by Centered Kernel Alignment, which compares centered Gram matrices rather than individual vectors, adjacent layers score consistently high. A high CKA implies the dominant left singular vectors of the two layers' caches are strongly aligned, meaning the basis vectors describing the principal variation in token space are shared even though individual token vectors are not similar.²

That dissociation is directly demonstrable, and it is the whole reason a shared basis compresses when merging does not.

import numpy as np
rng = np.random.default_rng(0)

def cka(x, y):
    n = x.shape[0]
    h = np.eye(n) - np.ones((n, n)) / n
    gx, gy = h @ x @ x.T @ h, h @ y @ y.T @ h
    return np.trace(gx @ gy) / np.sqrt(np.trace(gx @ gx) * np.trace(gy @ gy))

def token_cosine(x, y):
    num = np.sum(x * y, axis=1)
    den = np.linalg.norm(x, axis=1) * np.linalg.norm(y, axis=1)
    return float(np.mean(num / den))

L, d, r = 512, 128, 24
T = rng.standard_normal((L, r))                       # one shared token basis
layers = [T @ rng.standard_normal((r, d)) for _ in range(4)]           # aligned
indep  = [rng.standard_normal((L, r)) @ rng.standard_normal((r, d))
          for _ in range(4)]                                            # control

def mean_pairwise(g, fn):
    return float(np.mean([fn(g[i], g[i+1]) for i in range(len(g)-1)]))

cos_s, cka_s = mean_pairwise(layers, token_cosine), mean_pairwise(layers, cka)
cos_i, cka_i = mean_pairwise(indep, token_cosine), mean_pairwise(indep, cka)
print(f"shared basis: token cosine {cos_s:+.3f} | CKA {cka_s:+.3f}")
print(f"independent : token cosine {cos_i:+.3f} | CKA {cka_i:+.3f}")
assert abs(cos_s - cos_i) < 0.05, "cosine does not separate the groups"
assert cka_s > 4 * cka_i, "CKA must separate shared from independent bases"
print(f"cosine gap {abs(cos_s-cos_i):.3f} vs CKA gap {cka_s-cka_i:.3f}")

shared basis: token cosine -0.010 | CKA +0.842
independent : token cosine +0.002 | CKA +0.037
cosine gap 0.012 vs CKA gap 0.805

Cosine similarity is near zero for both groups and cannot tell them apart. CKA separates them by 0.805. A method that screens on cosine would conclude neither group is compressible; a method that screens on CKA correctly identifies the first.

How to use it: does grouping actually reduce the rank budget¶

The screening statistic is only useful if it predicts the thing you care about, which is bytes. The block below measures the actual storage budget two ways at matched reconstruction fidelity: per-layer SVD (each layer factorized alone) versus one shared basis across the group with per-layer coefficients. It runs on both the aligned group and the independent control, so the adversarial case is covered rather than assumed.

def rank_for_energy(mat, frac=0.95):
    s = np.linalg.svd(mat, compute_uv=False)
    e = np.cumsum(s ** 2) / np.sum(s ** 2)
    return int(np.searchsorted(e, frac) + 1)

def budgets(group, frac=0.95):
    """Floats stored per layer: per-layer SVD vs one shared cross-layer basis."""
    w = len(group)
    per = sum(rank_for_energy(m, frac) * (L + d) for m in group)
    kj = rank_for_energy(np.concatenate(group, axis=1), frac)
    shared = kj * L + kj * d * w          # one basis + per-layer coefficients
    return per / w, shared / w

per_l, joint_l = budgets(layers)
print(f"aligned group:     per-layer {per_l:8.0f} floats/layer | joint {joint_l:8.0f} "
      f"-> {per_l/joint_l:.2f}x better")
assert joint_l < per_l

per_i, joint_i = budgets(indep)
print(f"independent group: per-layer {per_i:8.0f} floats/layer | joint {joint_i:8.0f} "
      f"-> {per_i/joint_i:.2f}x")
assert joint_i >= per_i, "joint factorization must not help when bases are independent"

aligned group:     per-layer    13440 floats/layer | joint     5632 -> 2.39x better
independent group: per-layer    13600 floats/layer | joint    20736 -> 0.66x

Grouping is worth 2.39x on the aligned group and is actively harmful on the independent one, costing 1.5x more than compressing each layer separately. This is the practical rule the mechanism implies: compute CKA over candidate layer groups before enabling cross-layer factorization, and group only where it is high. Uniformly grouping every W consecutive layers because the paper used W consecutive layers will lose memory on the parts of the network where layers are not aligned.

How to integrate it¶

xKV is a prefill-side transform on the cache, so it slots in after prefill and before decode. Its decode-time companion, Selective Reconstruction, reconstructs only the tokens relevant to the query rather than the whole cache, which is what turns a memory win into a throughput win (up to 4.23x end-to-end, and 30% higher throughput than baselines at similar accuracy).² Without SR you have paid reconstruction cost on every decode step.
CLA is a model architecture decision, taken before training. Its serving-side integration is mostly the pipeline-parallel placement constraint noted above.
Both compose with the axes already in your stack. xKV reports composing with KV quantization and structured head-level eviction; CLA is orthogonal to MQA/GQA.²¹
Note the pre-RoPE detail: xKV's factorization is defined over the pre-RoPE key cache or the value cache.² Applying it to post-RoPE keys mixes position into the basis and is not what was measured.

How to run it in production¶

Measure throughput, not decode latency, when validating CLA. The correct success metric is tokens per second at the batch size the smaller cache now permits, or the longest context that now fits. A latency benchmark at fixed batch size will correctly show no change.
Re-screen CKA per model. Alignment is a property of the trained weights. A new checkpoint, or a fine-tune, can change which layer groups are compressible.
Validate on long-context retrieval, not perplexity. Both papers evaluate on RULER, LongBench or downstream benchmarks for a reason: a shared basis that averages away a rarely-attended token is nearly invisible to perplexity and fatal to needle-in-a-haystack retrieval.
Watch the reconstruction cost. Any low-rank scheme trades memory for compute at read time. If you are already compute-bound during decode, the trade may be negative even when the memory math looks good.

How to maintain it¶

Re-run the CKA screen and the rank-budget comparison after any weight change, using the two blocks above against your own cache dumps rather than synthetic data.
Track the sharing factor decision separately from the compression ratio. CLA3 and CLA4 achieved Pareto improvements over plain MQA but were worse than CLA2 at matched footprint, so a larger factor is not a safe default.¹
If you tune learning rates, retune for CLA models specifically. The authors found CLA models prefer higher learning rates (2.25e-3 against 1.5e-3 for the corresponding baseline at 1B scale), and the CLA-versus-baseline comparison only holds when both are tuned.¹

Failure modes¶

Expecting CLA to cut decode latency. It cannot, by construction; shared caches are still re-read per layer.
Screening cross-layer redundancy with cosine similarity. It is near zero even when layers share a basis, which is exactly why the interpolation-based prior work was limited to about 1.2x.²
Applying a uniform group width across the whole network. Where layers are not aligned, joint factorization costs more than per-layer compression, as the control case above shows.
Assuming the nominal sharing factor is the memory win. CLA3 on a 32-layer model gives 2.909x, not 3x.
Factorizing post-RoPE keys. The published method operates on pre-RoPE keys.
Adopting CLA from a benchmark run at a single learning rate. The comparison is sensitive to it, and the CLA model's optimum differs from the baseline's.¹
Reading the 3B-scale result as a straight replication of the 1B result. At 3B the authors report an outcome different from what they expected, with MQA-CLA2 beating the same-head-dimension MQA model outright rather than trading a little perplexity for half the cache.¹

Open questions and validation¶

Nothing on this page reproduces the published accuracy or compression numbers. The executed blocks validate the mechanisms and the screening rule only. xKV publishes code; CLA's experiments are pretraining runs that are expensive to replicate.
Whether xKV's alignment property holds for models trained with very different recipes, or for MLA-style architectures where the cache is already a latent projection, is not established here.
How cross-layer factorization interacts with token eviction is unclear in the direction that matters: evicting tokens changes the token basis, so a basis fitted before eviction may not be valid after it.
Whether cross-layer sharing composes with cross-model transfer, where a mapped cache is already an approximation, is unaddressed in both literatures.

References¶

Brandon, Mishra, Nrusimha, Panda and Ragan-Kelley, Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (arXiv 2405.12981, NeurIPS 2024): https://arxiv.org/abs/2405.12981
Chang et al., xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction (arXiv 2503.18893, ICML 2026): https://arxiv.org/abs/2503.18893 and https://github.com/abdelfattah-lab/xKV
Kornblith et al., Similarity of Neural Network Representations Revisited (ICML 2019), the source of the CKA statistic xKV screens with: https://arxiv.org/abs/1905.00414
Liu et al., MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (arXiv 2405.14366), the SLERP-based interpolation baseline xKV argues against: https://arxiv.org/abs/2405.14366
Wu and Tu, Layer-Condensed KV Cache for Efficient Inference of Large Language Models (ACL 2024): https://arxiv.org/abs/2405.10637

Brandon, Mishra, Nrusimha, Panda and Ragan-Kelley, "Reducing Transformer Key-Value Cache Size with Cross-Layer Attention", arXiv 2405.12981v1, 21 May 2024, MIT CSAIL and MIT-IBM Watson AI Lab, NeurIPS 2024. CLA computes K/V projections for only a subset of layers; remaining layers reuse a previous layer's KV activations. Sharing factor names the configuration (CLA2 = pairs of adjacent layers). Explicitly orthogonal to MHA/MQA/GQA and combinable with any of them. Systems properties enumerated in section 2.3: KV cache memory shrinks by the sharing factor "or slightly less if the sharing factor does not evenly divide the number of layers"; training-time KV activation memory shrinks; compatible with tensor parallelism, but under pipeline parallelism "either different layers which share a KV cache must be kept in the same pipeline stage, or else KV activations must be communicated between pipeline stages"; parameters and FLOPs fall slightly; decode latency may improve indirectly via larger batches and longer cache persistence; and critically, "Unlike MQA and GQA, CLA has no direct effect on the memory bandwidth consumed by the attention mechanism in each decoding step, because even shared KV cache layers must be separately re-read from main memory in each attention layer. CLA therefore has no direct effect on the latency of the core attention computation during decoding." Experiments: models trained from scratch at 1B and 3B on SlimPajama with the GPT-NeoX tokenizer. Design-space exploration at 1B used LR 3e-4 for all models; MQA-CLA2 models at head dims 64, 90 and 128 matched baseline KV footprints while improving perplexity by 0.21 to 0.48 points, and MQA-CLA2 achieved the best accuracy/memory tradeoffs overall. Learning-rate tuning (Table 3) found optima of 1.5e-3 for H128-MQA and 2.25e-3 for both H64-MQA and H128-MQA-CLA2; at their best learning rates the CLA2 model gave up only 0.04 points of validation perplexity against the H128 baseline while halving the cache, and improved 0.31 points over the H64 baseline at matched footprint. On Wikitext the tuned CLA2 model was 0.01 points better than the H128 baseline and 0.71 better than H64. Ablations: GQA+CLA2 only beat its matched baseline in the GQA2-CLA2 configuration, and then only matched MQA-CLA2 within 0.01 points; MQA-CLA3 and MQA-CLA4 Pareto-improved over plain MQA but were worse than MQA-CLA2 at the same footprint; non-uniform sharing patterns (KeepEnds, DenseFront, DenseBack) did not beat uniform CLA2 and cost slightly more memory. At 3B scale, after the same learning-rate tuning protocol, the authors report a result "different than we had expected": MQA-CLA2 at head dim 128 achieved substantially better perplexity than the plain MQA model with the same head dimension despite half the KV cache capacity. ↩↩↩↩↩↩↩↩↩↩↩
Chang, Lin, Lin, Chiang, Akhauri, Li, Jiang, Dai, Ceze, Wu and Abdelfattah, "xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction", arXiv 2503.18893 (v1 24 Mar 2025, v2 27 May 2026), ICML 2026 (PMLR 306); code at https://github.com/abdelfattah-lab/xKV. Note the arXiv listing title differs from the reference form used by other papers ("xKV: Cross-Layer SVD for KV-cache compression"). Motivation: prior inter-layer work splits into architecture changes requiring pretraining (CLA, YOCO) and post-hoc interpolation merging under a cosine-similarity assumption (MiniCache, via SLERP), the latter limited to about 1.2x compression with non-trivial accuracy degradation. Analysis (Figure 2, Llama-3.1-8B-Instruct on RULER multi-valued NIAH): token-wise cosine similarity between adjacent layers is surprisingly low, while CKA between adjacent layers is consistently high; the paper proves in Appendix A that high CKA implies the dominant left singular vectors of the two caches are strongly aligned. Rank analysis shows the rank ratio needed to capture 95% of cumulative eigenvalues falls as more layers are grouped and horizontally concatenated. Method: cross-layer factorization (CLF) extracts a shared token basis across W layers (W > 2) with per-layer reconstruction matrices; training-free and plug-and-play; defined over the pre-RoPE key cache or the value cache. Selective Reconstruction (SR) reconstructs at decode time only the tokens relevant to the query rather than all tokens. Reported results: up to 8x KV-cache compression on Llama-3.1, Qwen2.5 and DeepSeek-V2 with at most 3% accuracy loss on RULER and LongBench and in multi-turn settings; attention latency reduced up to 3.6x; end-to-end throughput improved up to 4.23x with SR; 30% higher throughput than notable baselines at similar accuracy. Reported to compose with KV quantization and structured head-level eviction. None of these numbers were reproduced for this page. ↩↩↩↩↩↩↩↩↩