Cross-layer KV sharing: CLA and xKV¶

Markdown

Scope: reducing KV cache size along the layer axis, by having several layers share one cached tensor. Two mechanisms are covered: Cross-Layer Attention (CLA), an architecture change that requires pretraining, and xKV, a post-training factorization of grouped layers into a shared low-rank subspace. Also covered: the screening statistic that decides whether a layer group is compressible at all, and the capacity-versus-bandwidth distinction that determines whether this shows up as a throughput win. Dropping cached tokens is token eviction; routing a different compression recipe per layer under one budget is per-layer heterogeneous compression, which treats the layer as a routing axis rather than a sharing axis; sharing KV heads across query heads within a layer is GQA/MQA, covered in FlashAttention and MLA.

The numpy blocks below are runnable, self-checking validations of the mechanisms this page teaches: the CKA-versus-cosine dissociation that motivates xKV, the joint-versus-per-layer rank budget (including an adversarial control where grouping must not help), and the CLA byte arithmetic separating stored bytes from bytes read per decode step. Each was executed on a stock python3 with numpy and asserts its result; the pasted output is the real output. The CKA block and the rank-budget block that follows it share state and must be run chained, in one session or as one concatenated file. The synthetic caches are built to have the structural property each paper claims about real caches; they validate the mechanism and the screening rule, not the published accuracy or compression numbers, which are quoted from the papers and were not reproduced here.

What it is¶

The KV cache is 2 * n_layers * seq_len * n_kv_heads * head_dim * dtype_bytes per sequence. Most compression work attacks the seq_len factor (eviction), the dtype_bytes factor (quantization), or the n_kv_heads factor (GQA/MQA). Cross-layer sharing attacks n_layers, by storing fewer distinct per-layer caches than the model has layers.

CLA changes the architecture. Only a subset of layers computes K/V projections; the rest reuse a previous layer's KV activations. The sharing factor names the configuration: CLA2 shares each KV projection across a pair of adjacent layers, CLA3 across three, and so on. Because this changes the model, it requires pretraining from scratch.¹
xKV changes nothing about the model. It takes the KV cache of a group of adjacent layers at inference time and jointly factorizes it into a shared token basis plus small per-layer coefficients, exploiting the observation that the dominant singular vectors of different layers' caches are already aligned. It is post-training and plug-and-play.²

The two are alternatives at different points in the lifecycle, not competitors on equal footing: if you are training a model, CLA is available; if you are serving someone else's checkpoint, only xKV is.

Why use it¶

The layer axis is large and mostly untouched by the compression methods already deployed in most stacks. It also composes: CLA is explicitly orthogonal to MQA/GQA/MHA and can be combined with any of them,¹ and xKV stacks with round-to-nearest KV quantization on top of its own factorization for roughly 25x total compression (numbers below).² For a stack that has already quantized to FP8 and enabled GQA, the layer axis is often the only remaining lever that does not touch token content.

Reported results:

CLA2 combined with MQA achieves a 2x KV cache reduction against a plain MQA baseline with minimal perplexity degradation, and this holds at both 1B and 3B parameter scales when every model is compared at its own tuned learning rate.¹
xKV's contribution list claims "8× compression on Llama-3.1, Qwen2.5, and DeepSeek-V2 with ≤ 3% accuracy loss" across RULER and LongBench, and up to 4.23x end-to-end throughput improvement when paired with its Selective Reconstruction decode path.² The models actually carried through the main evaluation are Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct-1M and Qwen3-4B-Instruct-2507; the paper's only DeepSeek evidence sits in an appendix, on different benchmarks and at a much lower ratio, and that gap is unresolved in the source.⁴

When to use it (and when not)¶

Use CLA when you control pretraining and KV memory is your binding constraint at serving time. Use a sharing factor of 2: the authors found CLA2 outperforms larger sharing factors, and that CLA is most robust when combined with MQA.¹

Use xKV when you are serving an existing checkpoint, contexts are long, and you have verified that your model's layers are actually aligned (see the screening rule below). It is a prefill-side transform, so it suits long-context, low-turn workloads better than short-prompt chat.

Do not use either when:

You are trying to reduce decode latency directly. CLA does not do this, by construction. See the next section; it is the single most misread property of the technique.
Your layers are not aligned. xKV's benefit is entirely contingent on cross-layer subspace alignment, and the screening statistic below will tell you before you pay for the factorization.
You are memory-bound on weights rather than cache. At short contexts and small batches the KV cache is not what is running you out of HBM.

CLA buys capacity, not bandwidth¶

CLA reduces how many distinct KV tensors are stored. It does not reduce how many bytes attention reads per decode step, because a layer that reuses an earlier layer's cache must still read that cache when its turn comes. The paper states this directly: CLA "has no direct effect on the memory bandwidth consumed by the attention mechanism in each decoding step" and therefore "no direct effect on the latency of the core attention computation during decoding".¹

The benefit is indirect and real: a smaller cache means larger batches and longer contexts fit, and larger batches raise throughput. But if you deploy CLA expecting per-token decode latency to fall at fixed batch size, you will measure nothing. The block below makes the distinction explicit.

import numpy as np

def cla_bytes(n_layers, share, n_kv_heads, head_dim, seq, dtype=2):
    """Return (bytes stored, bytes read per decode step) under a CLA sharing factor."""
    n_unique = int(np.ceil(n_layers / share))
    per_layer = 2 * n_kv_heads * head_dim * seq * dtype
    return n_unique * per_layer, n_layers * per_layer

base_s, base_r = cla_bytes(32, 1, 8, 128, 8192)
for f in (2, 3, 4):
    s, r = cla_bytes(32, f, 8, 128, 8192)
    print(f"CLA{f}: stored {s/2**20:7.1f} MiB ({base_s/s:.2f}x less) | "
          f"read/step {r/2**20:7.1f} MiB ({base_r/r:.2f}x less)")
    assert r == base_r, "CLA must not change bytes read per decode step"

s3, _ = cla_bytes(32, 3, 8, 128, 8192)
print(f"depth 32 under CLA3: {base_s/s3:.3f}x, not 3.000x (ceil(32/3) = 11 unique caches)")
assert base_s / s3 < 3.0

CLA2: stored   512.0 MiB (2.00x less) | read/step  1024.0 MiB (1.00x less)
CLA3: stored   352.0 MiB (2.91x less) | read/step  1024.0 MiB (1.00x less)
CLA4: stored   256.0 MiB (4.00x less) | read/step  1024.0 MiB (1.00x less)
depth 32 under CLA3: 2.909x, not 3.000x (ceil(32/3) = 11 unique caches)

Two operational notes fall out. The stored-bytes reduction is the sharing factor only when it divides depth evenly, otherwise slightly less. And under pipeline parallelism, layers sharing a cache must either sit in the same pipeline stage or the KV activations must be communicated across the stage boundary.¹ That constraint interacts with placement decisions and is easy to discover late.

Architecture: why grouping works at all¶

xKV rests on a specific empirical claim, and the claim is not the obvious one. Prior post-training work (MiniCache) merged adjacent layers' caches assuming high token-wise cosine similarity. xKV's analysis shows that assumption is weak in practice: adjacent layers show surprisingly low token-level similarity, which is why interpolation-based merging is confined to a very low compression ratio and degrades accuracy there. The paper states that ceiling two different ways: Figure 1 labels the prior approach "Training Free Limited 1.2x Comp. Rate", while section 4.1 reports that "MiniCache suffers dramatic accuracy loss even at a modest 1.3× compression rate".² Either way the usable headroom is under 1.5x, so the exact figure does not change a deployment decision.

What is shared is the geometry. Measured by Centered Kernel Alignment, which compares centered Gram matrices rather than individual vectors, adjacent layers score consistently high. A high CKA implies the dominant left singular vectors of the two layers' caches are strongly aligned, meaning the basis vectors describing the principal variation in token space are shared even though individual token vectors are not similar.²

That dissociation is directly demonstrable, and it is the whole reason a shared basis compresses when merging does not. The mechanism and the screen fit together as one gate: CKA decides whether a candidate window is compressible, and only a window that passes is replaced by a shared basis plus per-layer reconstruction matrices.

flowchart TB
  KV["Per-layer KV caches, layers l .. l+W-1<br/>(pre-RoPE keys or values)"] --> SCREEN{"CKA over the candidate window<br/>(centered Gram matrices, not token cosine)"}
  SCREEN -->|"low: bases unrelated"| KEEP["Do not group.<br/>Per-layer SVD, or leave uncompressed"]
  SCREEN -->|"high: dominant singular vectors aligned"| CLF["Cross-layer factorization over the<br/>horizontally concatenated window"]
  CLF --> BASIS["One shared token basis U<br/>rank rK_pre or rV, stored once per window"]
  CLF --> COEF["Per-layer reconstruction matrices<br/>one small matrix per layer in the window"]
  BASIS --> OUT["Window stored as basis + W coefficient blocks"]
  COEF --> OUT
  KEEP --> OUT2["Window stored per layer"]
  NOTE["Interleaved-attention models: never let one<br/>window straddle sliding-window and full-attention layers"] -.-> SCREEN

import numpy as np
rng = np.random.default_rng(0)

def cka(x, y):
    n = x.shape[0]
    h = np.eye(n) - np.ones((n, n)) / n
    gx, gy = h @ x @ x.T @ h, h @ y @ y.T @ h
    return np.trace(gx @ gy) / np.sqrt(np.trace(gx @ gx) * np.trace(gy @ gy))

def token_cosine(x, y):
    num = np.sum(x * y, axis=1)
    den = np.linalg.norm(x, axis=1) * np.linalg.norm(y, axis=1)
    return float(np.mean(num / den))

L, d, r = 512, 128, 24
T = rng.standard_normal((L, r))                       # one shared token basis
layers = [T @ rng.standard_normal((r, d)) for _ in range(4)]           # aligned
indep  = [rng.standard_normal((L, r)) @ rng.standard_normal((r, d))
          for _ in range(4)]                                            # control

def mean_pairwise(g, fn):
    return float(np.mean([fn(g[i], g[i+1]) for i in range(len(g)-1)]))

cos_s, cka_s = mean_pairwise(layers, token_cosine), mean_pairwise(layers, cka)
cos_i, cka_i = mean_pairwise(indep, token_cosine), mean_pairwise(indep, cka)
print(f"shared basis: token cosine {cos_s:+.3f} | CKA {cka_s:+.3f}")
print(f"independent : token cosine {cos_i:+.3f} | CKA {cka_i:+.3f}")
assert abs(cos_s - cos_i) < 0.05, "cosine does not separate the groups"
assert cka_s > 4 * cka_i, "CKA must separate shared from independent bases"
print(f"cosine gap {abs(cos_s-cos_i):.3f} vs CKA gap {cka_s-cka_i:.3f}")

shared basis: token cosine -0.010 | CKA +0.842
independent : token cosine +0.002 | CKA +0.037
cosine gap 0.012 vs CKA gap 0.805

Cosine similarity is near zero for both groups and cannot tell them apart. CKA separates them by 0.805. A method that screens on cosine would conclude neither group is compressible; a method that screens on CKA correctly identifies the first.

Appendix F of the xKV paper widens the CKA measurement to two architectures outside the main evaluation set: Llama-3.2-1B (small dense) and GPT-OSS 120B (a hybrid MoE with an interleaved 1:1 ratio of sliding-window and full-attention layers). Its Figure 8 carries a third panel, Qwen3-4B-Instruct, which is already one of the three main evaluation models (section 4, "Models"), so it re-measures rather than extends. The alignment is "clearly preserved" in all three. The hybrid case yields a concrete grouping rule rather than a general reassurance: on GPT-OSS the authors "specifically noted that CKA similarity is highest between adjacent layers of the same attention type (e.g., Window→Window or Full→Full)".² On any interleaved-attention model, group within an attention type and do not let a window straddle the boundary between a sliding-window layer and a full-attention layer, because that boundary is where alignment drops.

How to use it: does grouping actually reduce the rank budget¶

The screening statistic is only useful if it predicts the thing you care about, which is bytes. The block below measures the actual storage budget two ways at matched reconstruction fidelity: per-layer SVD (each layer factorized alone) versus one shared basis across the group with per-layer coefficients. It runs on both the aligned group and the independent control, so the adversarial case is covered rather than assumed.

def rank_for_energy(mat, frac=0.95):
    s = np.linalg.svd(mat, compute_uv=False)
    e = np.cumsum(s ** 2) / np.sum(s ** 2)
    return int(np.searchsorted(e, frac) + 1)

def budgets(group, frac=0.95):
    """Floats stored per layer: per-layer SVD vs one shared cross-layer basis."""
    w = len(group)
    per = sum(rank_for_energy(m, frac) * (L + d) for m in group)
    kj = rank_for_energy(np.concatenate(group, axis=1), frac)
    shared = kj * L + kj * d * w          # one basis + per-layer coefficients
    return per / w, shared / w

per_l, joint_l = budgets(layers)
print(f"aligned group:     per-layer {per_l:8.0f} floats/layer | joint {joint_l:8.0f} "
      f"-> {per_l/joint_l:.2f}x better")
assert joint_l < per_l

per_i, joint_i = budgets(indep)
print(f"independent group: per-layer {per_i:8.0f} floats/layer | joint {joint_i:8.0f} "
      f"-> {per_i/joint_i:.2f}x")
assert joint_i >= per_i, "joint factorization must not help when bases are independent"

aligned group:     per-layer    13440 floats/layer | joint     5632 -> 2.39x better
independent group: per-layer    13600 floats/layer | joint    20736 -> 0.66x

Grouping is worth 2.39x on the aligned group and is actively harmful on the independent one, costing 1.5x more than compressing each layer separately. This is the practical rule the mechanism implies: compute CKA over candidate layer groups before enabling cross-layer factorization, and group only where it is high. Carrying a fixed window across a region where the layers are unrelated loses memory rather than saving it.

Both blocks above share state: the second reuses layers, indep, L and d defined in the first, so run them chained in one session (or concatenate them into one file) to reproduce the pasted output.

Group width: wider is better, and it saturates at 4¶

Width is not a free parameter to be conservative about. Appendix D.4 (Table 8) sweeps the cross-layer window on RULER with Llama-3.1-8B-Instruct at a fixed compression rate, scaling the rank linearly so every row costs the same memory. The rank pairs are (rK_pre, rV) = (96, 144) at W=1, (192, 288) at W=2, (384, 576) at W=4 (the setting aligned with the main tables), and (768, 1152) at W=8.³

Window size W	xKV	xK-SR	xKV-SR
1	45.71	87.17	72.27
2	75.15	88.43	86.06
4	88.50	89.70	89.69
8	88.91	89.74	89.72

W=1 is single-layer SVD, and at identical memory cost it gives up 42.79 points of RULER average against W=4. That gap is the entire contribution of the method: at a fixed budget, sharing a basis across more layers buys reconstruction fidelity back. W=8 adds only 0.41 points over W=4 while raising prefill buffering, so the paper fixes W=4 as the default for every main experiment.² A uniform W=4 is therefore the working configuration, not a shortcut to be corrected; what has to be tuned is where the windows are placed, which is what the CKA screen decides.

How to integrate it¶

xKV is a prefill-side transform on the cache, so it slots in after prefill and before decode. Its decode-time companion, Selective Reconstruction, reconstructs only the tokens relevant to the query rather than the whole cache, which is what turns a memory win into a throughput win (up to 4.23x end-to-end, and 30% higher throughput than baselines at similar accuracy).² Without SR you have paid reconstruction cost on every decode step.
CLA is a model architecture decision, taken before training. Its serving-side integration is mostly the pipeline-parallel placement constraint noted above.
Both compose with the axes already in your stack, though the evidence differs in strength. CLA is stated to be orthogonal to MQA/GQA.¹ For xKV the only composition the paper actually runs is quantization (Appendix D.5, Table 9): round-to-nearest 4-bit applied on top of the factorized cache takes total compression from 8.03x to 25.70x on Llama-3.1-8B-Instruct and moves the RULER average from 88.85 to 87.64, and 3-bit reaches 32.12x at 84.64.² Eviction methods (StreamingLLM, SnapKV, PyramidKV) appear only as compared-against baselines; the paper runs no experiment stacking xKV on top of eviction, so treat that combination as unmeasured.
The quantization result is labelled "preliminary" in the paper and uses the simplest possible quantizer, round-to-nearest. A production stack running KIVI or an FP8 KV path should re-measure rather than inherit the 25.70x figure, especially as the compression rate there is computed against a 64K context.²
Note the pre-RoPE detail: xKV's factorization is defined over the pre-RoPE key cache or the value cache.² Applying it to post-RoPE keys mixes position into the basis and is not what was measured.

How to run it in production¶

Measure throughput, not decode latency, when validating CLA. The correct success metric is tokens per second at the batch size the smaller cache now permits, or the longest context that now fits. A latency benchmark at fixed batch size will correctly show no change.
Re-screen CKA per model. Alignment is a property of the trained weights. A new checkpoint, or a fine-tune, can change which layer groups are compressible.
Validate on long-context retrieval, not perplexity. Both papers evaluate on RULER, LongBench or downstream benchmarks for a reason: a shared basis that averages away a rarely-attended token is nearly invisible to perplexity and fatal to needle-in-a-haystack retrieval.
Watch the reconstruction cost. Any low-rank scheme trades memory for compute at read time. If you are already compute-bound during decode, the trade may be negative even when the memory math looks good.

How to maintain it¶

Re-run the CKA screen and the rank-budget comparison after any weight change, using the two blocks above against your own cache dumps rather than synthetic data.
Track the sharing factor decision separately from the compression ratio. CLA3 and CLA4 achieved Pareto improvements over plain MQA but were worse than CLA2 at matched footprint, so a larger factor is not a safe default.¹
If you tune learning rates, retune for CLA models specifically. The authors found CLA models prefer higher learning rates (2.25e-3 against 1.5e-3 for the corresponding baseline at 1B scale), and the CLA-versus-baseline comparison only holds when both are tuned.¹

Failure modes¶

Expecting CLA to cut decode latency. It cannot, by construction; shared caches are still re-read per layer.
Screening cross-layer redundancy with cosine similarity. It is near zero even when layers share a basis, which is exactly why the interpolation-based prior work was limited to 1.2x or 1.3x depending on which part of the paper is read.²
Placing a window across a boundary where alignment breaks. Uniform grouping is not the defect: W=4 uniform is xKV's own default and it works. The defect is grouping layers whose bases are unrelated, where joint factorization costs more than per-layer compression (the control case above), and on interleaved-attention models letting one window span both a sliding-window and a full-attention layer, which is precisely where Appendix F measures alignment dropping.²
Shrinking the group width to be conservative. At a fixed compression rate the window is what buys accuracy back: RULER average falls from 88.50 at W=4 to 75.15 at W=2 and 45.71 at W=1.² A cautious W=2 is a worse configuration at the same memory cost, not a safer one.
Assuming xKV stacks on top of token eviction. The paper composes it with quantization only; eviction appears solely as a baseline.²
Assuming the nominal sharing factor is the memory win. CLA3 on a 32-layer model gives 2.909x, not 3x.
Factorizing post-RoPE keys. The published method operates on pre-RoPE keys.
Adopting CLA from a benchmark run at a single learning rate. The comparison is sensitive to it, and the CLA model's optimum differs from the baseline's.¹
Reading the 3B-scale result as a straight replication of the 1B result. At 3B the authors report an outcome different from what they expected, with MQA-CLA2 beating the same-head-dimension MQA model outright rather than trading a little perplexity for half the cache.¹

Open questions and validation¶

Nothing on this page reproduces the published accuracy or compression numbers. The executed blocks validate the mechanisms and the screening rule only. xKV publishes code; CLA's experiments are pretraining runs that are expensive to replicate.
The MLA case is established, but at a much lower ratio than the headline. Appendix E applies xKV at window size 4 to DeepSeek-Coder-V2-Lite-Instruct, which combines Multi-head Latent Attention with a Mixture-of-Experts feedforward, and reports 3x compression on RepoBench-P and 3.5x on LCC measured by edit similarity, without compromising accuracy, while MiniCache and single-layer SVD fail at lower ratios on the same model.² A latent cache is compressible again along the layer axis; it is roughly a 3x lever there rather than the 8x seen on GQA models, and the DeepSeek result was never run on RULER or LongBench.⁴
Whether xKV's alignment property holds for models trained with very different recipes is not established here. Appendix F widens the CKA evidence to Llama-3.2-1B and GPT-OSS 120B (its Figure 8 also re-plots Qwen3-4B-Instruct, a main evaluation model), but that is a measurement of the alignment statistic, not of end-to-end accuracy under compression.²
How cross-layer factorization interacts with token eviction is unclear in the direction that matters: evicting tokens changes the token basis, so a basis fitted before eviction may not be valid after it.
Whether cross-layer sharing composes with cross-model transfer, where a mapped cache is already an approximation, is unaddressed in both literatures.

References¶

Brandon, Mishra, Nrusimha, Panda and Ragan-Kelley, Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (arXiv 2405.12981, NeurIPS 2024): https://arxiv.org/abs/2405.12981
Chang et al., xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction (arXiv 2503.18893, ICML 2026): https://arxiv.org/abs/2503.18893 and https://github.com/abdelfattah-lab/xKV
Kornblith et al., Similarity of Neural Network Representations Revisited (ICML 2019), the source of the CKA statistic xKV screens with: https://arxiv.org/abs/1905.00414
Liu et al., MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (arXiv 2405.14366), the SLERP-based interpolation baseline xKV argues against: https://arxiv.org/abs/2405.14366
Wu and Tu, Layer-Condensed KV Cache for Efficient Inference of Large Language Models (ACL 2024): https://arxiv.org/abs/2405.10637

Brandon, Mishra, Nrusimha, Panda and Ragan-Kelley, "Reducing Transformer Key-Value Cache Size with Cross-Layer Attention", arXiv 2405.12981v1, 21 May 2024, MIT CSAIL and MIT-IBM Watson AI Lab, NeurIPS 2024. CLA computes K/V projections for only a subset of layers; remaining layers reuse a previous layer's KV activations. Sharing factor names the configuration (CLA2 = pairs of adjacent layers). Explicitly orthogonal to MHA/MQA/GQA and combinable with any of them. Systems properties enumerated in section 2.3: KV cache memory shrinks by the sharing factor "or slightly less if the sharing factor does not evenly divide the number of layers"; training-time KV activation memory shrinks; compatible with tensor parallelism, but under pipeline parallelism "either different layers which share a KV cache must be kept in the same pipeline stage, or else KV activations must be communicated between pipeline stages"; parameters and FLOPs fall slightly; decode latency may improve indirectly via larger batches and longer cache persistence; and critically, "Unlike MQA and GQA, CLA has no direct effect on the memory bandwidth consumed by the attention mechanism in each decoding step, because even shared KV cache layers must be separately re-read from main memory in each attention layer. CLA therefore has no direct effect on the latency of the core attention computation during decoding." Experiments: models trained from scratch at 1B and 3B on SlimPajama with the GPT-NeoX tokenizer. Design-space exploration at 1B used LR 3e-4 for all models; MQA-CLA2 models at head dims 64, 90 and 128 matched baseline KV footprints while improving perplexity by 0.21 to 0.48 points, and MQA-CLA2 achieved the best accuracy/memory tradeoffs overall. Learning-rate tuning (Table 3) found optima of 1.5e-3 for H128-MQA and 2.25e-3 for both H64-MQA and H128-MQA-CLA2; at their best learning rates the CLA2 model gave up only 0.04 points of validation perplexity against the H128 baseline while halving the cache, and improved 0.31 points over the H64 baseline at matched footprint. On Wikitext the tuned CLA2 model was 0.01 points better than the H128 baseline and 0.71 better than H64. Ablations: GQA+CLA2 only beat its matched baseline in the GQA2-CLA2 configuration, and then only matched MQA-CLA2 within 0.01 points; MQA-CLA3 and MQA-CLA4 Pareto-improved over plain MQA but were worse than MQA-CLA2 at the same footprint; non-uniform sharing patterns (KeepEnds, DenseFront, DenseBack) did not beat uniform CLA2 and cost slightly more memory. At 3B scale, after the same learning-rate tuning protocol, the authors report a result "different than we had expected": MQA-CLA2 at head dim 128 achieved substantially better perplexity than the plain MQA model with the same head dimension despite half the KV cache capacity. ↩↩↩↩↩↩↩↩↩↩↩
Chang, Lin, Lin, Chiang, Akhauri, Li, Jiang, Dai, Ceze, Wu and Abdelfattah, "xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction", arXiv 2503.18893 (v1 24 Mar 2025, v2 27 May 2026), ICML 2026 (PMLR 306); code at https://github.com/abdelfattah-lab/xKV. The title changed between versions: v1 (24 Mar 2025) was titled "xKV: Cross-Layer SVD for KV-Cache Compression" with seven authors, and v2 (27 May 2026) renamed it to "xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction" with eleven. Citations to the older title point at the same arXiv identifier, not a different paper; everything below is read from v2. Motivation: prior inter-layer work splits into architecture changes requiring pretraining (CLA, YOCO) and post-hoc interpolation merging under a cosine-similarity assumption (MiniCache, via SLERP), the latter limited to a very low compression ratio with non-trivial accuracy degradation (1.2x per Figure 1, 1.3x per section 4.1; see below). Analysis (Figure 2, Llama-3.1-8B-Instruct on RULER multi-valued NIAH): token-wise cosine similarity between adjacent layers is surprisingly low, while CKA between adjacent layers is consistently high; the paper proves in Appendix A that high CKA implies the dominant left singular vectors of the two caches are strongly aligned. Rank analysis shows the rank ratio needed to capture 95% of cumulative eigenvalues falls as more layers are grouped and horizontally concatenated. Method: cross-layer factorization (CLF) extracts a shared token basis across W layers (W > 2) with per-layer reconstruction matrices; training-free and plug-and-play; defined over the pre-RoPE key cache or the value cache. Selective Reconstruction (SR) reconstructs at decode time only the tokens relevant to the query rather than all tokens. Default cross-layer window size W=4 (section 4, citing Appendix D.4); compression rates are computed assuming a 64K context. Main evaluation models (section 4, "Models"): Llama-3.1-8B-Instruct (8 KV heads), Qwen2.5-7B-Instruct-1M (4 KV heads) and Qwen3-4B-Instruct-2507 (8 KV heads), all GQA. Reported results: up to 8x KV-cache compression with at most 3% accuracy loss on RULER and LongBench and in multi-turn settings; 88.50% RULER average on Llama-3.1-8B-Instruct at 8.03x, and 89.22% on Qwen2.5-7B-Instruct-1M, 2.6% off the uncompressed baseline; end-to-end throughput improved up to 4.23x with SR at 122K context; 30% higher throughput than ShadowKV at similar accuracy. Attention-latency figure is inconsistent inside the paper: the contribution bullets in section 1 say "reducing attention latency by up to 3.6×" while section 5 says xKV-SR "achieves up to a 3.5× attention-operation speedup (Figure 5a, 4-122k)"; this page cites neither as a planning number. Same pattern on the MiniCache ceiling: Figure 1 labels prior training-free work "Training Free Limited 1.2x Comp. Rate" while section 4.1 says MiniCache "suffers dramatic accuracy loss even at a modest 1.3× compression rate". Composition evidence: Appendix D.5 (Table 9) is the only composition experiment, applying round-to-nearest quantization to the already-factorized cache on Llama-3.1-8B-Instruct at 64K RULER, giving 8.03x/88.85 avg for xKV alone, 25.70x/87.64 with 4-bit RTN (the body text rounds this to "25.6×") and 32.12x/84.64 with 3-bit (body text "32×"); the authors call these "preliminary experiments". Token eviction (StreamingLLM, PyramidKV, SnapKV) and quantization (KIVI-2) appear only in the baseline list of section 4 and in Table 1, never stacked on top of xKV; there is no head-level-eviction experiment anywhere in v2. Appendix D.4 (Table 8), RULER on Llama-3.1-8B-Instruct at fixed compression rate, xKV column: W=1 45.71, W=2 75.15, W=4 88.50, W=8 88.91 (xK-SR 87.17/88.43/89.70/89.74; xKV-SR 72.27/86.06/89.69/89.72); ranks scaled linearly to hold the rate constant. The Table 8 caption names three window sizes but only two rank pairs, so the rank-to-window assignment is not readable off the caption; it is resolved separately in this page's Table 8 footnote. Appendix F, titled "Broader CKA Analysis", names in prose only two models outside the main set, "a small-scale dense model (Llama-3.2-1B) and a large-scale hybrid/MoE model (GPT-OSS 120B)"; its Figure 8 has three panels, "(a) Llama3.2-1B", "(b) GPT-OSS 120B" and "(c) Qwen3-4B-Instruct", the last of which is a main evaluation model. It finds the alignments "clearly preserved", and notes for GPT-OSS that "CKA similarity is highest between adjacent layers of the same attention type (e.g., Window→Window or Full→Full)". None of these numbers were reproduced for this page. ↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩
The Table 8 caption in arXiv 2503.18893v2 is ambiguous and this page does not treat it as settled. It reads: "We align the rank setting with Table 1 and Table 2 for window size 4. For window sizes 1, 2, and 8, we scaled the rank linearly to maintain the same compression rate, with (rK pre , rV ) = (96, 144) and (192, 288), respectively." Three window sizes are named but only two rank pairs are given, so "respectively" cannot be read off the caption alone. Two independent sources resolve it the same way. (a) The authors' own reproduction script, examples/xKV/eval_ablation_window.sh at github.com/abdelfattah-lab/xKV commit 05d91ecb0d698279aa4220fda5d7a5108036d692, states "base rank (rK_pre, rV) = (96, 144); window W -> rank = (96W, 144W)" and pins the four runs to --rank_k 96/192/384/768 against --layer_group_size 1/2/4/8. (b) The paper's own fixed rate: a window costs 69632 * (rK + rV) against 536,870,912, which is 8.0313 at (384, 576), matching the 8.03x quoted for W=4 throughout the paper, and the linear scaling holds that rate at every other W. The caption's omission of the W=4 pair is the likeliest reading of the mismatch, but the paper does not say so. ↩
Internal inconsistency in arXiv 2503.18893v2 over the DeepSeek result, recorded here rather than resolved, because the paper does not resolve it. The third contribution bullet in section 1 reads: "Across RULER and LongBench, xKV achieves 8× compression on Llama-3.1, Qwen2.5, and DeepSeek-V2 with ≤ 3% accuracy loss." No DeepSeek model appears anywhere in the RULER results (Table 1, Table 2, Appendix D.1 Table 5) or the LongBench results (Appendix D.2, Table 6), and section 4's "Models" paragraph names only Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct-1M and Qwen3-4B-Instruct-2507 for the main evaluation. The paper's only DeepSeek evidence is Appendix E with Figure 7, on DeepSeek-Coder-V2-Lite-Instruct (written "DeepSeek-V2-Coder-Lite" in the Appendix E body, "DeepSeek-Coder-V2-Lite-Instruct" in section 4 and the Figure 7 caption), evaluated on RepoBench-P and LCC scored by edit similarity, where "With a window size of 4, xKV achieves a 3× compression rate on RepoBench (Liu et al., 2023a) and 3.5× on LCC (Guo et al., 2023) without compromising accuracy." The model is therefore not DeepSeek-V2, the benchmarks are not RULER or LongBench, and the ratio is 3x to 3.5x rather than 8x. Appendix E's stated purpose is compatibility with MLA and MoE architectures, not the headline ratio. ↩↩