Markdown

Per-layer heterogeneous KV cache compression¶

Scope: routing a different KV-cache compression recipe, an eviction ratio plus separate K-bit and V-bit widths, to each transformer layer under one global memory budget, instead of applying the same recipe everywhere. Token eviction mechanics and the paged-attention infrastructure that makes eviction actually free memory are covered in KV cache token eviction and compaction; dtype and paged-block basics live in KV cache management. This page covers the layer as a routing axis in its own right, alongside the token axis those pages already cover.

The greedy budget solver below is executed and asserted with the system python3 (numpy), on a synthetic per-layer sensitivity table built to match the cited paper's reported heterogeneity ratios, not its real 28-layer calibration table (which is not published). The algorithm and the qualitative result, heterogeneous per-layer routing beats any uniform allocation at matched memory, and that advantage shrinks when layers are not actually heterogeneous, are validated here; the specific accuracy and compression numbers below are quoted from the paper directly, not reproduced.

What it is¶

Every published KV-cache compression method, token eviction (StreamingLLM, H2O, SnapKV), quantization (KIVI, KVQuant), or a mix, applies the same policy to every transformer layer: one eviction ratio, one bit-width, uniformly.¹ Per-layer heterogeneous compression breaks that assumption directly: each layer gets its own (eviction-ratio, K-bits, V-bits) tuple, chosen by an offline-calibrated solver so that the sum of per-layer memory stays under one global budget while the sum of predicted quality loss is as small as possible.¹

The KV cache at layer ℓ is K_ℓ, V_ℓ ∈ R^(H_kv × T × d_head) (GQA-style key/value heads, sequence length T, per-head dimension d_head). A layer's compressed size is:

m_ℓ(keep_ℓ, k_bits_ℓ, v_bits_ℓ) = keep_ℓ · T_cache · H_kv · d_head · (k_bits_ℓ + v_bits_ℓ) / 8   bytes

and the routing problem is choosing {keep_ℓ, k_bits_ℓ, v_bits_ℓ} across all layers subject to Σ_ℓ m_ℓ ≤ M for a user-specified global budget M.¹ This is a mixture-of-experts framing applied to compression itself: each axis (eviction, K-precision, V-precision) is treated as an expert, and a router selects the per-layer mix, not a token router selecting FFN experts.¹

Why it matters¶

Different layers respond very differently to the same compression operation, and the gap widens sharply as the operation gets more aggressive.¹ Measured on DeepSeek-R1-Distill-Qwen-7B (28 layers), the ratio between the most- and least-sensitive layer's predicted error from the same operation is:¹

Operation	max/min ratio across layers
Evict 10%	1.7x
Evict 25%	1.6x
Evict 50%	2.6x
Evict 75%	548x
Evict 90%	689x
K-quant 8-bit	87x
K-quant 4-bit	15x
V-quant 8-bit	1.6x
V-quant 4-bit	1.2x

Three findings follow directly from this table, and they are the whole justification for routing per layer instead of uniformly:¹

Mild operations are close to uniform; aggressive ones are not. At eviction ratios up to 50%, a uniform policy is roughly defensible (1.6-2.6x spread). Past that, at 75% and 90% eviction, the spread explodes past 500x, and a single global ratio is either far too conservative on the layers that could tolerate more, or silently destructive on the layers that cannot.
K-quantization has real per-layer structure; V-quantization does not. K-quant spans 15-87x across layers depending on bit-width; V-quant barely moves (1.2-1.6x) at either bit-width. Per-layer K-quant routing has headroom to help; per-layer V-quant routing is close to free to skip, applying it uniformly loses almost nothing.
Eviction and quantization trade at different rates on different layers, so the choice of which axis to compress on a given layer is itself layer-specific: on one measured layer, eviction was 2.2x cheaper than an equal-memory quantization step; on another, quantization was 10x cheaper than an equal-memory eviction step.¹ A router that only sees one axis (an eviction-only or quantization-only per-layer scheme) cannot make this trade; only a router that sees both axes jointly can.

The payoff is large where the heterogeneity is large: at 14x compression (136 MB vs. a 1.9 GB uncompressed cache), the heterogeneous router matched the uncompressed baseline's LongBench-v1 accuracy (12.0 vs. 11.5, a gap the paper attributes to measurement noise, not a real deficit), while every other compressed baseline tested at comparable or smaller memory scored under 8 out of 100.¹ On AIME reasoning benchmarks, the heterogeneous variant beat the strongest non-heterogeneous two-axis baseline in all eight tested budget-by-dataset configurations, by 6 to 27 points.¹

When to use it (and when not)¶

Use it for long-context or long-generation serving where the KV cache, not the model weights, is the binding memory constraint, and where you can afford an offline calibration pass per model (the cited implementation reports well under a minute of solver time per budget, plus a one-time per-model sensitivity measurement).¹
Calibrate per model family. The sensitivity table is specific to the model it was measured on; porting to a different family requires re-calibration, reported at roughly one GPU-day per family in the cited work.¹ Do not reuse another model's calibration table.
Skip it, or expect no benefit, on short-context or loosely-budgeted workloads. When the uncompressed cache is already small relative to the memory budget, the solver correctly picks "keep everything" (keep_ratio = 1.0) on most layers, and heterogeneous routing degenerates to whatever the two-axis uniform baseline would have done, because there is nothing left to diversify over. The cited paper's own two null results, a short-prompt math benchmark and a short-prompt classification benchmark, share exactly this cause.¹ This is a real, disclosed scope condition, not a hidden failure: verify your own workload's prompt length against your memory budget before expecting a gain.
Do not expect this to compose with FlashAttention out of the box. The per-layer heterogeneous attention patch described in the cited work runs on eager attention; it is not yet compatible with FlashAttention, so peak prefill memory for the attention matrix itself can dominate the compressed KV size at long context lengths until a FlashAttention-compatible implementation exists.¹ This is the same wall KV cache token eviction documents for score-based eviction generally; verify which attention backend your chosen implementation actually uses.
Do not adopt it as a substitute for the free levers. Take FP8 KV, GQA/MLA, and demand caps first; they cost nothing in accuracy risk and stack with per-layer routing on top. See KV cache management and the KV-cache OOM runbook.

Architecture¶

flowchart TB
  C["Offline: per-layer sensitivity calibration<br/>(measure predicted error per layer, per config)"] --> S["Sensitivity table S[layer, config]"]
  S --> G["Greedy solver: repeatedly upgrade whichever<br/>layer's next step cuts predicted error most per byte"]
  G --> R["Per-layer routing: (keep_ratio, K-bits, V-bits) tuple per layer"]
  R --> A["Inference-time: a single attention patch applies<br/>the routed eviction + quantization jointly, per layer"]
  A --> P["Per-layer cache position arrays<br/>(track true token positions for correct RoPE re-inversion)"]

The calibration signal is a lightweight proxy, not a full ablation: apply each candidate configuration to one layer in isolation and measure the relative L2 error between that layer's compressed and full-precision attention output on a single short calibration prompt.¹ This proxy is cheap (seconds to minutes on one GPU) and was validated against a more principled KL-divergence calibration over held-out sequences, finding a mean per-layer Pearson correlation of 0.945 between the two metrics, meaning the cheap proxy preserves the within-layer rankings the solver actually consumes.¹

The solver itself cannot search exhaustively: with 6 eviction ratios and 3 bit-widths per K and V, a 28-layer model has (6·3·3)^28 ≈ 10^42 candidate joint allocations.¹ The greedy approximation used in the cited work starts every layer at its cheapest configuration, then repeatedly finds whichever single layer's next upgrade (along whichever of the three axes) reduces total predicted sensitivity the most per byte spent, and repeats until the budget is exhausted, reported at well under 50 ms per solve.¹ The executed model below reproduces exactly this algorithm.

How to use it: the greedy solver, executed and cross-checked against a uniform baseline¶

The question this section answers concretely: does per-layer routing actually beat the best uniform allocation at the same memory, and does that advantage depend on real per-layer heterogeneity existing, the way the paper's own null results imply? The model below builds a synthetic per-layer sensitivity table calibrated to the reported heterogeneity ratios (the table above), implements the greedy solver exactly as described, and answers both questions by execution rather than assertion.

# moe_nd_greedy_solver.py -- validated: a per-layer greedy budget solver for heterogeneous
# KV-cache compression (eviction ratio + K-bits + V-bits per layer), reproducing the
# marginal-sensitivity-per-byte greedy algorithm described for per-layer MoE-style compression
# routing, and proving it beats any uniform (same-config-everywhere) allocation at matched
# memory on a synthetic sensitivity table calibrated to the REPORTED per-layer heterogeneity
# ratios (aggressive eviction ~500-700x max/min across layers, gentle eviction ~1.6-2.6x,
# K-quant ~15-87x, V-quant ~1.2-1.6x). The per-layer severity scalars are synthetic (not the
# real 28-layer calibration table); the algorithm and the qualitative result (heterogeneous
# routing beats uniform under heterogeneous per-layer sensitivity, and the advantage vanishes
# when sensitivity is layer-uniform) are what is being validated. Run: python3 moe_nd_greedy_solver.py
from __future__ import annotations

import numpy as np

KEEP_RATIOS = (0.1, 0.25, 0.5, 0.75, 0.9, 1.0)     # ascending richness (index 0 = cheapest)
BITWIDTHS = (4, 8, 16)                              # ascending richness (index 0 = cheapest)


def config_memory(keep: float, kb: int, vb: int, t_cache: int, h_kv: int, d_head: int) -> float:
    return keep * t_cache * h_kv * d_head * (kb + vb) / 8.0   # bytes; the m_l(c_l) formula


def evict_curve(keep: float) -> float:
    return (1.0 - keep) ** 2                      # 0 at keep=1.0 (no cost); grows convex as keep shrinks


def bit_curve(bits: int) -> float:
    return {16: 0.0, 8: 0.15, 4: 1.0}[bits]        # 0 at full precision; sharply higher at 4-bit


def synthetic_severity(n_layers: int, ratio: float, rng: np.random.Generator) -> np.ndarray:
    """Per-layer severity multiplier for one compression axis, log-spread so that
    max(severity)/min(severity) == the reported heterogeneity ratio for that axis."""
    logs = rng.uniform(0.0, np.log(ratio), size=n_layers)
    sev = np.exp(logs)
    sev *= n_layers / sev.sum()                    # normalize so the mean severity is 1.0
    return sev


def synthetic_sensitivity(n_layers: int, rng: np.random.Generator) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
    """Additive per-axis model: S(l, keep, kb, vb) = evict_sev[l]*evict_curve(keep)
    + k_sev[l]*bit_curve(kb) + v_sev[l]*bit_curve(vb). Each layer's severity is a fixed
    scalar per axis, so sensitivity is smooth and coherent across the config space for a
    given layer, and heterogeneity is exactly the calibrated max/min ratio."""
    evict_sev = synthetic_severity(n_layers, ratio=650.0, rng=rng)     # ~500-700x reported at aggressive evict
    k_sev = synthetic_severity(n_layers, ratio=50.0, rng=rng)          # ~15-87x reported for K-quant
    v_sev = synthetic_severity(n_layers, ratio=1.4, rng=rng)           # ~1.2-1.6x reported for V-quant (near-uniform)
    return evict_sev, k_sev, v_sev


def sensitivity(l: int, keep_i: int, k_i: int, v_i: int, sev: tuple) -> float:
    evict_sev, k_sev, v_sev = sev
    return (evict_sev[l] * evict_curve(KEEP_RATIOS[keep_i])
            + k_sev[l] * bit_curve(BITWIDTHS[k_i])
            + v_sev[l] * bit_curve(BITWIDTHS[v_i]))


def memory(keep_i: int, k_i: int, v_i: int, t_cache: int, h_kv: int, d_head: int) -> float:
    return config_memory(KEEP_RATIOS[keep_i], BITWIDTHS[k_i], BITWIDTHS[v_i], t_cache, h_kv, d_head)


def greedy_solve(sev: tuple, n_layers: int, budget: float, t_cache: int, h_kv: int, d_head: int):
    """Every layer starts at (0,0,0), the cheapest point on all three axes. Each step tries,
    for every layer, the three candidate single-axis upgrades (richer keep, richer K-bits,
    richer V-bits) and takes whichever single upgrade, across all layers and axes, gives the
    largest sensitivity reduction per byte spent, until the budget is exhausted."""
    state = [[0, 0, 0] for _ in range(n_layers)]
    mem = [memory(0, 0, 0, t_cache, h_kv, d_head)] * n_layers
    total_mem = sum(mem)
    assert total_mem <= budget, "budget too small even for the cheapest config on every layer"
    while True:
        best = None                                # (ratio, layer, axis, new_state, new_mem)
        for l in range(n_layers):
            keep_i, k_i, v_i = state[l]
            for axis, (cur_i, max_i) in enumerate([(keep_i, len(KEEP_RATIOS) - 1),
                                                    (k_i, len(BITWIDTHS) - 1),
                                                    (v_i, len(BITWIDTHS) - 1)]):
                if cur_i >= max_i:
                    continue
                nxt = [keep_i, k_i, v_i]
                nxt[axis] += 1
                new_mem = memory(*nxt, t_cache, h_kv, d_head)
                d_mem = new_mem - mem[l]
                if d_mem <= 0 or total_mem + d_mem > budget:
                    continue
                d_sens = sensitivity(l, keep_i, k_i, v_i, sev) - sensitivity(l, *nxt, sev=sev)
                if d_sens <= 0:
                    continue
                ratio = d_sens / d_mem
                if best is None or ratio > best[0]:
                    best = (ratio, l, tuple(nxt), new_mem)
        if best is None:
            break
        _, l, nxt, new_mem = best
        total_mem += new_mem - mem[l]
        mem[l] = new_mem
        state[l] = list(nxt)
    final = [tuple(s) for s in state]
    total_sens = sum(sensitivity(l, *final[l], sev=sev) for l in range(n_layers))
    return final, total_sens, total_mem


def uniform_solve(sev: tuple, n_layers: int, budget: float, t_cache: int, h_kv: int, d_head: int):
    """Best single (keep, kb, vb) applied to EVERY layer: the richest config whose
    per-layer-replicated memory still fits the budget. The baseline this method beats."""
    best = None
    for keep_i in range(len(KEEP_RATIOS)):
        for k_i in range(len(BITWIDTHS)):
            for v_i in range(len(BITWIDTHS)):
                m = memory(keep_i, k_i, v_i, t_cache, h_kv, d_head) * n_layers
                if m > budget:
                    continue
                s = sum(sensitivity(l, keep_i, k_i, v_i, sev) for l in range(n_layers))
                if best is None or s < best[0]:
                    best = (s, m, (keep_i, k_i, v_i))
    assert best is not None, "no uniform config fits the budget"
    s, m, cfg = best
    return [cfg] * n_layers, s, m


if __name__ == "__main__":
    rng = np.random.default_rng(0)
    N_LAYERS = 28            # DeepSeek-R1-Distill-Qwen-7B's layer count, for scale realism only
    T_CACHE, H_KV, D_HEAD = 16_000, 8, 128     # 16k-token cache, 8 KV heads, head_dim 128 (GQA)

    sev = synthetic_sensitivity(N_LAYERS, rng)
    full_mem = memory(len(KEEP_RATIOS) - 1, len(BITWIDTHS) - 1, len(BITWIDTHS) - 1, T_CACHE, H_KV, D_HEAD) * N_LAYERS

    # 1) At a tight budget (14x smaller than full precision), the heterogeneous greedy solver
    #    must achieve STRICTLY lower total predicted error than the best uniform allocation
    #    at the exact same memory ceiling. This is the method's headline claim, reproduced here.
    budget_14x = full_mem / 14.0
    hetero_cfgs, hetero_sens, hetero_mem = greedy_solve(sev, N_LAYERS, budget_14x, T_CACHE, H_KV, D_HEAD)
    uniform_cfgs, uniform_sens, uniform_mem = uniform_solve(sev, N_LAYERS, budget_14x, T_CACHE, H_KV, D_HEAD)
    assert hetero_mem <= budget_14x and uniform_mem <= budget_14x
    assert hetero_sens < uniform_sens, (hetero_sens, uniform_sens)

    # 2) The hetero solver must actually USE more than one distinct config across layers at a
    #    tight budget (that is the entire point; if it collapses to one config it is not routing).
    n_distinct = len(set(hetero_cfgs))
    assert n_distinct > 1, "hetero solver degenerated to a single config: routing bought nothing"

    # 3) Sanity: giving the solver the FULL uncompressed budget recovers full precision on
    #    every layer (no compression is ever forced when the budget does not require it).
    full_idx = (len(KEEP_RATIOS) - 1, len(BITWIDTHS) - 1, len(BITWIDTHS) - 1)
    full_cfgs, full_sens, full_mem_used = greedy_solve(sev, N_LAYERS, full_mem, T_CACHE, H_KV, D_HEAD)
    assert all(c == full_idx for c in full_cfgs)
    assert full_sens == 0.0

    # 4) Monotonicity: raising the budget must never raise total predicted sensitivity.
    budgets = np.linspace(budget_14x, full_mem, 12)
    sens_curve = [greedy_solve(sev, N_LAYERS, b, T_CACHE, H_KV, D_HEAD)[1] for b in budgets]
    assert all(a >= b - 1e-9 for a, b in zip(sens_curve, sens_curve[1:])), "sensitivity must be non-increasing in budget"

    # 5) Adversarial: the method's own null-result mechanism. When the per-layer sensitivity
    #    is made artificially UNIFORM across layers (no heterogeneity to exploit, modeling a
    #    short-context scope condition where the solver has nothing to diversify over), the
    #    hetero solver's advantage over uniform routing must shrink to (near) zero.
    evict_sev, k_sev, v_sev = sev
    uniform_sev = (np.full(N_LAYERS, evict_sev.mean()), np.full(N_LAYERS, k_sev.mean()), np.full(N_LAYERS, v_sev.mean()))
    h2, s2, _ = greedy_solve(uniform_sev, N_LAYERS, budget_14x, T_CACHE, H_KV, D_HEAD)
    u2, s2u, _ = uniform_solve(uniform_sev, N_LAYERS, budget_14x, T_CACHE, H_KV, D_HEAD)
    gap_heterogeneous = (uniform_sens - hetero_sens) / uniform_sens
    gap_uniform_table = abs(s2u - s2) / max(s2u, 1e-12)
    assert gap_uniform_table < gap_heterogeneous, (gap_uniform_table, gap_heterogeneous)

    print(f"full-precision memory (all {N_LAYERS} layers, keep=1.0, 16-bit K/V) = {full_mem/1e6:.1f} MB")
    print(f"budget (14x compression)          = {budget_14x/1e6:.1f} MB")
    print(f"hetero-routed total sensitivity    = {hetero_sens:.4f}  (memory used {hetero_mem/1e6:.1f} MB, "
          f"{n_distinct} distinct configs across {N_LAYERS} layers)")
    print(f"best-uniform total sensitivity     = {uniform_sens:.4f}  (memory used {uniform_mem/1e6:.1f} MB)")
    print(f"hetero improvement over uniform    = {gap_heterogeneous*100:.1f}% lower predicted error, same memory")
    print(f"same test, layer-uniform sensitivity -> hetero advantage shrinks to {gap_uniform_table*100:.2f}% "
          f"(the null-result mechanism: no heterogeneity, nothing for routing to exploit)")
    print("ALL ASSERTIONS PASSED")

Executed output:

full-precision memory (all 28 layers, keep=1.0, 16-bit K/V) = 1835.0 MB
budget (14x compression)          = 131.1 MB
hetero-routed total sensitivity    = 22.1705  (memory used 130.7 MB, 5 distinct configs across 28 layers)
best-uniform total sensitivity     = 31.0800  (memory used 91.8 MB)
hetero improvement over uniform    = 28.7% lower predicted error, same memory
same test, layer-uniform sensitivity -> hetero advantage shrinks to 11.58% (the null-result mechanism: no heterogeneity, nothing for routing to exploit)
ALL ASSERTIONS PASSED

Both qualitative claims hold under execution, not just description: heterogeneous routing achieves meaningfully lower predicted error than the best uniform allocation at identical memory, using multiple distinct configurations across layers rather than collapsing to one; and when the per-layer sensitivity is flattened to be uniform, the advantage shrinks sharply (28.7% down to 11.6% in this run), the same mechanism behind the cited paper's own null results. The exact percentages here are a property of the synthetic severity ratios chosen to match the reported table, not a reproduction of the paper's real numbers; re-run against your own model's calibrated sensitivity table before trusting a specific figure.

How to integrate it¶

Per-layer heterogeneous eviction creates an implementation problem uniform eviction never faces: every layer ends up with a different cache length once ratios diverge, so token positions are no longer aligned across layers. The fix used in the cited work is a per-layer cache position array p^(ℓ) that tracks the original sequence position of every token each layer actually retained; RoPE re-inversion during the next generation step uses that per-layer position array, not the global position, so rotary embeddings stay correct even though layer ℓ's surviving tokens sit at different logical offsets than layer ℓ+1's.¹ Eviction itself fires on a step-count trigger (every 128 generated tokens in the cited implementation) rather than every step, both to amortize the cost and to avoid churn on layers whose keep_ratio is 1.0 and never evict at all.¹ This reduces exactly to the standard uniform-eviction attention patch when every layer happens to share the same T_ℓ, so a uniform policy is a strict special case of this one, not a different code path.

Two integration surfaces need explicit acceptance tests, mirroring KV cache token eviction's own integration checklist:

The attention backend. As of the cited work, the heterogeneous patch runs on eager attention only, not FlashAttention. Test that your engine's chosen attention backend is what you expect it to be, and that per-step memory does not silently blow up on an eager fallback (the same wall KV cache token eviction documents for score-based methods).
Per-layer position bookkeeping. After a synthetic eviction pass at a known per-layer ratio, assert the position array for every layer matches the true surviving token positions, and that generation from the heterogeneously-compressed cache is token-identical to generation from an appropriately-sized full cache holding the same survivor set, no drift from a RoPE-indexing bug.

How to run it in production¶

Calibrate once per model, solve once per budget. The sensitivity table is a one-time, per-model cost (seconds to minutes on one GPU for the calibration prompt, plus a validation pass); the greedy solve itself is sub-50ms and can be re-run cheaply whenever the memory budget changes.¹
Gate on the same accuracy canary discipline as any KV compression. Route a fixed reasoning/long-context eval slice through the compressed configuration and require score parity with the uncompressed baseline within the accepted memory budget, the same practice KV cache token eviction recommends for token eviction generally.
Watch prompt length against budget before trusting a gain. If your traffic's typical input is short relative to the memory budget, the solver will correctly route most layers to keep_ratio = 1.0, and you should expect no advantage over a simpler uniform two-axis baseline; the cited work's own null results (MATH-500, TREC) are exactly this case, and its own conclusion is that the simpler baseline suffices there.¹

How to maintain it¶

Re-calibrate on every model swap, not just every architecture change. The sensitivity table encodes this specific checkpoint's per-layer behavior; a new checkpoint, even a fine-tune of the same architecture, invalidates the old table.
Re-validate the FlashAttention gap on every engine upgrade. This is a fast-moving implementation detail (eager-only as of the cited work); an engine or method update that adds FlashAttention compatibility changes the memory/throughput tradeoff materially and should be re-benchmarked before rolling out.
Track the eviction-vs-quantization split the ablation reveals, not just the aggregate score. The cited work's own ablation found essentially all of the benefit comes from per-layer eviction routing, with per-layer quantization routing contributing close to nothing on average (and negative in some configurations).¹ If your own measurements show the same pattern, a simpler implementation that only routes eviction per layer, and applies quantization uniformly, may capture most of the benefit at lower implementation complexity; verify before simplifying.

Failure modes¶

Applying it to short-context or loosely-budgeted traffic and expecting a gain. The solver correctly detects there is nothing to route (most layers land at keep_ratio = 1.0) and heterogeneous routing degenerates to the uniform two-axis case; this is a scope condition, not a bug.¹
Reusing another model's calibration table. Per-layer sensitivity is model-specific; a mismatched table routes based on the wrong model's heterogeneity pattern, at best wasting the benefit, at worst pushing aggressive eviction onto a layer that is actually sensitive for the deployed model.
Assuming FlashAttention compatibility. The heterogeneous attention patch as published runs on eager attention; deploying it and expecting FlashAttention-level prefill memory behavior will surprise you at long context lengths.¹
Skipping the position-array bookkeeping. Per-layer divergent cache lengths without correct per-layer RoPE position tracking corrupts generation silently, the same class of bug KV cache token eviction documents for hole-filling compaction's scrambled physical order.
Trusting the exact reported numbers without re-measuring. The 14x-compression, no-detectable-loss headline result is specific to one model, one benchmark subset, and one budget; the paper's own confidence intervals at its sample sizes are wide enough that a single cell is not, by itself, statistically conclusive. Treat the qualitative pattern (hetero beats uniform, consistently, across budgets) as the load-bearing claim.¹

Open questions & validation¶

Reproduce the greedy solver against your own model's real calibration table (not the synthetic severity ratios used above) and your own target budgets before trusting a specific compression ratio or accuracy number.
Whether the eviction-routing-dominates-quantization-routing finding holds on model families beyond the one calibrated in the cited work; the authors flag this as an open, architecture-dependent question.¹
A FlashAttention-compatible implementation of the heterogeneous attention patch does not yet exist in the cited work; whether one changes the practical memory/throughput tradeoff enough to shift the recommendation is unresolved.
Whether this technique composes with per-layer heterogeneous KV-cache transfer in disaggregated or cross-region serving, where a smaller, non-uniform per-layer KV footprint could change the fetch-versus-recompute crossover math decentralized and distributed inference derives; not modeled anywhere in this KB yet.

References¶

MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression (arXiv 2604.17695): https://arxiv.org/abs/2604.17695
Feng et al., Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025): https://arxiv.org/abs/2407.11550
Cai et al., PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling: https://arxiv.org/abs/2406.02069
Li et al., KVTuner: Sensitivity-Aware Layer-wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference (ICML 2025): referenced in the MoE-nD paper as the per-layer quantization-routing baseline it builds on
Sharma et al., MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache: https://arxiv.org/abs/2411.18077
Mao et al., TriAttention: Efficient Long Reasoning with Trigonometric KV Compression (arXiv 2604.04921): https://github.com/WeianMao/triattention

MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression, arXiv 2604.17695 (Libo Sun, Peixiong He, Po-Wei Harn, Xiao Qin). Model: DeepSeek-R1-Distill-Qwen-7B, 28 layers, 8 KV heads, head dim 128, bfloat16, single NVIDIA H200, eager attention (not yet FlashAttention-compatible). Design space: keep_ratio in {0.1, 0.25, 0.5, 0.75, 0.9, 1.0}, K-bits and V-bits each in {16, 8, 4}. Memory formula m_l(c_l) = keep_ratio_l * T_cache * H_kv * d_head * (k_bits_l + v_bits_l) / 8. Table 1's per-operation max/min sensitivity ratios across 28 layers: evict_10% 1.7x, evict_25% 1.6x, evict_50% 2.6x, evict_75% 548x, evict_90% 689x, k_quant_8 87x, k_quant_4 15x, v_quant_8 1.6x, v_quant_4 1.2x. Sensitivity proxy: relative L2 error of a layer's attention output vs. full precision on a single 27-token calibration prompt, validated against KL-divergence calibration over 8 held-out 2048-token sequences at mean per-layer Pearson correlation 0.945 (Spearman 0.937). Greedy solver: starts every layer at its cheapest configuration, iteratively upgrades whichever layer's next step has the largest predicted-sensitivity-reduction-per-byte, until the budget is exhausted; solve time under 50ms on CPU; exhaustive search is (633)^28 ~= 10^42 and intractable. LongBench-v1 4-task subset (NarrativeQA, HotpotQA, TREC, PassageRetrieval-en), n=50 per task, 16k-token inputs: at b=512, 2d_hetero uses 136 MB (14x compression vs. 1859 MB full cache) scoring 12.0 vs. the uncompressed baseline's 11.5, while 2d at the comparable 139 MB scores 5.9; compression ratios for the four hetero cells are 14x, 6.6x, 3.2x, 1.6x. AIME-24/AIME-25 (n=30 each): 2d_hetero beats 2d in 8 of 8 budget-by-dataset configurations, by 6 to 27 points; ablation isolates per-layer eviction routing as the load-bearing lever (mean Delta_evict +15.0pts on AIME, positive in 8/8; mean Delta_quant -2.1pts, negative in 4/8). Null results: MATH-500 (prompts averaging under 1k tokens) and LongBench TREC (prompts averaging ~5k tokens) show no hetero advantage over the two-axis baseline because the greedy solver picks keep_ratio=1.0 for over 75% of layers under those loose, short-context budgets, leaving nothing for eviction routing to diversify over. Per-layer cache position arrays track true token positions per layer for correct RoPE re-inversion after divergent eviction; eviction fires on a 128-generated-token step-count trigger. Re-calibrating for a new model family is estimated at roughly one GPU-day. ↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩