Markdown

DeepSeek-V4: compressed sparse attention¶

Scope: the architecture and serving consequences of the DeepSeek-V4 series (V4-Pro at 1.6T parameters with 49B activated, V4-Flash at 284B with 13B activated, both at 1M context). This page covers the two-rate hybrid attention (CSA and HCA) that produces the efficiency claim, why the layer mix rather than either compression rate is the real design knob, the manifold-constrained hyper-connections that replace plain residual connections, the heterogeneous and on-disk KV cache machinery serving requires, and how to read the published efficiency numbers without over-generalising them. The related architecture page is Kimi K3 multi-node serving, which solves the same long-context problem with a different hybrid; the optimiser is covered in Muon.

The numpy block below is executed and asserted in this page (Python 3.11, numpy 2.x). It validates the compression arithmetic, the sparsity-versus-context relationship, and the Sinkhorn-Knopp projection at the paper's own iteration count. Where it derives something the paper does not state (the CSA-to-HCA layer mix), it is labelled as a derivation from the published ratio, not as a reported fact. No model was run.

What it is¶

DeepSeek-V4 keeps the DeepSeekMoE structure and replaces the attention layer with a hybrid of two compressed attention mechanisms operating at different compression rates:

CSA (Compressed Sparse Attention) compresses the KV cache of every m tokens into one entry, then applies DeepSeek Sparse Attention over the compressed entries, where each query attends to only the top k of them. Compression plus selection.
HCA (Heavily Compressed Attention) compresses far more aggressively, consolidating every m' tokens (with m' much greater than m) into a single entry, and does not apply sparse attention on top. Compression only, no selection. It also drops the overlapped compression that CSA uses.

After the first couple of layers, CSA and HCA are interleaved. A small sliding-window attention branch runs alongside.

The published configurations:

	V4-Flash	V4-Pro
Layers	43	61
Hidden dimension	4,096	7,168
First layers	2x pure sliding-window	2x HCA
CSA compression rate `m`	4	4
HCA compression rate `m'`	128	128
DSA top-k	512	1,024
Query heads	64	128
Head dimension	512	512
Query compression dim	1,024	1,536
Output projection groups	8	16
SWA window `n_win`	128	128
Routed / shared experts	256 / 1	384 / 1
Active routed experts	6	6
Expert intermediate dim	2,048	3,072
MTP depth	1	1
mHC expansion `n_hc`	4	4
Sinkhorn-Knopp iterations	20	20
Total / activated params	284B / 13B	1.6T / 49B

Two further changes matter beyond attention. mHC (Manifold-Constrained Hyper-Connections) replaces the plain residual connection between adjacent Transformer blocks, constraining the residual mixing matrix to a specific manifold. And the Muon optimiser is used for most parameters (AdamW for the rest), for faster convergence and greater training stability.

Why it matters¶

The headline is an efficiency claim, and it is specifically a long-context claim. At a 1M-token context, relative to DeepSeek-V3.2:

V4-Pro: 27% of the single-token inference FLOPs (measured in equivalent FP8 FLOPs) and 10% of the KV cache, despite having more activated parameters (49B against 37B).
V4-Flash: 10% of the single-token FLOPs and 7% of the KV cache, with fewer total parameters (284B against 671B) and fewer activated (13B against 37B).

The V4-Flash line is the more striking of the two, and the paper's base-model comparison claims it outperforms V3.2-Base across a wide range of benchmarks despite the smaller budget, with the advantage most visible on world knowledge and long context.

For anyone sizing a long-context deployment, the KV cache figure is usually the binding one. A 90% reduction in KV bytes at 1M context changes which requests fit on a node, how many you can batch, and whether prefix caching is affordable, which is a different and generally larger lever than the FLOPs number.

When this design applies (and when it does not)¶

It applies when your workload actually runs at long context. The whole mechanism is a bet that at large n, most compressed entries are not worth reading.

It does not apply when:

Your contexts are short. The executed block makes this precise: top-k is a constant, so at 4,096 tokens the V4-Pro k = 1024 selects every compressed entry there is, and the sparse attention is not sparse at all. At 32k it selects one eighth. Only at 1M does it reach 0.41%. Quoting the efficiency numbers at short context is meaningless.
You need exact full attention. Compression is lossy by construction; both CSA and HCA replace m real KV entries with one learned mixture.
You are evaluating on aggregate benchmarks only. The gains concentrate where the architecture is designed to help. Judge it on long-context evaluations.

Architecture¶

flowchart TB
  H["hidden states H (n x d)"]
  H --> SWA["SWA branch<br/>window n_win = 128<br/>uncompressed, every layer"]
  subgraph HYB["Interleaved hybrid attention"]
    direction TB
    subgraph CSAB["CSA layer"]
      C1["compress every m=4 tokens -> 1 entry<br/>(overlapped)"]
      C2["lightning indexer:<br/>select top-k compressed entries"]
      C3["attention over k entries only"]
      C1 --> C2 --> C3
    end
    subgraph HCAB["HCA layer"]
      D1["compress every m'=128 tokens -> 1 entry<br/>softmax weights Z + positional bias B"]
      D2["dense attention over all compressed entries"]
      D1 --> D2
    end
  end
  H --> CSAB
  H --> HCAB
  CSAB --> MIX["mHC residual mixing<br/>B constrained to doubly stochastic<br/>(Sinkhorn-Knopp, 20 iters)"]
  HCAB --> MIX
  SWA --> MIX
  MIX --> MOE["DeepSeekMoE FFN<br/>6 of 256/384 routed + 1 shared"]
  MOE --> OUT["next block"]

How the compression arithmetic works, validated¶

import numpy as np

# ===================== 1. mHC: the manifold is the Birkhoff polytope =========
# DeepSeek-V4 constrains each residual mixing matrix B_l to
#   M := { M in R^{n x n} : M 1 = 1, 1^T M = 1^T, M >= 0 },
# i.e. the doubly stochastic matrices. Sinkhorn-Knopp is the projection.
N_HC = 4          # mHC expansion factor
T_MAX = 20        # Sinkhorn-Knopp iterations used by DeepSeek-V4


def sinkhorn_knopp(logits: np.ndarray, iters: int) -> np.ndarray:
    m = np.exp(logits - logits.max())      # non-negativity, stably
    for _ in range(iters):
        m = m / m.sum(axis=1, keepdims=True)   # rows -> 1
        m = m / m.sum(axis=0, keepdims=True)   # cols -> 1
    return m


def ds_error(m: np.ndarray) -> float:
    return float(max(np.abs(m.sum(axis=1) - 1).max(), np.abs(m.sum(axis=0) - 1).max()))


rng = np.random.default_rng(11)
logits = rng.normal(size=(N_HC, N_HC)) * 2.0
B = sinkhorn_knopp(logits, T_MAX)

assert np.all(B >= 0)                                   # non-negativity
assert abs(B.sum() - N_HC) < 1e-6                       # total mass = n, not 1
# t_max=20 satisfies the constraint APPROXIMATELY, not exactly: the residual is
# around 1e-7, which is far below BF16 resolution (~8e-3 relative) and therefore
# invisible in training, but it is not an exact projection.
assert 1e-9 < ds_error(B) < 1e-6, ds_error(B)
assert ds_error(B) < np.finfo(np.float32).eps          # under fp32 epsilon too
# The iteration count is not arbitrary: 2 is nowhere near converged, and running
# 10x longer buys another few orders of magnitude that no one can observe.
assert ds_error(sinkhorn_knopp(logits, 2)) > 1e-4
assert ds_error(sinkhorn_knopp(logits, 200)) < ds_error(B)
# A doubly stochastic mix is norm-non-expanding in the L1 sense: it cannot
# amplify the total magnitude of the residual streams it combines.
x = rng.normal(size=(N_HC, 16))
assert np.abs(B @ x).sum() <= np.abs(x).sum() + 1e-9
# An unconstrained (row-stochastic only) mixer can amplify, which is exactly the
# instability the manifold constraint removes.
row_only = np.exp(logits); row_only /= row_only.sum(axis=1, keepdims=True)
assert abs(row_only.sum(axis=0).max() - 1.0) > 0.1     # columns are NOT balanced

# ===================== 2. what the compression rates actually buy ============
CTX = 1_000_000
CSA_M = 4          # CSA compresses m tokens -> 1 entry
HCA_M = 128        # HCA compresses m' tokens -> 1 entry
PRO = {"layers": 61, "hidden": 7168, "topk": 1024, "q_heads": 128, "head_dim": 512,
       "routed": 384, "active_experts": 6, "shared": 1, "total": 1.6e12, "act": 49e9}
FLASH = {"layers": 43, "hidden": 4096, "topk": 512, "q_heads": 64, "head_dim": 512,
         "routed": 256, "active_experts": 6, "shared": 1, "total": 284e9, "act": 13e9}


def entries_per_layer(ctx: int, m: int) -> int:
    return ctx // m


assert entries_per_layer(CTX, CSA_M) == 250_000
assert entries_per_layer(CTX, HCA_M) == 7_812
# HCA is 32x smaller than CSA per layer, which is why the mix ratio dominates
# the KV-cache total far more than either rate on its own.
assert entries_per_layer(CTX, CSA_M) / entries_per_layer(CTX, HCA_M) > 31


def kv_ratio(csa_layers: int, hca_layers: int, baseline_layers: int,
             entry_dim: int = 512, baseline_dim: int = 576) -> float:
    """KV bytes relative to an MLA baseline that caches baseline_dim per token
    per layer. Compression acts on the sequence dimension only."""
    v4 = (csa_layers * entries_per_layer(CTX, CSA_M)
          + hca_layers * entries_per_layer(CTX, HCA_M)) * entry_dim
    base = baseline_layers * CTX * baseline_dim
    return v4 / base


# The paper reports ~10% of DeepSeek-V3.2's KV cache for V4-Pro at 1M context.
# Sweep the CSA:HCA split to see which mixes are even consistent with that.
consistent = [(c, PRO["layers"] - c) for c in range(PRO["layers"] + 1)
              if 0.08 <= kv_ratio(c, PRO["layers"] - c, 61) <= 0.12]
assert consistent, "no layer mix reproduces the published ratio"
lo, hi = consistent[0][0], consistent[-1][0]
assert 20 <= lo <= hi <= 40, (lo, hi)
# An all-CSA model lands ~2x above the published figure; an all-HCA model is an
# order of magnitude below it. The published number needs a genuine mix, and the
# mix is the design knob, not either compression rate on its own.
assert kv_ratio(61, 0, 61) > 0.2 and kv_ratio(61, 0, 61) / 0.12 > 1.8
assert kv_ratio(0, 61, 61) < 0.01

# ===================== 3. how sparse the sparse attention really is ==========
def dsa_sparsity(ctx: int, m: int, topk: int) -> float:
    return topk / entries_per_layer(ctx, m)


assert abs(dsa_sparsity(CTX, CSA_M, PRO["topk"]) - 0.004096) < 1e-9
assert abs(dsa_sparsity(CTX, CSA_M, FLASH["topk"]) - 0.002048) < 1e-9
# Both attend to well under 1% of available compressed entries at 1M context...
assert dsa_sparsity(CTX, CSA_M, PRO["topk"]) < 0.005
# ...but top-k is a CONSTANT, so "how sparse" is entirely a function of context.
# The same k=1024 selects EVERY compressed entry at 4k tokens (zero sparsity),
# an eighth of them at 32k, and a four-hundredth at 1M.
assert dsa_sparsity(4_096, CSA_M, PRO["topk"]) == 1.0
assert dsa_sparsity(32_768, CSA_M, PRO["topk"]) == 0.125
assert dsa_sparsity(131_072, CSA_M, PRO["topk"]) == 0.03125
# So the efficiency claim is a LONG-CONTEXT claim. Quoting it at short context
# would be meaningless: there is nothing to skip.
assert dsa_sparsity(CTX, CSA_M, PRO["topk"]) * 30 < dsa_sparsity(32_768, CSA_M, PRO["topk"])
# The compressed-entry budget a query reads is fixed, so per-token attention
# cost stops growing with context. That is the actual mechanism.
for ctx in (200_000, 1_000_000, 4_000_000):
    assert min(PRO["topk"], entries_per_layer(ctx, CSA_M)) == PRO["topk"]

# ===================== 4. MoE sparsity and activated-parameter share =========
for cfg, name in ((PRO, "Pro"), (FLASH, "Flash")):
    frac = cfg["act"] / cfg["total"]
    assert frac < 0.05, (name, frac)
assert abs(PRO["act"] / PRO["total"] - 0.030625) < 1e-6
assert abs(FLASH["act"] / FLASH["total"] - 0.045775) < 1e-5
# Flash has FEWER total parameters than V3.2 (671B) and fewer activated (37B).
V32_TOTAL, V32_ACT = 671e9, 37e9
assert FLASH["total"] < V32_TOTAL and FLASH["act"] < V32_ACT
assert PRO["total"] > V32_TOTAL and PRO["act"] > V32_ACT
# Both V4 models activate 6 of their routed experts plus 1 shared expert.
assert PRO["active_experts"] == FLASH["active_experts"] == 6
assert PRO["routed"] / PRO["active_experts"] == 64.0
assert abs(FLASH["routed"] / FLASH["active_experts"] - 42.667) < 1e-3

print("all DeepSeek-V4 assertions passed")
print("  mHC doubly-stochastic error at t_max=20:", f"{ds_error(B):.2e}")
print("  mHC error at 2 iterations:", f"{ds_error(sinkhorn_knopp(logits, 2)):.2e}")
print("  CSA / HCA entries per layer at 1M ctx:",
      entries_per_layer(CTX, CSA_M), "/", entries_per_layer(CTX, HCA_M))
print("  CSA layer counts consistent with a 10% KV ratio:", f"{lo}-{hi} of 61")
print("  DSA coverage at 1M ctx  Pro / Flash:",
      f"{dsa_sparsity(CTX, CSA_M, PRO['topk']):.4%}",
      "/", f"{dsa_sparsity(CTX, CSA_M, FLASH['topk']):.4%}")
print("  DSA coverage at 32k ctx (Pro):", f"{dsa_sparsity(32_768, CSA_M, PRO['topk']):.2%}")

Executed output:

all DeepSeek-V4 assertions passed
  mHC doubly-stochastic error at t_max=20: 6.74e-08
  mHC error at 2 iterations: 8.79e-02
  CSA / HCA entries per layer at 1M ctx: 250000 / 7812
  CSA layer counts consistent with a 10% KV ratio: 21-32 of 61
  DSA coverage at 1M ctx  Pro / Flash: 0.4096% / 0.2048%
  DSA coverage at 32k ctx (Pro): 12.50%

The layer mix is the design knob, not the compression rates¶

At 1M tokens a CSA layer holds 250,000 compressed entries and an HCA layer holds 7,812, a factor of 32 apart. That gap is so large that the KV-cache total is dominated by how many layers are CSA, not by either rate. Sweeping the split shows that a mix of roughly 21 to 32 CSA layers out of 61 is consistent with the published 10% figure, while an all-CSA model would land about twice as high and an all-HCA model an order of magnitude lower. (This range is derived from the published ratio under a stated MLA baseline, not a number the paper reports; treat it as a sanity envelope, not a spec.)

The practical reading: if you are designing something similar, the interesting search space is the interleaving pattern. The two rates are almost free parameters by comparison.

The sparse attention is only sparse because the context is long¶

This is the point most easily misread. top-k is a fixed budget of compressed entries per query, so the fraction of entries examined falls as context grows. At V4-Pro's k = 1024 and m = 4, the sparse attention degenerates to full attention at 4,096 tokens, reads one eighth of entries at 32k, and 0.41% at 1M.

Read positively, that constant budget is the mechanism: per-token attention cost stops growing with context, which the block checks at 200k, 1M and 4M. Read as a caveat, it means the efficiency headline is not a general statement about the model, it is a statement about the regime the model was built for. Benchmark at your context length.

mHC constrains residual mixing to doubly stochastic matrices¶

The paper defines the manifold as matrices with non-negative entries whose rows and columns both sum to one. That is exactly the set of doubly stochastic matrices, the Birkhoff polytope, and Sinkhorn-Knopp is the standard iterative projection onto it. The block implements it at the paper's n_hc = 4 and t_max = 20 and confirms the result is doubly stochastic to about 7e-8, comfortably under fp32 epsilon and vastly under BF16 resolution, so the residual error is invisible in training even though the projection is not exact. Two iterations, by contrast, leaves an error near 9e-2, so the iteration count is doing real work.

Why constrain it at all? The block gives the intuition: a doubly stochastic mixer cannot amplify the total magnitude of the streams it combines, whereas the row-stochastic-only alternative (what you get from a plain softmax over inputs) leaves column sums unbalanced and can amplify. Hyper-connections generalise the residual connection into a learned mixture over multiple streams; without a constraint that mixture is a place for signal to grow layer over layer. The manifold is a stability device.

How to run it in production: the cache is the hard part¶

The hybrid attention buys FLOPs and memory, and pays for it in cache complexity. The serving design has to deal with heterogeneous KV entries: compressed CSA entries, more heavily compressed HCA entries, uncompressed SWA entries in every layer, plus the lightning indexer's own dimensions for sparse selection. These have different sizes and different update rules, and the paper designs a custom layout to manage them together.

The on-disk prefix cache shows the same asymmetry, and the numbers are instructive:

Compressed CSA and HCA entries are simply written to disk. On a prefix hit, entries are read and reused up to the last complete compression block. Tokens in the trailing incomplete block must be recomputed, because uncompressed entries are not stored. With m' = 128 for HCA, that tail can be up to 127 tokens of recompute per hit, which is the price of the compression granularity.
SWA entries are the problem. They are uncompressed and present in every layer, so their volume is roughly 8 times larger than the compressed CSA and HCA entries combined. The paper offers three strategies with different storage-versus-recompute trade-offs. Full SWA caching gives zero recompute but, as the paper notes, "only a small subset of the stored SWA KV cache will be accessed for each hitting request, which leads to an unbalanced write-intensive access pattern" that suits SSDs poorly. Periodic checkpointing of the last n_win tokens trades storage for some recompute.

The generalisable lesson: in a compressed-attention model the cheap-to-store part of the cache is the compressed part, and the uncompressed sliding-window branch you added for local fidelity becomes the dominant storage cost. Budget for it explicitly. The same tension appears in Kimi K3's hybrid, where a fixed-size recurrent state and a growing KV cache have to share one allocator.

Training-side, the systems work follows the same theme: a hybrid ZeRO strategy for Muon, recomputation and fused kernels to make mHC affordable, and a two-stage contextual parallelism to manage compressed attention.

How it interacts with reasoning effort¶

V4-Pro and V4-Flash each support three reasoning-effort modes, trained as separate specialists with "distinct length penalties and context windows during RL", demarcated by <think> and </think> formats, with a specific system-prompt instruction prepended for the Max mode. The report notes that Max, "which employs longer contexts and reduced length penalties in RL, outperforms the High mode on the most challenging tasks". The general mechanism, and the equal-cost analysis that decides which mode you should actually serve, is on reasoning-effort control.

Failure modes¶

Quoting the efficiency numbers outside the long-context regime. At 32k the sparse attention reads an eighth of entries, not a four-hundredth; at 4k it reads all of them. The claim is context-conditional.
Sizing the KV cache from the compressed entries alone. The uncompressed SWA branch is roughly 8x the compressed volume and lives in every layer.
Expecting exact prefix reuse. Compression-block alignment forces recompute of the trailing partial block, up to m' - 1 tokens for HCA.
Full SWA on-disk caching on a general SSD tier. Write-heavy, poorly amortised access pattern; pick the checkpointing trade-off deliberately.
Porting hyper-connections without the manifold constraint. Unconstrained multi-stream residual mixing can amplify layer over layer; the doubly stochastic projection is what prevents that, and 2 Sinkhorn iterations is not a projection.
Assuming more total parameters means more cost. V4-Pro has more activated parameters than V3.2 and still uses 27% of its per-token FLOPs at 1M context. Parameter count is a poor proxy here.
Comparing V4-Flash to V3.2 on parameters instead of results. Flash is smaller on both counts; the interesting comparison is quality per unit of serving cost.

References¶

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence: https://arxiv.org/abs/2606.19348
DeepSeek Sparse Attention, introduced in DeepSeek-V3.2-Exp: https://arxiv.org/abs/2509.24276
MLA and the DeepSeek attention lineage (DeepSeek-V2): https://arxiv.org/abs/2405.04434
DeepSeekMoE (shared plus routed expert structure): https://arxiv.org/abs/2401.06066
Hyper-Connections (the unconstrained predecessor to mHC): https://arxiv.org/abs/2409.19606
Muon optimizer: https://kellerjordan.github.io/posts/muon/
Sinkhorn-Knopp and doubly stochastic scaling: https://projecteuclid.org/euclid.pjm/1102992505