Markdown

Mixture-of-experts: sparse scaling¶

Scope: why sparse MoE decouples total parameters from per-token FLOPs: the routing math, shared vs routed experts, and the memory-to-fit vs compute-to-run tradeoff that governs how you size and serve these models.

What it is¶

A Mixture-of-Experts (MoE) layer replaces the single feed-forward (MLP) block of a transformer with many parallel expert MLPs plus a small gating network. For each token, the gate scores the experts and routes the token to only the top-k of them. The token's activation is computed by those k experts (their outputs combined by the gate weights); the rest stay idle for that token. The attention sublayers remain dense and shared.

The consequence is a deliberate split between two parameter counts:

Total parameters: every expert's weights, summed. Sets the memory you must hold resident.
Active (activated) parameters per token: what the gate actually selects per token. Sets the FLOPs (and therefore latency and cost) of a forward pass.

Because the gate picks a fixed k regardless of how many experts exist, you can grow total parameters (and model capacity) while active parameters, and per-token FLOPs, stay roughly constant. Sparse MoE models contrast with dense LLMs (the classic GPT-series), where every parameter is active for every token.¹

DeepSeek-V3 is the canonical reference. It has 671B total parameters with 37B activated per token: each MoE layer holds 1 shared expert and 256 routed experts, and the gate activates 8 routed experts per token (so 9 experts run per token: the shared one plus the 8 routed).³ The book's Chapter 1 describes the same architecture and rounds total parameters to "~680 billion."² On disagreement, the DeepSeek-V3 technical report's 671B is the precise figure; the book's "~680B" is a rounding, not a contradiction.

Shared vs routed experts¶

Routed experts are the sparse pool. The gate selects a subset per token. This is where capacity scales: 256 routed experts in DeepSeek-V3, only 8 fired per token.
Shared expert(s) run for every token, unconditionally. They capture common knowledge that would otherwise be redundantly learned across many routed experts, letting the routed pool specialize. DeepSeek-V3 uses 1 shared expert.³

flowchart LR
  T["Token activation"] --> G["Gate / router"]
  T --> S["Shared expert<br/>(always active)"]
  G -->|"top-8 of 256"| R1["Routed expert i"]
  G -->|"top-8 of 256"| R2["Routed expert j"]
  R1 --> C["Weighted combine"]
  R2 --> C
  S --> C
  C --> O["Output to next layer"]

Why it matters¶

Two separate scaling walls, decoupled:

Compute to run. Dense scaling ties FLOPs/token to total parameters, so doubling capacity doubles inference cost. Sparse MoE breaks that link: a 100-expert model that fires 2 experts per token costs roughly what a small dense model costs per token, while holding the capacity of a much larger one.⁶ Fewer active parameters per token also means lower request latency than a dense model of equal total size.¹ Google's Switch Transformer (1.6T-parameter MoE, 2021) reached the accuracy of a dense baseline at a fraction of the compute and trained ~7x faster.⁴
Memory to fit. Sparsity does not save memory. Every expert must be resident even though most are idle for any given token, because at scale (across many concurrent tokens and users) effectively all experts are active simultaneously and contend for GPU resources.⁶ DeepSeek-V3's full 671B weights have to live in HBM regardless of the 37B active figure. This is the central tradeoff: sparsity buys you cheap compute, not cheap memory. You still need enough aggregate GPU memory to hold the entire parameter set, which is what forces expert parallelism across many GPUs (see Expert Parallelism for Inference).

This is why MoE is the practical path toward multi-trillion-parameter models. The book frames a dense 100T-parameter model as needing on the order of 1.2 x 10^29 FLOPs to train and ~182 TB just to load weights in 16-bit; sparsity reduces the effective compute, but the memory-to-fit wall remains and must be solved with parallelism.⁵

When it is needed (and when not)¶

Reach for sparse MoE when:

You need capacity beyond what a dense model can afford to run: high parameter count for quality, but a per-token compute and latency budget that a same-size dense model would blow.
You have the aggregate GPU memory and a high-bandwidth fabric (NVLink/NVSwitch intra-node, InfiniBand/RoCE inter-node) to host all experts and absorb the all-to-all routing traffic.⁶

Avoid or reconsider when:

The model already fits comfortably on your GPUs and per-token cost is acceptable; a dense model is simpler (no routing, no all-to-all, no load-balancing machinery).
Your fabric is weak. The all-to-all dispatch/combine at every MoE layer is communication-heavy and becomes the bottleneck if interconnect bandwidth is low.⁶
Memory is your binding constraint and you cannot scale out across enough GPUs; sparsity will not rescue you, because it saves compute, not footprint.

How: implement, integrate, maintain¶

You rarely implement MoE routing yourself; you configure a serving engine that shards experts across GPUs via expert parallelism (EP). The sizing rule is: total parameters dictate how many GPUs you need to fit the model; active parameters and routing dictate how you tune for latency and balance.

vLLM¶

Expert parallelism is enabled with --enable-expert-parallel. The EP group size is computed automatically as EP_SIZE = TP_SIZE x DP_SIZE; without the flag, MoE layers fall back to tensor parallelism over the same TP x DP group, like a dense model. The flag has no effect unless TP_SIZE x DP_SIZE > 1.¹⁰

# Single-node MoE serving with expert parallelism over 8 GPUs.
vllm serve deepseek-ai/DeepSeek-V3 \
  --tensor-parallel-size 1 \
  --data-parallel-size 8 \
  --enable-expert-parallel

To even out hot experts, vLLM exposes the Expert Parallel Load Balancer:

# Add EPLB to redistribute expert-to-rank mappings under skewed routing.
vllm serve deepseek-ai/DeepSeek-V3 \
  --data-parallel-size 8 \
  --enable-expert-parallel \
  --enable-eplb

For large-scale EP, select an all-to-all backend with --all2all-backend (e.g. deepep_low_latency, deepep_high_throughput).¹⁰

SGLang¶

SGLang sets EP size with --ep (commonly --tp and --ep equal, with dp=1, to use all GPUs as one EP group), enables the load balancer with --enable-eplb, and selects MoE communication/compute backends with --moe-a2a-backend and --moe-runner-backend:¹¹

# SGLang DeepSeek serving: 8-way TP, 8-way EP, EPLB on.
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3 \
  --tp 8 --ep 8 \
  --enable-eplb \
  --moe-a2a-backend deepep

Expert offload to CPU¶

The memory-to-fit wall above assumes every expert lives in GPU memory. On a single small card it does not have to. Keep the dense path (attention, router, normalization, shared experts) on the GPU and park the routed experts in system RAM. Every token traverses the dense path, so it belongs on the fast memory; any individual routed expert is touched by only a fraction of tokens.

In llama.cpp the flags are -cmoe / --cpu-moe ("keep all Mixture of Experts (MoE) weights in the CPU") and -ncmoe N / --n-cpu-moe N ("keep the Mixture of Experts (MoE) weights of the first N layers in the CPU"), added in PR #15077; llama-bench gained --n-cpu-moe in PR #15952 so the knob can be swept.¹² Both are conveniences over -ot / --override-tensor: they push a tensor buffer-type override so the expert tensors are allocated in the CPU buffer, which means the CPU backend executes those matmuls. This is compute offload, not weight streaming, at the small batch sizes decode runs at: the expert weights stay resident in CPU memory and only the hidden-state vectors cross PCIe. That distinction is what makes the decode arithmetic below viable, and it does not hold unconditionally: llama.cpp's own scheduler has an op_offload escape hatch, on by default, that lets a CUDA device claim a CPU-resident op once its batch dimension clears a threshold (GGML_OP_OFFLOAD_MIN_BATCH, default 32). A typical prefill batch clears it, so at prefill the same CPU-placed expert matmul can run on the GPU while reading the weight across PCIe on the fly, converging toward the same behavior as vLLM's UVA path, which streams weights to the GPU over PCIe on demand at every batch size, not just above a threshold.

The physics is bytes moved per token. Decode is memory-bandwidth bound, so a model's decode ceiling is its per-token weight traffic divided by the bandwidth of whichever memory holds those weights. A sparse MoE moves only its active parameters; a dense model of the same footprint moves all of them. The block below derives the ratio and the resulting ceilings (a model with stated assumptions, not a benchmark; no vendor throughput numbers are used):

# Runnable on system python3. Why expert offload works: decode is bandwidth-bound, so the
# first-order term is BYTES MOVED PER TOKEN. This roofline omits CPU compute limits,
# dequantization, KV traffic, activations, and launch overhead. Model, not a benchmark.
GB = 1e9
HBM = 900 * GB    # a small workstation card's memory bandwidth, bytes/s
DDR = 80 * GB     # dual-channel DDR5 system RAM, bytes/s
BPP = 0.55        # bytes/param at ~4.4 bit/param (a Q4_K_M-class quant)


def tok_per_s(gpu_params: float, cpu_params: float) -> float:
    """Decode ceiling: weights streamed from HBM and from system RAM, in series."""
    t = (gpu_params * BPP) / HBM + (cpu_params * BPP) / DDR
    return 1.0 / t


# A 35B-total / 3B-active MoE (A3B class) vs a dense 34B. Same disk footprint, same card.
moe_total, moe_active, moe_dense_part = 35e9, 3e9, 1.5e9  # attention + router + shared
dense_total = 34e9

all_on_gpu = tok_per_s(moe_active, 0.0)                       # if all 35B fit in VRAM
experts_on_cpu = tok_per_s(moe_dense_part, moe_active - moe_dense_part)
dense_all_gpu = tok_per_s(dense_total, 0.0)                   # 34B x 0.55 = 18.7 GB of VRAM
dense_half_cpu = tok_per_s(dense_total / 2, dense_total / 2)  # dense model, half offloaded

print("bytes moved per token (%.2f bytes/param):" % BPP)
print("  MoE 35B-A3B, active 3B  : %5.2f GB" % (moe_active * BPP / GB))
print("  dense 34B               : %5.2f GB   (%.1fx more)" % (dense_total * BPP / GB, dense_total / moe_active))
print()
print("decode ceiling (weight streaming only, no compute, no KV):")
print("  MoE, everything in VRAM         : %6.1f tok/s" % all_on_gpu)
print("  MoE, routed experts in system RAM: %6.1f tok/s" % experts_on_cpu)
print("  dense 34B, everything in VRAM    : %6.1f tok/s" % dense_all_gpu)
print("  dense 34B, half in system RAM    : %6.1f tok/s  <- the collapse" % dense_half_cpu)
print()
print("The MoE with its experts parked in system RAM (%.1f tok/s) still out-streams the dense" % experts_on_cpu)
print("34B held ENTIRELY in VRAM (%.1f tok/s) in this bandwidth-only model." % dense_all_gpu)
assert experts_on_cpu > dense_all_gpu > dense_half_cpu
assert all_on_gpu > experts_on_cpu, "offload is not free: it costs the MoE most of its ceiling"

Executed output:

bytes moved per token (0.55 bytes/param):
  MoE 35B-A3B, active 3B  :  1.65 GB
  dense 34B               : 18.70 GB   (11.3x more)

decode ceiling (weight streaming only, no compute, no KV):
  MoE, everything in VRAM         :  545.5 tok/s
  MoE, routed experts in system RAM:   89.1 tok/s
  dense 34B, everything in VRAM    :   48.1 tok/s
  dense 34B, half in system RAM    :    7.9 tok/s  <- the collapse

The MoE with its experts parked in system RAM (89.1 tok/s) still out-streams the dense
34B held ENTIRELY in VRAM (48.1 tok/s) in this bandwidth-only model.

Under the stated bandwidth-only assumptions, the 11.3x reduction in active weight bytes offsets the slower memory tier: the modeled ceiling falls from 545 to 89 tok/s with expert CPU placement but remains above the dense comparison. This is not an interactivity claim. CPU matrix throughput, dequantization, KV traffic, and runtime overhead can lower the measured rate substantially, so benchmark both prefill and decode.

Operational notes:

-ncmoe N is a scan knob, not a setting. Its stated goal is "to avoid having to write complex regular expressions when trying to optimize the number of MoE layers to keep in the CPU".¹² Sweep N with llama-bench on your own card. Every expert layer kept on the GPU also consumes VRAM that the KV cache wants, so the optimum is workload-dependent and moves with context length.
Prefill and decode want different splits. Prompt processing is compute-bound and batched; decode is bandwidth-bound. A split tuned on one will not be optimal for the other.
vLLM exposes a different mechanism. vllm/config/offload.py defines UVAOffloadConfig(cpu_offload_gb, cpu_offload_params), which matches parameter name segments (the docstring's own example: "mlp.experts.w2_weight" is matched by "experts" or "experts.w2_weight"), plus PrefetchOffloadConfig for group-based async host-to-device prefetch, selected by OffloadBackend = auto | uva | prefetch. UVA is zero-copy streaming over PCIe, so unlike llama.cpp's buffer-type override it moves weights, not just activations, and PCIe bandwidth becomes the ceiling.
Heterogeneous placement is the research frontier. kvcache-ai/ktransformers builds a CPU/GPU hybrid MoE runtime with "heterogeneous expert placement (hot experts on GPU, cold experts on CPU)". The academic anchors are MoE-Infinity (sparsity-aware expert caching, arXiv:2401.14361), Fiddler (CPU-GPU orchestration, arXiv:2402.07033), and Pre-gated MoE (predicting the next layer's experts so they can be prefetched, arXiv:2308.12066).
The kernel still matters. Which expert-compute kernel runs on whichever device holds the weights is a separate axis, covered in MoE expert backends and grouped-GEMM kernels.

Routing, balance, and maintenance¶

Gating width. Top-2 gating is the common production default (better quality and more even load than top-1, at the cost of doubled expert outputs and communication). Top-1 is faster but risks lower quality and uneven load.⁷
Capacity factor. Serving systems cap tokens per expert per batch, typically at 1.2–1.5x average load; overflow tokens spill to a second-choice expert or a second pass. This bounds the straggler effect, where one overloaded ("hot") expert stalls the layer because all experts must finish before the layer proceeds.⁸
Hot-expert replication. Full-featured servers can replicate a hot expert onto multiple GPUs to split its load, at the cost of extra GPU memory.⁸ This is the inference-side complement to DeepSeek's training-time load balancing.
Training-time balance. DeepSeek-V3 uses an auxiliary-loss-free load-balancing strategy (a per-expert bias added to gate affinity scores, nudged down for overloaded and up for underloaded experts) to keep routing even without the quality hit of a heavy auxiliary loss.³ Google's GLaM established the earlier approach of load-balancing losses plus gating noise.⁷
Validate the win. Profile interconnect (NVLink/NIC) traffic and Tensor Core utilization before and after every change; an overloaded all-to-all or a silent imbalance erases the sparsity benefit.⁹ These are reference configurations grounded in vendor docs and the book; none of the throughput figures here were measured on local hardware.

For how experts are physically sharded and the all-to-all communication cost, see Expert Parallelism for Inference and MoE Routing & Load Balancing.

References¶

Chris Fregly, AI Systems Performance Engineering (O'Reilly). Ch. 1 (sparse MoE decouples FLOPs/token from total parameters; DeepSeek-V3 ~680B/37B active, 1 shared + 8 of 256 routed; Switch Transformer 1.6T, ~7x faster; 100T-parameter memory/compute wall). Ch. 15 (expert parallelism, all-to-all, capacity factor 1.2–1.5x, top-1 vs top-2 gating, hot-expert replication, hybrid parallelism, GLaM load-balancing).
DeepSeek-V3 Technical Report, arXiv:2412.19437 (671B total / 37B active, 1 shared + 256 routed experts, top-8 routing, auxiliary-loss-free load balancing) — https://arxiv.org/abs/2412.19437
deepseek-ai/DeepSeek-V3, Hugging Face model card — https://huggingface.co/deepseek-ai/DeepSeek-V3
vLLM, Expert Parallel Deployment (--enable-expert-parallel, EP_SIZE = TP_SIZE x DP_SIZE, --enable-eplb, --all2all-backend) — https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/
SGLang, Expert Parallelism (--ep, --enable-eplb, --moe-a2a-backend, --moe-runner-backend) — https://github.com/sgl-project/sglang/blob/main/docs/advanced_features/expert_parallelism.md
llama.cpp, --cpu-moe / --n-cpu-moe (expert offload to system RAM via tensor buffer-type overrides) — https://github.com/ggml-org/llama.cpp/pull/15077 and llama-bench support — https://github.com/ggml-org/llama.cpp/pull/15952
vLLM offload configuration (UVAOffloadConfig, PrefetchOffloadConfig, OffloadBackend) — https://github.com/vllm-project/vllm/blob/main/vllm/config/offload.py
kvcache-ai/ktransformers (CPU/GPU heterogeneous MoE inference; hot experts on GPU, cold experts on CPU) — https://github.com/kvcache-ai/ktransformers
MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache, arXiv:2401.14361 — https://arxiv.org/abs/2401.14361
Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models, arXiv:2402.07033 — https://arxiv.org/abs/2402.07033
Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference, arXiv:2308.12066 — https://arxiv.org/abs/2308.12066

Fregly, Ch. 1: "Sparse models like MoEs activate only parts of the model for each input token... the FLOPS per token stays roughly constant even as total parameters grow," and "since these models use fewer active parameters during inference, request-response latencies are typically much lower." ↩↩
Fregly, Ch. 1: "~680 billion total parameters with about 37 billion active per input token... an MoE with 1 shared expert and 8 selected experts out of 256 per token, which yields about 9 active experts." ↩
DeepSeek-V3 Technical Report, arXiv:2412.19437: "671B total parameters with 37B activated for each token"; "1 shared expert and 256 routed experts"; "8 experts will be activated for each token"; "auxiliary-loss-free load balancing strategy" introducing a per-expert bias term on affinity scores. ↩↩↩
Fregly, Ch. 1: Google's Switch Transformer "1.6-trillion-parameter MoE model achieved the same accuracy as a dense model with only a fraction of the computation... trained 7x faster." ↩
Fregly, Ch. 1: a dense 100T-parameter model "would require on the order of 1.2 x 10^29 floating point operations" and "approximately 182 TB of GPU memory... if each parameter is stored in 16-bit." ↩
Fregly, Ch. 15: "expert parallelism allows the total model capacity to scale almost linearly with the number of GPUs... each GPU handles only a fraction of the tokens"; "at scale... all of the experts will likely be active simultaneously... contending for all GPU resources concurrently." ↩↩↩↩
Fregly, Ch. 15: "Modern MoE inference often uses top-2 gating"; top-1 "is faster, it can lead to a lower model quality and uneven load"; "Google's GLaM introduced load-balancing losses (and gating noise)." ↩↩
Fregly, Ch. 15: "capacity factor parameter, typically set at 1.2–1.5x the average token load"; the straggler effect where "an overloaded expert... will stall the inference pipeline"; "replicate hot experts onto multiple GPUs to split the load... at a cost of additional GPU memory." ↩↩
Fregly, Ch. 15: "always verify interconnect traffic, Tensor Core utilization, and other performance-enhancing mechanisms before and after making each change." ↩
vLLM Expert Parallel Deployment docs: "Enable EP by setting the --enable-expert-parallel flag"; "EP_SIZE = TP_SIZE x DP_SIZE"; --enable-eplb; --all2all-backend options including deepep_low_latency, deepep_high_throughput. ↩↩
SGLang Expert Parallelism docs: example launch uses --tp 8 --ep 8; --enable-eplb activates the EPLB; --moe-a2a-backend and --moe-runner-backend select communication and compute backends. ↩
llama.cpp common/arg.cpp on master, read 2026-07-13: {"-cmoe", "--cpu-moe"} "keep all Mixture of Experts (MoE) weights in the CPU" and {"-ncmoe", "--n-cpu-moe"}, "N" "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU"; both push entries into params.tensor_buft_overrides bound to ggml_backend_cpu_buffer_type(). PR #15077: "These options work by adding the necessary tensor overrides"; "The goal is to avoid having to write complex regular expressions when trying to optimize the number of MoE layers to keep in the CPU." ↩↩