Markdown

Cookbook: serving Kimi K3 across four nodes¶

Scope: standing up Kimi K3, a 2.78T-parameter hybrid-attention MoE with 104.2B activated parameters and a 1M-token context, on 32 GPUs. This page covers the architecture facts that constrain deployment, the parallelism arithmetic that leaves exactly one viable shape, the per-GPU memory budget, the kernel-selection consequences of MXFP4 weights on Hopper, the two-cache prefix-caching problem the hybrid architecture creates, and the launch and validation surface. The model's post-training story lives in reasoning-effort control and multi-teacher on-policy distillation; the general serving background is inference parallelism strategies and expert parallelism.

Provenance and honesty note. The architecture constants, cache design, and kernel notes are from the Kimi K3 technical report and are cited as such. The concrete launch flags, per-GPU memory split, and throughput figures come from a published third-party multi-node deployment walkthrough on 32x H100-80GB; they are reproduced here as a reference template that this knowledge base has not executed. The Python block below is executed and asserted: it validates the parallelism legality, the memory arithmetic, and the prefix-cache boundary logic, which is exactly the part you can check without renting the cluster. Do not treat the throughput numbers as measured by us.

What it is¶

Kimi K3 is an open-weights native-multimodal MoE. The numbers that drive every deployment decision, from Table 1 of the technical report:

Property	Value
Layers	93
Total parameters	2.78T
Activated parameters	104.2B
Hidden dimension	7,168
Attention heads	96
Attention composition	69 KDA + 24 Gated MLA
Routed experts / active per token	896 / 16
Shared experts	2
MoE hidden dimension per expert	3,072
Latent MoE dimension	3,584 (0.5x)
Vocabulary	160K
Training context length	1M
Activation	SiTU-GLU
MTP layers	1
Vision tower	MoonViT-V2, 401M, 27 layers, patch 14

Three of those rows are the ones that will hurt you if you skim past them.

Hybrid attention. Each block is three Kimi Delta Attention (KDA) layers followed by one Gated MLA layer, a 3:1 ratio, with one additional Gated MLA at the end of the backbone so the final layer is always global attention. That is 23 blocks of 4 layers plus 1, which is exactly 93. The consequence is that the engine maintains two fundamentally different caches: a paged MLA KV cache that grows with sequence length, and a fixed-size KDA recurrent state with a single copy per request.
MLA layers use NoPE. No positional encoding is applied to MLA queries and keys; the interleaved KDA layers supply position sensitivity. This is why K3 extends to 1M-token contexts "without any positional-encoding modification, such as RoPE rescaling or interpolation". There is no RoPE base to retune, and if your serving config has a YaRN or rope-scaling knob, it does not apply here.
MXFP4 weights are native, not a post-hoc quantization. The MoE expert weights are quantized to MXFP4 with MXFP8 activations under quantization-aware training applied "throughout the entire post-training stage, covering both SFT and RL". Non-expert components (attention projections, latent MoE projections, shared experts, MoE routers) stay in higher precision. You are not choosing a quantization; you are serving the trained numerics.

Why this shape and not another¶

You cannot shrink the weights. At roughly 1.45 TiB of loaded weights the model does not fit in fewer than 32 GPUs of 80 GB, and the checkpoint is already 4-bit where 4-bit is possible. There is no "just use a smaller quant" escape.
The parallelism is decided for you. As the executed block shows, only six tensor-parallel degrees are even legal, and only one of them fits.
Expert parallelism is what makes the FLOPs tractable. 16 active of 896 routed experts is about 1.8% density; activated parameters are 3.75% of total. Without EP you would be paying to hold experts that never fire on the local rank.
Long-context serving is a cache problem, not a compute problem. At 1M tokens, the interesting engineering in the report is entirely about jointly managing two caches and getting prefix reuse to work across them.

When to use this recipe (and when not)¶

Use it when you need the full model: 1M-context agentic work, native vision, or maximum-effort reasoning where the frontier gap justifies 32 GPUs.

Do not use it when:

Your traffic is low-concurrency and latency-sensitive. The published walkthrough reports about 5.8 tokens/second single-stream and 14.0 aggregate tokens/second at four-way concurrency. That is a batch-oriented deployment. A single interactive user will find it slow.
You have fewer than 32 GPUs of 80 GB, or no fast interconnect. TP=32 spans four nodes; the all-reduce path is on the critical path of every token.
A smaller model clears your bar. The whole point of measuring at your own effort levels (reasoning-effort control) is to find out whether you need this.
You need guaranteed throughput per dollar today. Reported concurrency tops out around 49 requests in that walkthrough's configuration, and reasoning tokens ran about 81% of completion tokens even at low effort, so a large fraction of what you decode is not the answer.

Architecture¶

flowchart TB
  subgraph NODES["4 nodes x 8 GPUs = 32 ranks"]
    N0["node 0 (rank 0)<br/>dist-init address"]
    N1["node 1"]
    N2["node 2"]
    N3["node 3"]
  end
  subgraph RANK["Per rank (TP=32, EP=32)"]
    W["weights ~45.4 GiB<br/>MXFP4 experts + HP non-expert"]
    E["28 of 896 routed experts"]
    KV["MLA KV cache (paged, grows)"]
    ST["KDA recurrent state (fixed, 1/request)"]
  end
  subgraph CACHE["Unified paged pool"]
    POOL["one page size, one allocator<br/>one eviction + refcount path"]
    HASH["fine hash blocks (512 tok)<br/>for prefix matching"]
    CKPT["sparse KDA checkpoints<br/>at hash endpoints"]
  end
  N0 --- N1 --- N2 --- N3
  RANK --> POOL
  KV --> HASH
  ST --> CKPT
  HASH --> HIT["hit boundary = longest position<br/>satisfying BOTH stages"]
  CKPT --> HIT

How to size it: the arithmetic, executed¶

# ---- Table 1 of the Kimi K3 technical report --------------------------------
LAYERS = 93
HIDDEN = 7168
ATTN_HEADS = 96
ROUTED_EXPERTS = 896
EXPERTS_ACTIVE = 16
KDA_LAYERS, MLA_LAYERS = 69, 24
TOTAL_PARAMS = 2.78e12
ACTIVE_PARAMS = 104.2e9

# The layer composition is self-consistent with the stated 3 KDA : 1 MLA block
# pattern plus one extra global-attention layer at the end of the backbone.
assert KDA_LAYERS + MLA_LAYERS == LAYERS
blocks, rem = divmod(KDA_LAYERS, 3)
assert rem == 0 and blocks == 23
assert MLA_LAYERS == blocks + 1          # one Gated MLA per block, plus the final one
assert blocks * 4 + 1 == LAYERS

# Sparsity: 16 of 896 routed experts, so the routed FFN is ~1.8% dense.
assert abs(EXPERTS_ACTIVE / ROUTED_EXPERTS - 0.017857) < 1e-6
assert abs(ACTIVE_PARAMS / TOTAL_PARAMS - 0.0375) < 1e-3


# ---- 1. Which tensor-parallel degrees are even legal? ------------------------
def divisors(n: int) -> set[int]:
    out = set()
    for d in range(1, int(n ** 0.5) + 1):
        if n % d == 0:
            out.add(d)
            out.add(n // d)
    return out


def legal_tp(heads: int, hidden: int) -> list[int]:
    """TP must split attention heads AND the hidden dimension evenly."""
    return sorted(divisors(heads) & divisors(hidden))


legal = legal_tp(ATTN_HEADS, HIDDEN)
assert legal == [1, 2, 4, 8, 16, 32], legal
# The intuitive candidates fail for a concrete reason each:
assert ATTN_HEADS % 3 == 0 and HIDDEN % 3 != 0      # 3, 6, 12, 24, 48 killed by hidden
assert HIDDEN % 64 == 0 and ATTN_HEADS % 64 != 0    # 64 killed by head count
assert 96 not in legal and HIDDEN % 96 != 0         # TP=96 is not legal either

# Expert parallelism has a different constraint: it must divide the expert count.
assert ROUTED_EXPERTS % 32 == 0 and ROUTED_EXPERTS // 32 == 28
assert ROUTED_EXPERTS % 48 != 0                     # EP=48 would be illegal


# ---- 2. Does it fit? --------------------------------------------------------
GIB = 1024 ** 3
HBM_TOTAL_GIB = 79.19          # usable HBM on an 80GB H100 after ECC/context
WEIGHTS_TOTAL_GIB = 1453.7     # MXFP4 experts + higher-precision non-expert modules


def per_gpu_weights(tp: int) -> float:
    return WEIGHTS_TOTAL_GIB / tp


def fits(tp: int, hbm: float = HBM_TOTAL_GIB, runtime_reserve: float = 8.0) -> bool:
    return per_gpu_weights(tp) + runtime_reserve <= hbm


feasible = [tp for tp in legal if fits(tp)]
assert feasible == [32], feasible
# TP=16 misses by a wide margin, not a tunable one.
assert per_gpu_weights(16) > HBM_TOTAL_GIB
assert abs(per_gpu_weights(16) - 90.86) < 0.01
assert abs(per_gpu_weights(32) - 45.43) < 0.01

# Reported per-GPU budget at TP=32/EP=32 on 32x H100-80GB.
budget = {"weights": 45.43, "mla_kv_cache": 3.62,
          "kda_recurrent_state": 3.40, "free_headroom": 10.34}
used = sum(budget.values())
assert 62.0 < used < 63.0, used
assert used < HBM_TOTAL_GIB
# Whatever the engine does not itself account for is framework and activation
# overhead; it must be positive or the budget is lying.
unaccounted = HBM_TOTAL_GIB - used
assert unaccounted > 0
assert abs(unaccounted - 16.4) < 0.1, unaccounted


# ---- 3. Why MXFP4 weights still run through a 16-bit multiply ---------------
def dequantized_bytes(elems: int, bits: int, block: int = 32,
                      scale_bytes: int = 1) -> float:
    """Packed MXFP4: `bits` per element plus one shared scale per `block`."""
    return elems * bits / 8 + (elems / block) * scale_bytes


packed = dequantized_bytes(TOTAL_PARAMS, 4)
as_bf16 = TOTAL_PARAMS * 2
assert as_bf16 / packed > 3.5          # dequantizing to BF16 in memory is a >3.5x blowup
assert packed / GIB < WEIGHTS_TOTAL_GIB  # experts alone are less than the full checkpoint
assert dequantized_bytes(TOTAL_PARAMS, 4) < as_bf16


# ---- 4. Prefix-cache hit boundaries (report Fig. 12) ------------------------
PHYSICAL_BLOCK = 6144
HASH_BLOCK = 512
assert PHYSICAL_BLOCK % HASH_BLOCK == 0
assert PHYSICAL_BLOCK // HASH_BLOCK == 12


def hit_boundary(matched_tokens: int, kda_checkpoints: set[int],
                 hash_block: int = HASH_BLOCK) -> int:
    """Longest boundary that is hash-aligned, within the match, and has a
    persisted KDA checkpoint. Returns 0 when nothing is reusable."""
    cand = (matched_tokens // hash_block) * hash_block
    while cand > 0 and cand not in kda_checkpoints:
        cand -= hash_block
    return cand


ckpts = {1024, 2560}
assert hit_boundary(2800, ckpts) == 2560          # the report's worked example
assert 2560 == 5 * HASH_BLOCK
assert 2560 % PHYSICAL_BLOCK != 0                  # deep inside a physical block
# Coarse-only caching (the pre-optimisation behaviour) would have reused nothing.
assert hit_boundary(2800, ckpts, hash_block=PHYSICAL_BLOCK) == 0
# A match with no persisted KDA checkpoint is also worth nothing, however long.
assert hit_boundary(6000, {5120}) == 5120
assert hit_boundary(6000, set()) == 0
saved = hit_boundary(2800, ckpts) / 2800
assert 0.9 < saved < 0.92


# ---- 5. Decode arithmetic vs. the observed single-stream rate ---------------
def bytes_moved_per_token(active_params: float, bits: int = 4,
                          block: int = 32, scale_bytes: int = 1) -> float:
    return dequantized_bytes(active_params, bits, block, scale_bytes)


HBM_BW_H100 = 3.35e12          # bytes/s, H100 SXM HBM3
per_tok = bytes_moved_per_token(ACTIVE_PARAMS)
roofline_tok_s = (HBM_BW_H100 * 32) / per_tok
# A pure weight-bandwidth roofline sits far above the ~5.8 tok/s a single stream
# actually gets, so single-stream decode on this shape is latency-bound, not
# bandwidth-bound. The gap is expected and it is large.
assert roofline_tok_s > 1000
assert roofline_tok_s / 5.8 > 100

print("all Kimi K3 sizing assertions passed")
print("  legal TP degrees:", legal)
print("  feasible TP on 80GB HBM:", feasible)
print("  weights/GPU at TP=16 / TP=32 (GiB):",
      round(per_gpu_weights(16), 2), "/", round(per_gpu_weights(32), 2))
print("  experts per rank at EP=32:", ROUTED_EXPERTS // 32)
print("  prefix hit at:", hit_boundary(2800, ckpts), "of 2800 matched tokens")
print("  single-GPU-equivalent bytes/token (MXFP4):", round(per_tok / 1e9, 2), "GB")

Executed output:

all Kimi K3 sizing assertions passed
  legal TP degrees: [1, 2, 4, 8, 16, 32]
  feasible TP on 80GB HBM: [32]
  weights/GPU at TP=16 / TP=32 (GiB): 90.86 / 45.43
  experts per rank at EP=32: 28
  prefix hit at: 2560 of 2800 matched tokens
  single-GPU-equivalent bytes/token (MXFP4): 55.36 GB

TP=32 is not the largest shape that works, it is the only one¶

Tensor parallelism has to split the 96 attention heads and the 7,168 hidden dimension evenly. The common divisors of those two numbers are exactly {1, 2, 4, 8, 16, 32}. Every candidate a practitioner would reach for first dies for a specific reason: 3, 6, 12, 24 and 48 divide 96 but not 7,168 (which is 2^10 * 7, so it has no factor of 3); 64 divides 7,168 but not 96. Then the memory filter removes everything but 32, because at TP=16 each GPU would need 90.86 GiB of weights alone against 79.19 GiB of usable HBM. That is not a tuning gap you close with a lower mem-fraction-static.

Expert parallelism is a separate constraint on a separate number: EP must divide 896 routed experts. EP=32 gives 28 experts per rank cleanly. Do not assume a TP degree is automatically a legal EP degree.

The memory budget, and the 16.4 GiB you should expect to lose¶

The reported per-GPU split accounts for 62.79 GiB of the 79.19 GiB usable: 45.43 weights, 3.62 MLA KV cache, 3.40 KDA recurrent state pools, 10.34 free headroom. The remaining 16.4 GiB is framework overhead, activations, communication buffers and the allocator's own fragmentation. The block asserts it is positive, which is the honest way to read a published memory table: if the itemised lines add up to more than the card holds, the table is wrong.

The KDA recurrent state pool is the line most people have never budgeted for. It does not grow with sequence length (that is the point of a linear-attention state), but it is per request, so it scales with concurrency rather than context. Two different scaling laws in the same pool is the whole reason the report packs both cache types into a unified paged allocator with a single page size, so allocation, reference counting, and eviction have one implementation instead of two.

Why Hopper routes MXFP4 experts through a W4A16 kernel¶

Blackwell has native MXFP4 tensor-core support. Hopper does not. The naive fallback is to dequantize the 4-bit expert weights to BF16 in memory and run a normal GEMM, and block 3 shows what that costs: BF16 is more than 3.5x the packed footprint, which would take the checkpoint far past what 32 GPUs can hold. The working path is a mixed-precision W4A16 kernel (Marlin in the referenced walkthrough) that multiplies 16-bit activations against 4-bit weights, dequantizing each block inside the multiply so the expansion lives in registers and never lands in HBM.

Practical consequence: if your MoE runner backend silently falls back to a dequantize-then-GEMM path, you will not get a slow deployment, you will get an out-of-memory failure at load time. Check which MoE kernel the engine actually selected in the startup log rather than assuming the flag took effect.

Prefix caching has to satisfy two caches at once¶

Block-hash prefix caching normally reuses KV at the granularity of one physical block. K3 breaks that assumption, because a prefix hit is reusable only if both the MLA KV entries and the KDA recurrent state can be restored at the same boundary. KDA keeps one large state per sequence rather than per-token entries, so snapshots are affordable only at sparse boundaries, which would force the shared block size to 1024-6144 tokens. At that granularity the report is blunt: "caching is nearly useless: requests shorter than one block can never be reused, and chunked prefill exports no cacheable prefix until it crosses a full block boundary."

The fix is to decouple the granularities. Prefix hashing runs on fine 512-token hash blocks inside MLA pages while the physical block stays the coarse allocation unit, and KDA checkpoints are saved only at a sparse subset of those hash endpoints. Block 4 implements the resulting lookup: the hit is the longest boundary that is hash-aligned, inside the match, and has a persisted KDA checkpoint. The report's worked example hits at 2560 = 5 x 512, deep inside a 6144-token physical block, saving about 91% of the prefill. Under coarse-only matching the same request would have reused nothing at all, which the block also asserts.

Two operational corollaries fall out of this design:

Checkpoints are retained preferentially at conversation-turn boundaries, and intermediate ones are recycled. Multi-turn agentic traffic is therefore the traffic this cache is built for; single-shot long prompts benefit much less.
Cached checkpoints are read-only snapshots. A hit copies the state into the request's private running state before the next forward pass, and new checkpoints go to fresh slots, so a checkpoint visible to other requests is never mutated in place. If you are implementing something similar, that copy-on-hit is not optional.

How to launch it¶

Reference template, not executed here. Substitute your own addresses and interfaces, and pin the image digest.

The shape is a standard multi-node engine launch: identical command on all four nodes, differing only in node-rank and the node's own IP.

# On each of the 4 nodes, with NODE_RANK in 0..3 and HEAD_IP = node 0's private IP.
docker run -d --name k3 --gpus all --network host --ipc=host \
  --shm-size 32g --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /ephemeral/hf:/root/.cache/huggingface \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e SGLANG_HOST_IP="${THIS_NODE_PRIVATE_IP}" \
  -e NCCL_SOCKET_IFNAME="${PRIVATE_IFACE}" \
  -e GLOO_SOCKET_IFNAME="${PRIVATE_IFACE}" \
  -e HF_TOKEN="${HF_TOKEN}" \
  <engine-image> \
  serve moonshotai/Kimi-K3 \
    --tp-size 32 --ep-size 32 \
    --nnodes 4 --node-rank "${NODE_RANK}" \
    --dist-init-addr "${HEAD_IP}:20000" \
    --moe-runner-backend marlin \
    --decode-attention-backend flashmla \
    --mem-fraction-static 0.85 \
    --dist-timeout 3600 \
    --reasoning-parser kimi_k3 \
    --tool-call-parser kimi_k3 \
    --host 0.0.0.0 --port 8000

Flag-by-flag, the ones that are not boilerplate:

--tp-size 32 --ep-size 32: the only feasible pair, per the arithmetic above.
--moe-runner-backend marlin: selects the W4A16 mixed-precision expert GEMM on Hopper. On Blackwell you would want the native MXFP4 path instead.
--decode-attention-backend flashmla: the MLA decode kernel. Note this covers only the 24 global-attention layers; the 69 KDA layers run their own kernels.
--mem-fraction-static 0.85: the static pool fraction. Given the budget above there is little room to raise it; raising it is the first thing people try when they want more KV cache and the first thing that causes an OOM during a long prefill.
--dist-timeout 3600: a 1.45 TiB load across four nodes is slow. The default timeout will fire before the model is ready.
--reasoning-parser kimi_k3 --tool-call-parser kimi_k3: K3 serialises trajectories with its XTML chat template, and reasoning effort is a global option message rather than a prompt prefix. A generic parser will mis-split reasoning from answer content and mangle tool calls.

Environment variables worth calling out:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True: with two cache pools of different lifetimes, allocator fragmentation is a real failure source.
NCCL_SOCKET_IFNAME / GLOO_SOCKET_IFNAME: pin these to the private interface. Leaving them unset is the classic cause of a cross-node bring-up that hangs at rendezvous or silently falls back to a slow path. See NCCL socket fallback.
If your fabric has no InfiniBand, the referenced walkthrough disables IB explicitly. On a real RDMA fabric do not disable it; verify with nccl-tests first.

How to validate the deployment¶

Do these in order; each one catches a different class of failure before you waste an hour.

Shard integrity before launch. The checkpoint is roughly 1.56 TB across 96 safetensors shards. Verify every shard's checksum after download. A truncated shard surfaces as a confusing weight-loading error 30 minutes into bring-up.
Confirm the MoE kernel selection in the startup log. If it did not pick the W4A16 path, stop; you are about to OOM.
Confirm the parallel topology. TP=32 across 4 nodes, EP=32, 28 experts per rank.
One short request. Checks the chat template, the reasoning parser, and the tool-call parser.
One long request near your real context length. This is what exercises the KDA state pool and the unified allocator. Short requests will not.
A repeated multi-turn conversation. Confirms prefix caching is actually hitting. If your second turn re-prefills from token 0, the KDA checkpoints are not being persisted or the hash granularity is misconfigured, and you have lost the largest single optimisation available.
Measure at your own effort setting. Reasoning tokens were reported at about 81% of completion tokens even at low effort, so the effort knob dominates your cost model. Measure before you size capacity.

Failure modes¶

Chose a TP degree that "looked" fine. TP=64 or TP=48 fails on divisibility, not on memory, and the error can appear as a shape mismatch deep in weight loading rather than a clean validation message.
OOM at load because the MoE kernel fell back. Dequantizing MXFP4 to BF16 in memory is a more than 3.5x blowup that no amount of mem-fraction-static tuning survives.
Distributed init timeout. The default is far too short for a 1.45 TiB multi-node load.
Rendezvous hang or slow collectives from unpinned NCCL_SOCKET_IFNAME on a multi-homed node.
Prefix caching silently disabled or ineffective. With coarse-only block matching the hybrid architecture reuses nothing; you pay full prefill on every turn of every conversation. This is invisible in correctness and brutal in cost.
Applying RoPE scaling. The MLA layers use NoPE. A rope-scaling override is at best a no-op and at worst a config that the engine rejects or misapplies.
Budgeting the KDA state pool as if it scaled with context. It scales with concurrency. Raising max concurrency raises it; raising context length does not.
Assuming single-stream throughput generalises. The reported 5.8 tokens/second single-stream against a weight-bandwidth roofline over 100x higher says decode here is latency-bound. Throughput comes from concurrency, so size and benchmark at your real batch shape.
Treating the vendor walkthrough's numbers as your numbers. Different fabric, different storage, different image, different results. Everything in the launch section is a starting point to be measured, not a specification.

References¶

Kimi K3 Technical Report (architecture Table 1, KDA-aware prefix cache, MXFP4 QAT, EAGLE-3 draft): https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf
Kimi K3 model weights: https://huggingface.co/moonshotai/Kimi-K3
Kimi Linear / Kimi Delta Attention: https://arxiv.org/abs/2510.26692
MLA (introduced in DeepSeek-V2): https://arxiv.org/abs/2405.04434
EAGLE-3 (the draft-model structure K3 fine-tunes its MTP layer into): https://arxiv.org/abs/2503.01840
Marlin mixed-precision W4A16 kernels: https://github.com/IST-DASLab/marlin
OCP Microscaling (MX) formats specification (MXFP4, MXFP8): https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf
SGLang serving documentation: https://docs.sglang.ai/