Skip to content
Markdown

Cookbook: running DeepSeek-V4-Flash locally

Scope: getting the released DeepSeek-V4-Flash-0731 weights to serve tokens on hardware you own or rent, from a multi-GPU FP4 deployment to llama.cpp CPU-MoE execution with a GPU-resident target backbone. This page separates the full checkpoint from the target-only GGUF, covers pinned engine recipes, distinguishes CPU expert execution from weight streaming, and documents the custom prompt encoder used when a framework lacks native DeepSeek-V4 support. The architecture behind the efficiency claims is DeepSeek-V4 compressed sparse attention; the speculative module is DSpark; the general serving recipe is vLLM inference deployment.

Primary sources, verified 2026-08-02: deepseek-ai/DeepSeek-V4-Flash-0731 revision 7872f01b1d1fe23eabc4c98b48bffcef5a386062, the technical report arXiv:2606.19348, official GGUF revision 471257ef58a5e48de553e065de0f084224dd1f2e, llama.cpp commit f5919bf458ef190468b5c329bb293f8a54a1e69c, vLLM commit 0601850791155003afbe5a0d5d086350cada8deb, and the SGLang DeepSeek-V4 cookbook.

What was executed here. The 48 safetensors shard headers were fetched and summed; the byte breakdown below reproduces the index's total_size of 166,878,536,440 exactly, with a delta of zero. The encoder probe downloads the model repository's own encoding_dsv4.py and asserts against it; its four shipped test cases were also run and all four pass. The vLLM and SGLang launch commands are reference templates quoted from the model card and were not executed: no GPU capable of holding this model was available here. Treat them as starting points to be validated on your hardware, and pin the container digest before you rely on either.

What it is

Three checkpoints ship under MIT licence, all with a 1M-token position limit:

Repository Total / active Notes
deepseek-ai/DeepSeek-V4-Flash-0731 284B target / 13B active; about 304B physical Current release. Supersedes the preview and attaches three DSpark draft blocks.
deepseek-ai/DeepSeek-V4-Flash 284B / 13B The earlier preview. Substantially weaker on agentic benchmarks.
deepseek-ai/DeepSeek-V4-Pro 1.6T / 49B The large sibling. Out of scope for single-node work.

DeepSeek-V4-Flash-Base (pre-post-training) and DeepSeek-V4-Flash-DSpark also exist. The latter uses preview target weights with an attached DSpark module; it is not weight-equivalent to 0731. Take 0731 unless reproducing a preview result: the model card's own comparison puts it at 82.7 on Terminal Bench 2.1 against the preview's 61.8, and at 54.4 on DeepSWE against the preview's 7.3.

The weights are natively mixed precision. Most non-expert linear matrices are FP8 e4m3 with ue8m0 block scales over 128x128 blocks. Embeddings and the untied LM head are BF16; norms, routing tables, biases, and mHC tensors use BF16, F32, or I64 as appropriate. Routed experts are stored as packed 4-bit values in int8 containers with F8_E8M0 scales, one per 32 elements, which is an MXFP4 layout. The native checkpoint needs no additional quantization for a supporting engine. Lower-bit repacking can reduce footprint and expert bytes read, with a quality and kernel-compatibility tradeoff that must be measured.

What the checkpoint is actually made of

# Byte composition measured by summing all 48 safetensors headers of
# deepseek-ai/DeepSeek-V4-Flash-0731 on 2026-08-02. The parts sum to the
# index's own total_size with a delta of exactly zero.
TOTAL = 166_878_536_440
PARTS = {
    "routed experts, 43 target layers": 147_169_738_752,
    "routed experts, 3 attached draft blocks": 10_267_656_192,
    "attention": 5_721_447_936,
    "shared experts": 1_157_698_560,
    "embedding": 1_059_061_760,
    "LM head (untied)": 1_059_061_760,
    "routers, mHC, norms, other": 443_871_480,
}
assert sum(PARTS.values()) == TOTAL
assert len(PARTS) == 7
routed = PARTS["routed experts, 43 target layers"] + PARTS["routed experts, 3 attached draft blocks"]
assert routed / TOTAL > 0.94                 # 94.3% of the file is routed-expert weight
assert (TOTAL - routed) / 1e9 < 10.0         # everything else is 9.44 GB

# One routed expert, from the shard-3 tensor shapes: w1 and w3 are [2048, 2048]
# int8 (packed 4-bit) with [2048, 128] E8M0 scales, w2 is [4096, 1024] with [4096, 64].
PER_EXPERT = 2 * (2048 * 2048 + 2048 * 128) + (4096 * 1024 + 4096 * 64)
assert PER_EXPERT * 256 * 43 == PARTS["routed experts, 43 target layers"]
assert abs(PER_EXPERT / 1e6 - 13.37) < 0.01  # 13.37 MB per expert

# Single-stream decode reads top-6 of 256 experts in each of 43 layers.
LAYERS, N_ROUTED, TOPK = 43, 256, 6
per_token = TOPK * LAYERS * PER_EXPERT
assert 3.4e9 < per_token < 3.5e9             # 3.45 GB of expert weight per token
host_memory_ceiling = 80e9 / per_token
assert 23.1 < host_memory_ceiling < 23.3       # raw 80 GB/s byte-rate ceiling, not measured speed

# Partial offload: what one layer's worth of routed experts costs and saves.
per_layer = N_ROUTED * PER_EXPERT
assert abs(per_layer / 1e9 - 3.42) < 0.01     # 3.42 GB of VRAM per layer kept on the GPU
assert abs(TOPK * PER_EXPERT / 1e6 - 80.2) < 0.1   # 80.2 MB selected per layer and token

# Expected distinct experts under an independent uniform 6-of-256 routing model.
# Actual routes are token-dependent, correlated, and deterministic in the first 3 hash layers.
expected_touched = lambda batch: N_ROUTED * (1 - (1 - TOPK / N_ROUTED) ** batch)
for batch in (1, 8, 32, 64):
    expectation = expected_touched(batch)
    assert TOPK <= expectation <= min(N_ROUTED, batch * TOPK)
assert abs(expected_touched(8) - 44.2) < 0.1
assert abs(expected_touched(32) - 136.1) < 0.2
assert expected_touched(64) > 0.75 * N_ROUTED

# Weights-only fit, before KV cache, activations, and CUDA context.
GIB = TOTAL / 2**30
assert abs(GIB - 155.4) < 0.1
assert TOTAL / 8 / 1e9 < 21                  # 8x80 GB: 20.9 GB/GPU before overhead
assert TOTAL > 2 * 80e9                      # native weights exceed 2x80 GB before runtime state

# Official GGUF is target-only: conversion skips 4,705 attached MTP/DSpark tensors.
GGUF_SIZE = 154_991_536_896
GGUF_NON_ROUTED = GGUF_SIZE - PARTS["routed experts, 43 target layers"]
assert GGUF_NON_ROUTED == 7_821_798_144       # 7.82 GB before runtime overhead and KV

print("all assertions passed")

Four facts follow, and they determine every decision on this page.

94.3% of the full checkpoint is routed-expert weight, 157.44 GB of 166.88. Attention is 5.72 GB, shared experts 1.16 GB, embedding and the untied LM head 1.06 GB each. Expert placement dominates the total footprint, but the remaining precision still sets the minimum device memory available for the target backbone, KV state, activations, and runtime buffers.

The full checkpoint has 9.44 GB outside routed experts; the official GGUF has 7.82 GB outside target experts. The difference matters because the official GGUF conversion skips all 4,705 attached MTP/DSpark tensors. It is a target-only artifact and cannot provide DSpark. Runtime metadata, KV state, compute buffers, and allocator overhead sit on top of the 7.82 GB weight floor.

Each target token selects 3.45 GB of stored expert weights across 43 layers. In llama.cpp CPU-MoE mode those expert tensors live in CPU buffers and the expert operations execute on the CPU; the weights are not demand-copied to the GPU. An 80 GB/s host-memory byte-rate gives a raw ceiling near 23.2 tok/s before CPU kernel cost, cache behavior, activation transfers, and the rest of the model. PCIe bandwidth does not yield a valid llama.cpp token-rate estimate from expert bytes alone.

Batch-level expert coverage requires route traces. Under an independent uniform-routing model, batches of 8 and 64 touch an expected 44 and 200 of 256 experts per layer. Those are model expectations, not checkpoint facts: learned routes are token-dependent and correlated, while the first three layers use deterministic hash tables. Measure expert IDs, CPU utilization, memory bandwidth, and activation-transfer time before projecting concurrency.

Why use it

Because 13B activated parameters at 1M context is an unusual point on the cost curve, and the weights are MIT-licensed. The report's efficiency claim at 1M context is 10% of DeepSeek-V3.2's single-token FLOPs and 7% of its KV cache, from a model with 284B total parameters against V3.2's 671B. For anyone running a long-context agentic workload on owned hardware, the KV figure is usually the binding one.

The 0731 release also ships its speculative decoder inside the checkpoint. There is no second model to host, no draft-target version skew, and no separate weight path: --speculative-config '{"method":"dspark",...}' in vLLM or --speculative-algorithm DSPARK in SGLang, and target and draft weights come from the same files.

When to use it (and when not)

8x 80 GB is a credible starting configuration, not a universal fit result. An even full-checkpoint partition would average 20.86 GB of weight per rank and leave about 59 GB of each marketed 80 GB capacity before runtime allocations. Data-parallel attention can replicate the non-expert portion, so the real per-rank floor depends on the engine's tensor, data, and expert-parallel placement. The model card's own reference is 4x GB300 with data-parallel 4 plus expert parallel. Measure the chosen context limit and concurrency before calling any other topology sufficient.

2x 80 GB cannot hold the native full checkpoint entirely in HBM. The 166.88 GB weights already exceed 160 GB before KV state, activations, or runtime buffers. A two-GPU deployment therefore needs a different artifact, CPU placement, or larger devices.

A single conventional GPU requires CPU expert placement or another reduced artifact. With the official target-only GGUF, non-routed weights account for 7.82 GB before KV state and runtime overhead, while the whole file is 154.99 GB. Host RAM must exceed the mapped weights plus CPU workspaces and the operating system; a nominal 160 GB total leaves no defensible headroom. This path needs workload-specific latency and peak-RSS measurements before use beyond development.

On Hopper you need a W4A16 path for the MXFP4 experts. H100 and H200 have no native FP4 arithmetic, so the experts must be upconverted in the kernel. Check that your engine build has that kernel before you size the node; the alternative is dequantizing to BF16 and losing the entire footprint advantage.

Benchmark it against a smaller dense model for short-context, high-QPS work. The 1M context capability does not itself establish an advantage at 2K tokens, and the 256-expert execution path adds operational surface. Compare quality, TTFT, TPOT, throughput, and total node cost under the target request distribution.

Architecture

flowchart TB
  subgraph CKPT["Checkpoint, 166.88 GB / 155.4 GiB, 48 shards, 72,317 tensors"]
    RE["Routed experts<br/>43 layers x 256 + 3 MTP x 256<br/>MXFP4, 13.37 MB each<br/>157.44 GB = 94.3%"]
    DENSE["Attention 5.72 GB (FP8 e4m3)<br/>Shared experts 1.16 GB<br/>Embed 1.06 + head 1.06<br/>Routers, mHC, norms 0.44"]
  end
  subgraph BIG["Node deployment: 4x GB300 or 8x H200"]
    EP["Expert parallel<br/>experts sharded across ranks"]
    DP["Data parallel attention<br/>KV cache fp8, block 256"]
    SPEC["DSpark from attached draft blocks<br/>vLLM example: 7 tokens<br/>SGLang checkpoint default: 5"]
  end
  subgraph SMALL["Single GPU + host RAM"]
    RES["Target-only GGUF<br/>7.82 GB non-routed weights + KV/runtime"]
    OFF["Expert tensors and operations on CPU<br/>3.45 GB selected weight per token"]
  end
  RE --> EP
  RE --> OFF
  DENSE --> DP
  DENSE --> RES
  EP --- DP --- SPEC
  RES --- OFF

How to use it

1. Pull the right artifact for your engine

# Native FP8 + MXFP4 weights, for vLLM / SGLang / the repo's own inference code.
hf download deepseek-ai/DeepSeek-V4-Flash-0731 \
  --revision 7872f01b1d1fe23eabc4c98b48bffcef5a386062 \
  --local-dir ./dsv4-flash-0731

# Preview-only NVFP4 recast for Blackwell. Not weight-equivalent to 0731; no DSpark attachment.
hf download nvidia/DeepSeek-V4-Flash-NVFP4 \
  --revision e3cd60e7de98e9867116860d522499a728de1cf9 \
  --local-dir ./dsv4-flash-preview-nvfp4

# Target-only GGUF for llama.cpp. It omits the attached DSpark/MTP tensors.
hf download ggml-org/DeepSeek-V4-Flash-0731-GGUF \
  --revision 471257ef58a5e48de553e065de0f084224dd1f2e \
  --local-dir ./dsv4-gguf

The pinned 0731 safetensors and official target-only GGUF contain 166.88 GB and 154.99 GB of tensor or GGUF data respectively. The NVIDIA artifact declares deepseek-ai/DeepSeek-V4-Flash as its base model, which is the preview. Community GGUF repackagings at lower bit widths can reduce both footprint and bytes per selected expert. Routing fixes the number of selected experts; the storage format and scale overhead determine the bytes read for each one.

2. Serve on a node with vLLM

Reference template for vLLM 0.25.0 on a single 4x GB300 node, adapted from the model card to use the pinned local artifact and explicit DeepSeek-V4 tokenizer mode. Unexecuted here.

vllm serve ./dsv4-flash-0731 \
  --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
  --tokenizer-mode deepseek_v4 \
  --data-parallel-size 4 --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

Four settings carry real weight. --enable-expert-parallel with --data-parallel-size controls placement for the expert-dominated checkpoint. FP8 KV state and V4's compressed-attention design reduce context memory; block size 256 is the model-card-tested allocation and kernel setting, not a compression mechanism. use_fp4_indexer_cache keeps the sparse-attention indexer's cache in FP4. The vLLM example requests seven DSpark speculative tokens from the attached draft blocks without a second model path.

Start without --speculative-config, confirm correctness, then add it and measure. Speculative decoding changes the accepted-token distribution but should not change greedy output; if it does, that is a bug worth chasing before you tune anything else.

3. Serve on a node with SGLang

Reference template verified by the SGLang project on 4x GB300 with SGLang v0.5.16. It uses the pinned local artifact and was unexecuted here.

sglang serve \
  --trust-remote-code \
  --model-path ./dsv4-flash-0731 \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --speculative-algorithm DSPARK \
  --mem-fraction-static 0.90 \
  --chunked-prefill-size 4096 \
  --swa-full-tokens-ratio 0.1

Do not pass --speculative-draft-model-path. Target and draft weights come from the same checkpoint. With no explicit DSpark block-size flag, SGLang reads dspark_block_size: 5 from the pinned 0731 config and proposes five tokens; the seven-token value belongs to the vLLM model-card example.

4. Low-VRAM path: keep the target backbone resident, execute experts on the CPU

llama.cpp exposes CPU expert placement through its MoE flags. At commit f5919bf458ef190468b5c329bb293f8a54a1e69c, the tensor override puts routed-expert tensors in a CPU buffer and the graph executes those expert operations on the CPU. It does not stream selected expert weights into HBM on demand.

Upstream carries purpose-built flags for this, so there is no need to hand-write a tensor regex. --cpu-moe (-cmoe) keeps all mixture-of-experts weights on the CPU; --n-cpu-moe N (-ncmoe) keeps the MoE weights of the first N layers on the CPU and leaves the rest on the GPU, which is the knob for filling whatever VRAM you actually have.

# Reference template, unexecuted here. The target-only GGUF has 7.82 GB outside
# its routed experts, before KV state, compute buffers, and allocator overhead.
llama-server -m ./dsv4-gguf/DeepSeek-V4-Flash-0731-MXFP4.gguf \
  --gpu-layers all --cpu-moe \
  --ctx-size 32768 --threads $(nproc) \
  --temp 1.0 --top-p 1.0 --min-p 0.0

# Partial placement: each target layer's routed experts occupy 3.42 GB, so
# -ncmoe N places N x 3.42 GB of those tensors and operations on the CPU.
llama-server -m ./dsv4-gguf/DeepSeek-V4-Flash-0731-MXFP4.gguf \
  --gpu-layers all --n-cpu-moe 32 \
  --ctx-size 32768 --threads $(nproc)

For full CPU-MoE placement, 7.82 GB is only the target-only GGUF's non-routed weight floor on the GPU. KV state, graph buffers, activations, and runtime allocations add to it. The host must map a 154.99 GB file and provide CPU workspaces and operating-system headroom, so 160 GB of total RAM is insufficient as a production sizing rule. Each target layer moved from CPU to GPU adds 3.42 GB of routed-expert weight. Selected expert operations also move activations across the device boundary, so the 3.45 GB-per-token weight-read calculation is a host-memory byte-rate model, not a PCIe traffic or token-rate prediction. Measure peak RSS, memory bandwidth, CPU time, transfer time, TTFT, and TPOT on the exact build and workload.

The model repository also ships a self-contained reference implementation in inference/, which converts the Hugging Face shards into its own layout and runs torchrun-based interactive or batch generation. It needs torch>=2.10.0, transformers>=5.0.0, safetensors>=0.7.0, fast_hadamard_transform, and tilelang==0.1.8. Use it to understand the model or to check an engine against a reference, not to serve traffic.

5. Sampling settings

The model card's recommendation is temperature = 1.0 throughout, top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise, and a maximum output length of 384K tokens for the high and max reasoning-effort levels. generation_config.json ships temperature 1.0, top_p 1.0, do_sample true. Do not carry a temperature of 0.6 or 0.7 over from another DeepSeek generation.

How to develop with it

There is no Hugging Face chat template

tokenizer_config.json has no chat_template field. A generic Transformers path that calls tokenizer.apply_chat_template(...) without supplying one raises an error. Current vLLM has a native deepseek_v4 tokenizer and renderer, and SGLang has its own V4 prompt and tool handling, so their supported OpenAI endpoints do not depend on a Jinja template in this repository. For other clients, the model repository ships encoding/encoding_dsv4.py, a dependency-free reference implementation with four test cases, plus encoding/README.md documenting the format. All four shipped tests were run here and pass.

Three properties of that encoder change how you build on it. The block below downloads the file from the model repository and asserts each one.

# Probes the encoder the model repository ships in place of a chat template.
import importlib.util, tempfile, os, urllib.request, json

URL = ("https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731"
       "/resolve/7872f01b1d1fe23eabc4c98b48bffcef5a386062"
       "/encoding/encoding_dsv4.py")
path = os.path.join(tempfile.mkdtemp(), "encoding_dsv4.py")
urllib.request.urlretrieve(URL, path)
spec = importlib.util.spec_from_file_location("encoding_dsv4", path)
enc = importlib.util.module_from_spec(spec)
spec.loader.exec_module(enc)

CHAT = [{"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is 2+2?"}]
encode = lambda **kw: enc.encode_messages(CHAT, **kw)
BOS = "<｜begin▁of▁sentence｜>"

# 1. Reasoning effort is a prompt prefix at position 0, not a sampling parameter.
low = encode(thinking_mode="thinking", reasoning_effort="low")
high = encode(thinking_mode="thinking", reasoning_effort="high")
mx = encode(thinking_mode="thinking", reasoning_effort="max")
assert encode(thinking_mode="thinking") == low          # None means "low", not "high"
assert low.startswith(BOS) and high.startswith(BOS + "Reasoning Effort:")

def shared_prefix(a, b):
    n = 0
    for x, y in zip(a, b):
        if x != y:
            break
        n += 1
    return n

assert shared_prefix(low, high) == len(BOS)             # they diverge right after BOS
assert shared_prefix(high, mx) == len(BOS) + 18         # and after "Reasoning Effort: "
assert len(high) - len(low) == 476 and len(mx) - len(low) == 526
assert encode(thinking_mode="chat", reasoning_effort="max") == encode(thinking_mode="chat")

# 2. drop_thinking is silently disabled the moment a tool schema appears.
TURNS = [{"role": "user", "content": "hi"},
         {"role": "assistant", "content": "Hello.", "reasoning_content": "X" * 4000},
         {"role": "user", "content": "and now?"}]
no_tools = enc.encode_messages(TURNS, thinking_mode="thinking")
TOOL = {"type": "function", "function": {"name": "t", "description": "d",
        "parameters": {"type": "object", "properties": {}}}}
with_tools = enc.encode_messages(
    [{"role": "system", "content": "S", "tools": [TOOL]}] + TURNS, thinking_mode="thinking")
assert "X" * 4000 not in no_tools                       # earlier reasoning dropped by default
assert "X" * 4000 in with_tools                         # kept automatically once tools exist
assert len(with_tools) - len(no_tools) > 4000

# 3. Tool calls are DSML markup, not JSON, and typing is carried by an attribute.
COMPLETION = (
    "The user wants Beijing weather.</think>\n\n"
    "<｜DSML｜tool_calls>\n"
    "<｜DSML｜invoke name=\"get_weather\">\n"
    "<｜DSML｜parameter name=\"location\" string=\"true\">Beijing</｜DSML｜parameter>\n"
    "<｜DSML｜parameter name=\"days\" string=\"false\">3</｜DSML｜parameter>\n"
    "</｜DSML｜invoke>\n</｜DSML｜tool_calls><｜end▁of▁sentence｜>")
parsed = enc.parse_message_from_completion_text(COMPLETION, thinking_mode="thinking")
assert parsed["tool_calls"][0]["function"]["name"] == "get_weather"
args = json.loads(parsed["tool_calls"][0]["function"]["arguments"])
assert args == {"location": "Beijing", "days": 3}       # string="false" is decoded as JSON
assert isinstance(args["days"], int) and isinstance(args["location"], str)

print("all assertions passed")

Reasoning effort is a prompt prefix, so it destroys prefix-cache sharing. The high and max levels prepend 476 and 526 characters of instruction text immediately after the BOS token and before the system message. A low prompt and a high prompt for the identical conversation share exactly the BOS token and nothing else. On a fleet serving mixed effort levels, that means zero prefix reuse across levels even for identical system prompts, and it means the effort level must be decided before the prompt is built rather than per request at sampling time. Route effort levels to separate cache pools or accept the miss; see prompt caching and tenant cache isolation.

The default is low, not high. Third-party documentation for repackaged builds has described the default as high-effort thinking. The shipped encoder sets DEFAULT_REASONING_EFFORT = "low" and treats None identically. If your evaluation numbers are lower than published ones, check this first: the model card evaluates the agentic benchmarks at max with temperature = 1.0, top_p = 0.95.

Tool schemas silently switch the history policy. Without tools, reasoning content from assistant turns before the last user message is stripped. With a tool schema on the system or developer message, drop_thinking is disabled automatically and every turn keeps its full <think> block. Retained message content still grows in either mode, but tool-enabled histories also retain prior reasoning and therefore grow faster. Budget for it explicitly; the context-management options are in agentic context management.

Tool calls are DSML markup with an explicit string/JSON discriminator. A parameter carries string="true" for raw strings and string="false" for anything else, and the value is decoded as JSON in the second case. An OpenAI-shaped client expecting a JSON arguments blob sees markup it cannot parse. Use the engine's own DeepSeek-V4 tool parser, or the shipped parse_message_from_completion_text, and note its own warning: it handles well-formed output only and does not recover from malformed generations, so production needs a wrapper.

The developer role exists in the encoder but is used only in an internal search pipeline and is rejected by the official API. Do not build on it.

How to run it in production

Decide the effort level per route, not per request. Given the prefix-cache consequence above, treat low, high, and max as three deployments sharing one weight set: separate cache namespaces, separate latency SLOs, separate output-length budgets (384K for the two deliberative levels). Mixing them on one endpoint converts your prefix cache into a miss generator.

Pin the KV cache dtype and block size deliberately. fp8 with --block-size 256 is the model card's tested vLLM configuration. The compressed-attention design and cache dtype determine the dominant cache footprint; block size is an allocator and kernel setting rather than compression. Changing either can alter capacity or behavior, so treat both as versioned serving parameters. KV cache management covers the general trade.

Enable DSpark second, and measure acceptance. The vLLM model-card example requests seven speculative tokens with greedy draft sampling; the checkpoint's dspark_block_size is five and SGLang uses that value when no override is supplied. Speculative decoding trades extra compute per step for fewer steps, so its benefit depends on the acceptance rate under your workload; speculative decoding economics has the break-even arithmetic. Because the draft comes from the same checkpoint, there is no separate draft-model revision to track.

Instrument expert-parallel imbalance. With 256 routed experts and 6 active, load across expert-parallel ranks is not automatically uniform. The first three target layers use deterministic hash routing, each with one tid2eid table of shape [129280, 6]; later layers use learned routing. MoE routing and load balancing and expert parallelism for inference cover the metrics to watch.

Cap context deliberately. max_position_embeddings is 1,048,576 and model_max_length in the tokenizer config matches. That is a capability, not a default: allocate the KV cache for the context you actually serve. The report says training length progressed through 4K, 16K, 64K, and 1M stages. The config's YaRN reference length is 65,536 with a factor of 16, but the released 1M window is not an inference-only extrapolation from a model trained only at 64K.

How to maintain it

Separate target-model MTP depth from attached draft blocks. The report and Transformers config.json describe target-model next-token prediction depth one. The 0731 physical checkpoint also attaches three DSpark draft blocks: inference/config.json sets n_mtp_layers: 3, and the tensor index contains block indices [0, 1, 2] totaling 10.27 GB. A full-checkpoint memory planner must count those attached blocks. A target-only converter may deliberately omit them, as the official GGUF conversion does.

No tensor in the checkpoint is named for DSpark. The dspark_* entries in both configs (dspark_block_size: 5, dspark_target_layer_ids: [40, 41, 42], dspark_markov_rank: 256, dspark_noise_token_id: 128799) configure speculative decoding over the three attached draft blocks and the last three target-model layers, not a separate draft-model directory.

Pin everything by digest. Pin the engine image, model revision, and quantized build. --trust-remote-code grants a repository permission to load custom code, so a floating revision under that flag is avoidable supply-chain risk. The pinned 0731 config.json has no auto_map, and the audited vLLM path uses native model and tokenizer support, so the flag does not by itself prove that repository code executes for this artifact. Remove it if the pinned engine works without it; otherwise retain the pin and audit what the engine imports. See container image provenance.

Re-run the encoder tests after any model or engine bump. They are four self-contained cases with no dependencies and they take under a second. They are the cheapest available detector for a prompt-format regression, which is otherwise invisible until quality drops.

Failure modes

  • A generic client calls apply_chat_template without a template. Transformers raises because the tokenizer config has no Jinja template. Use the shipped encoder or an engine's native DeepSeek-V4 renderer; do not substitute an unrelated default template.
  • Effort level set at sampling time. It is a prompt prefix. Changing it after the prompt is built does nothing; changing it per request destroys prefix-cache reuse.
  • A separate draft model passed to SGLang. --speculative-draft-model-path with DSPARK is wrong; the draft weights are in the checkpoint.
  • Two 80 GB GPUs. Their 160 GB aggregate marketed capacity is smaller than the 166.88 GB native checkpoint before runtime state. Full-HBM loading cannot fit.
  • CPU-MoE performance inferred from the uniform-routing model. The 44- and 200-expert figures are expectations under an independent model, not route traces. Batching may improve reuse or saturate CPU compute and memory; benchmark it instead of assigning a token rate from those figures.
  • MXFP4 experts on Hopper without a W4A16 kernel. Either the engine dequantizes to BF16 and the footprint triples, or the load fails. Verify the kernel path before sizing.
  • Under-counting attached draft blocks. The target-model MTP depth is one, but the full 0731 checkpoint also contains 10.27 GB across three DSpark draft blocks. A full-checkpoint planner that ignores them will fail its weight budget.
  • A tool-calling session that never compacts. Tool schemas disable reasoning-history dropping, so context grows with every turn until it hits the window.
  • Sampling carried over from an earlier DeepSeek release. Temperature 1.0 is the recommendation here, not 0.6.

References

Related: DeepSeek-V4 compressed sparse attention · DSpark speculative decoding · vLLM inference deployment · Serving OSS models · Expert parallelism for inference · MoE routing and load balancing · MoE kernels and expert backends · KV cache management · Prompt caching · Reasoning-effort control · Speculative decoding economics · Quantization for inference · Agentic context management · Tenant cache isolation · Container image provenance · vLLM on consumer GPUs · Glossary