Skip to content
Markdown

Cookbook: running DeepSeek-V4-Flash locally

Scope: getting the released DeepSeek-V4 weights to serve tokens on hardware you own or rent, at every scale from a multi-node FP4 deployment down to a single GPU with the expert pool in host memory. This page covers which checkpoint to pull and why, the exact byte composition of the weights and what it implies for placement, the launch commands the model card publishes for vLLM and SGLang, the prompt-encoding contract that ships instead of a Jinja chat template, and the failure modes that show up in that order. The architecture behind the efficiency claims is DeepSeek-V4 compressed sparse attention; the speculative module is DSpark; the general serving recipe is vLLM inference deployment.

Primary sources, verified 2026-08-02: the model card, config.json, model.safetensors.index.json, the inference/ folder and the encoding/ folder of deepseek-ai/DeepSeek-V4-Flash-0731 on Hugging Face, plus the technical report arXiv:2606.19348. This page is deliberately engine-first and vendor-neutral: every quantized build named below is identified by its repository, and the operational advice does not depend on which one you pick.

What was executed here. The 48 safetensors shard headers were fetched and summed; the byte breakdown below reproduces the index's total_size of 166,878,536,440 exactly, with a delta of zero. The encoder probe downloads the model repository's own encoding_dsv4.py and asserts against it; its four shipped test cases were also run and all four pass. The vLLM and SGLang launch commands are reference templates quoted from the model card and were not executed: no GPU capable of holding this model was available here. Treat them as starting points to be validated on your hardware, and pin the container digest before you rely on either.

What it is

Three checkpoints ship under MIT licence, all with a 1M-token position limit:

Repository Total / active Notes
deepseek-ai/DeepSeek-V4-Flash-0731 284B / 13B Current release. Supersedes the preview, adds agentic capability, ships the speculative module in-checkpoint.
deepseek-ai/DeepSeek-V4-Flash 284B / 13B The earlier preview. Substantially weaker on agentic benchmarks.
deepseek-ai/DeepSeek-V4-Pro 1.6T / 49B The large sibling. Out of scope for single-node work.

DeepSeek-V4-Flash-Base (pre-post-training) and DeepSeek-V4-Flash-DSpark (same structure as 0731) also exist. Take 0731 unless you have a specific reason not to: the model card's own comparison puts it at 82.7 on Terminal Bench 2.1 against the preview's 61.8, and at 54.4 on DeepSWE against the preview's 7.3.

The weights are natively mixed precision. Non-expert tensors are FP8 e4m3 with ue8m0 block scales over 128x128 blocks; routed experts are stored as packed 4-bit values in int8 containers with F8_E8M0 scales, one per 32 elements, which is an MXFP4 layout. Nothing needs to be quantized before you can serve it, and re-quantizing the experts is likely to lose more than it saves.

What the checkpoint is actually made of

# Byte composition measured by summing all 48 safetensors headers of
# deepseek-ai/DeepSeek-V4-Flash-0731 on 2026-08-02. The parts sum to the
# index's own total_size with a delta of exactly zero.
TOTAL = 166_878_536_440
PARTS = {"routed experts, 43 layers": 147.17e9, "routed experts, 3 MTP blocks": 10.27e9,
         "attention": 5.72e9, "shared experts": 1.16e9, "embedding": 1.06e9,
         "LM head (untied)": 1.06e9}
PARTS["routers, mHC, norms"] = TOTAL - sum(PARTS.values())
routed = PARTS["routed experts, 43 layers"] + PARTS["routed experts, 3 MTP blocks"]
assert routed / TOTAL > 0.94                 # 94.3% of the file is routed-expert weight
assert (TOTAL - routed) / 1e9 < 10.0         # everything else is 9.44 GB

# One routed expert, from the shard-3 tensor shapes: w1 and w3 are [2048, 2048]
# int8 (packed 4-bit) with [2048, 128] E8M0 scales, w2 is [4096, 1024] with [4096, 64].
PER_EXPERT = 2 * (2048 * 2048 + 2048 * 128) + (4096 * 1024 + 4096 * 64)
assert abs(PER_EXPERT * 256 * 43 - PARTS["routed experts, 43 layers"]) / 147.17e9 < 0.01
assert abs(PER_EXPERT / 1e6 - 13.37) < 0.01  # 13.37 MB per expert

# Single-stream decode reads top-6 of 256 experts in each of 43 layers.
LAYERS, N_ROUTED, TOPK = 43, 256, 6
per_token = TOPK * LAYERS * PER_EXPERT
assert 3.4e9 < per_token < 3.5e9             # 3.45 GB of expert weight per token
for gbs, ceiling in ((25.0, 7.2), (50.0, 14.5), (80.0, 23.2)):
    assert abs(gbs * 1e9 / per_token - ceiling) < 0.2

# Partial offload: what one layer's worth of routed experts costs and saves.
per_layer = N_ROUTED * PER_EXPERT
assert abs(per_layer / 1e9 - 3.42) < 0.01     # 3.42 GB of VRAM per layer kept on the GPU
assert abs(TOPK * PER_EXPERT / 1e6 - 80.2) < 0.1   # 80.2 MB/token removed from the host link

# Batching destroys the offload argument: distinct experts touched per layer.
touched = lambda B: N_ROUTED * (1 - (1 - TOPK / N_ROUTED) ** B)
assert abs(touched(8) - 44.2) < 0.1 and abs(touched(32) - 136.1) < 0.2
assert touched(64) > 0.75 * N_ROUTED         # batch 64 already reads 3/4 of every expert

# Weights-only fit, before KV cache, activations, and CUDA context.
GIB = TOTAL / 2**30
assert abs(GIB - 155.4) < 0.1
assert GIB / 8 < 0.30 * 80                   # 8x80GB: 19.4 GiB/GPU, ~60 GiB left for KV
assert GIB / 2 > 0.95 * 80                   # 2x80GB: 77.7 GiB/GPU, under 3 GiB left. No.

print("all assertions passed")

Four facts follow, and they determine every decision on this page.

94.3% of the checkpoint is routed-expert weight, 157.44 GB of 166.88. Attention is 5.72 GB, shared experts 1.16 GB, embedding and the untied LM head 1.06 GB each. Expert placement is therefore the only footprint lever that matters; nothing you do to attention or the embedding moves the number.

Everything that is not a routed expert fits in 9.44 GB. That is the whole basis of the low-VRAM path: keep 9.44 GB of dense weight resident on a consumer card, and stream the experts.

Single-stream decode reads 3.45 GB of expert weight per token. At a realistic 25 GB/s over PCIe 4.0 x16 that caps decode at about 7.2 tok/s; at 50 GB/s over PCIe 5.0, about 14.5; with the expert pool in DDR5 host memory at 80 GB/s, about 23.2. Those are ceilings assuming perfect overlap and no other traffic.

Offload is a single-stream technique and nothing else. With top-6 of 256 routing, a batch of 8 already touches 44 of 256 experts per layer (25.4 GB per step) and a batch of 64 touches 200 (114.9 GB per step). The moment you batch, you are reading most of the model anyway and the offload saving evaporates. Do not plan a shared serving tier around it.

Why use it

Because 13B activated parameters at 1M context is an unusual point on the cost curve, and the weights are MIT-licensed. The report's efficiency claim at 1M context is 10% of DeepSeek-V3.2's single-token FLOPs and 7% of its KV cache, from a model with 284B total parameters against V3.2's 671B. For anyone running a long-context agentic workload on owned hardware, the KV figure is usually the binding one.

The 0731 release also ships its speculative decoder inside the checkpoint. There is no second model to host, no draft-target version skew, and no separate weight path: --speculative-config '{"method":"dspark",...}' in vLLM or --speculative-algorithm DSPARK in SGLang, and target and draft weights come from the same files.

When to use it (and when not)

8x 80GB or better is the comfortable configuration. 155.4 GiB of weights across 8 GPUs is 19.4 GiB each, leaving roughly 60 GiB per card for KV cache and activations, which is what a 1M-context model needs. The model card's own reference is 4x GB300 with data-parallel 4 plus expert parallel.

2x 80GB does not work. 77.7 GiB per GPU of weights leaves under 3 GiB for KV cache, activations, and the CUDA context. Do not attempt it.

A single GPU works only with expert offload, only single-stream, and only if you accept the bandwidth ceiling above. You need 9.44 GB of VRAM for the dense part plus KV cache, and at least 160 GB of host RAM (or an NVMe-backed mmap and a lot of patience) for the expert pool. This is a development and evaluation configuration, not a serving one.

On Hopper you need a W4A16 path for the MXFP4 experts. H100 and H200 have no native FP4 arithmetic, so the experts must be upconverted in the kernel. Check that your engine build has that kernel before you size the node; the alternative is dequantizing to BF16 and losing the entire footprint advantage.

Do not use it for short-context, high-QPS work. The compressed-attention machinery, the 1M position table, and a 256-expert MoE are all overhead you do not need if your prompts are 2K tokens. A dense model of comparable active size will serve that workload with less operational surface.

Architecture

flowchart TB
  subgraph CKPT["Checkpoint, 166.88 GB / 155.4 GiB, 48 shards, 72,317 tensors"]
    RE["Routed experts<br/>43 layers x 256 + 3 MTP x 256<br/>MXFP4, 13.37 MB each<br/>157.44 GB = 94.3%"]
    DENSE["Attention 5.72 GB (FP8 e4m3)<br/>Shared experts 1.16 GB<br/>Embed 1.06 + head 1.06<br/>Routers, mHC, norms 0.44"]
  end
  subgraph BIG["Node deployment: 4x GB300 or 8x H200"]
    EP["Expert parallel<br/>experts sharded across ranks"]
    DP["Data parallel attention<br/>KV cache fp8, block 256"]
    SPEC["DSpark, 7 speculative tokens<br/>from the same checkpoint"]
  end
  subgraph SMALL["Single GPU + host RAM"]
    RES["Resident 9.44 GB dense + KV"]
    OFF["Expert pool in host RAM<br/>3.45 GB read per token"]
  end
  RE --> EP
  RE --> OFF
  DENSE --> DP
  DENSE --> RES
  EP --- DP --- SPEC
  RES --- OFF

How to use it

1. Pull the right artifact for your engine

# Native FP8 + MXFP4 weights, for vLLM / SGLang / the repo's own inference code.
hf download deepseek-ai/DeepSeek-V4-Flash-0731 --local-dir ./dsv4-flash-0731

# NVFP4 recast, for Blackwell-class hardware.
hf download nvidia/DeepSeek-V4-Flash-NVFP4 --local-dir ./dsv4-flash-nvfp4

# GGUF for the llama.cpp family. Upstream carries the architecture as `deepseek4`.
hf download ggml-org/DeepSeek-V4-Flash-0731-GGUF --local-dir ./dsv4-gguf

The three are 166.9 GB, 168.3 GB, and 155.0 GB respectively. Community GGUF repackagings exist at smaller sizes; they trade quality for footprint in the usual way and none of them changes the per-token expert traffic derived above, because that is set by the routing, not by the bit width.

2. Serve on a node with vLLM

Reference template, quoted from the model card for a single 4x GB300 node. Unexecuted here.

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
  --data-parallel-size 4 --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

Four flags carry real weight. --enable-expert-parallel with --data-parallel-size is the placement that matters given that 94% of the weights are experts. --kv-cache-dtype fp8 with --block-size 256 is what makes 1M context affordable. use_fp4_indexer_cache keeps the sparse-attention indexer's cache in FP4. And the DSpark config is the only speculative setting that does not require a second model.

Start without --speculative-config, confirm correctness, then add it and measure. Speculative decoding changes the accepted-token distribution but should not change greedy output; if it does, that is a bug worth chasing before you tune anything else.

3. Serve on a node with SGLang

Reference template, quoted from the model card. Unexecuted here.

sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --speculative-algorithm DSPARK \
  --mem-fraction-static 0.90 \
  --chunked-prefill-size 4096 \
  --swa-full-tokens-ratio 0.1

Do not pass --speculative-draft-model-path. Target and draft weights come from the same checkpoint, and pointing at a separate draft is the most common way to break this launch.

4. Low-VRAM path: keep the dense part resident, stream the experts

The GGUF family exposes this through the standard MoE-offload flags: put the non-expert tensors on the GPU and leave expert tensors in host memory. The sizing is the arithmetic above, not a guess.

Upstream carries purpose-built flags for this, so there is no need to hand-write a tensor regex. --cpu-moe (-cmoe) keeps all mixture-of-experts weights on the CPU; --n-cpu-moe N (-ncmoe) keeps the MoE weights of the first N layers on the CPU and leaves the rest on the GPU, which is the knob for filling whatever VRAM you actually have.

# Reference template, unexecuted here. Full offload: 9.44 GB of dense weights on the GPU.
llama-server -m ./dsv4-gguf/DeepSeek-V4-Flash-0731-MXFP4.gguf \
  --gpu-layers all --cpu-moe \
  --ctx-size 32768 --threads $(nproc) \
  --temp 1.0 --top-p 1.0 --min-p 0.0

# Partial offload: each layer's routed experts are 3.42 GB, so -ncmoe N frees N x 3.42 GB
# of VRAM and moves that much traffic onto the host link. 43 layers total.
llama-server -m ./dsv4-gguf/DeepSeek-V4-Flash-0731-MXFP4.gguf \
  --gpu-layers all --n-cpu-moe 32 \
  --ctx-size 32768 --threads $(nproc)

Budget: about 9.44 GB of VRAM for the resident dense weights plus your KV cache, and 160 GB or more of host RAM for the expert pool. Every layer you pull back onto the GPU costs 3.42 GB of VRAM and removes 80.2 MB per token from the host link, which is the arithmetic to use when tuning -ncmoe against a specific card. Expect single-stream decode in the high single digits to low twenties of tokens per second depending on host memory bandwidth, and expect it to collapse if you raise the batch size.

The model repository also ships a self-contained reference implementation in inference/, which converts the Hugging Face shards into its own layout and runs torchrun-based interactive or batch generation. It needs torch>=2.10, transformers>=5.0, fast_hadamard_transform, and tilelang==0.1.8. Use it to understand the model or to check an engine against a reference, not to serve traffic.

5. Sampling settings

The model card's recommendation is temperature = 1.0 throughout, top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise, and a maximum output length of 384K tokens for the high and max reasoning-effort levels. generation_config.json ships temperature 1.0, top_p 1.0, do_sample true. Do not carry a temperature of 0.6 or 0.7 over from another DeepSeek generation.

How to develop with it

There is no chat template. This will break your client.

tokenizer_config.json has no chat_template field. Any code path that calls tokenizer.apply_chat_template(...) fails or, worse, silently falls back to a default that produces a prompt the model was never trained on. In its place the repository ships encoding/encoding_dsv4.py, a dependency-free reference implementation with four test cases, plus encoding/README.md documenting the format. All four shipped tests were run here and pass.

Three properties of that encoder change how you build on it. The block below downloads the file from the model repository and asserts each one.

# Probes the encoder the model repository ships in place of a chat template.
import importlib.util, tempfile, os, urllib.request, json

URL = ("https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731"
       "/resolve/main/encoding/encoding_dsv4.py")
path = os.path.join(tempfile.mkdtemp(), "encoding_dsv4.py")
urllib.request.urlretrieve(URL, path)
spec = importlib.util.spec_from_file_location("encoding_dsv4", path)
enc = importlib.util.module_from_spec(spec)
spec.loader.exec_module(enc)

CHAT = [{"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is 2+2?"}]
encode = lambda **kw: enc.encode_messages(CHAT, **kw)
BOS = "<｜begin▁of▁sentence｜>"

# 1. Reasoning effort is a prompt prefix at position 0, not a sampling parameter.
low = encode(thinking_mode="thinking", reasoning_effort="low")
high = encode(thinking_mode="thinking", reasoning_effort="high")
mx = encode(thinking_mode="thinking", reasoning_effort="max")
assert encode(thinking_mode="thinking") == low          # None means "low", not "high"
assert low.startswith(BOS) and high.startswith(BOS + "Reasoning Effort:")

def shared_prefix(a, b):
    n = 0
    for x, y in zip(a, b):
        if x != y:
            break
        n += 1
    return n

assert shared_prefix(low, high) == len(BOS)             # they diverge right after BOS
assert shared_prefix(high, mx) == len(BOS) + 18         # and after "Reasoning Effort: "
assert len(high) - len(low) == 476 and len(mx) - len(low) == 526
assert encode(thinking_mode="chat", reasoning_effort="max") == encode(thinking_mode="chat")

# 2. drop_thinking is silently disabled the moment a tool schema appears.
TURNS = [{"role": "user", "content": "hi"},
         {"role": "assistant", "content": "Hello.", "reasoning_content": "X" * 4000},
         {"role": "user", "content": "and now?"}]
no_tools = enc.encode_messages(TURNS, thinking_mode="thinking")
TOOL = {"type": "function", "function": {"name": "t", "description": "d",
        "parameters": {"type": "object", "properties": {}}}}
with_tools = enc.encode_messages(
    [{"role": "system", "content": "S", "tools": [TOOL]}] + TURNS, thinking_mode="thinking")
assert "X" * 4000 not in no_tools                       # earlier reasoning dropped by default
assert "X" * 4000 in with_tools                         # kept automatically once tools exist
assert len(with_tools) - len(no_tools) > 4000

# 3. Tool calls are DSML markup, not JSON, and typing is carried by an attribute.
COMPLETION = (
    "The user wants Beijing weather.</think>\n\n"
    "<｜DSML｜tool_calls>\n"
    "<｜DSML｜invoke name=\"get_weather\">\n"
    "<｜DSML｜parameter name=\"location\" string=\"true\">Beijing</｜DSML｜parameter>\n"
    "<｜DSML｜parameter name=\"days\" string=\"false\">3</｜DSML｜parameter>\n"
    "</｜DSML｜invoke>\n</｜DSML｜tool_calls><｜end▁of▁sentence｜>")
parsed = enc.parse_message_from_completion_text(COMPLETION, thinking_mode="thinking")
assert parsed["tool_calls"][0]["function"]["name"] == "get_weather"
args = json.loads(parsed["tool_calls"][0]["function"]["arguments"])
assert args == {"location": "Beijing", "days": 3}       # string="false" is decoded as JSON
assert isinstance(args["days"], int) and isinstance(args["location"], str)

print("all assertions passed")

Reasoning effort is a prompt prefix, so it destroys prefix-cache sharing. The high and max levels prepend 476 and 526 characters of instruction text immediately after the BOS token and before the system message. A low prompt and a high prompt for the identical conversation share exactly the BOS token and nothing else. On a fleet serving mixed effort levels, that means zero prefix reuse across levels even for identical system prompts, and it means the effort level must be decided before the prompt is built rather than per request at sampling time. Route effort levels to separate cache pools or accept the miss; see prompt caching and tenant cache isolation.

The default is low, not high. Third-party documentation for repackaged builds has described the default as high-effort thinking. The shipped encoder sets DEFAULT_REASONING_EFFORT = "low" and treats None identically. If your evaluation numbers are lower than published ones, check this first: the model card evaluates the agentic benchmarks at max with temperature = 1.0, top_p = 0.95.

Tool schemas silently switch the history policy. Without tools, reasoning content from assistant turns before the last user message is stripped. With a tool schema on the system or developer message, drop_thinking is disabled automatically and every turn keeps its full <think> block. In a long agentic session that is the difference between a bounded context and one that grows with every step. Budget for it explicitly; the context-management options are in agentic context management.

Tool calls are DSML markup with an explicit string/JSON discriminator. A parameter carries string="true" for raw strings and string="false" for anything else, and the value is decoded as JSON in the second case. An OpenAI-shaped client expecting a JSON arguments blob sees markup it cannot parse. Use the engine's own DeepSeek-V4 tool parser, or the shipped parse_message_from_completion_text, and note its own warning: it handles well-formed output only and does not recover from malformed generations, so production needs a wrapper.

The developer role exists in the encoder but is used only in an internal search pipeline and is rejected by the official API. Do not build on it.

How to run it in production

Decide the effort level per route, not per request. Given the prefix-cache consequence above, treat low, high, and max as three deployments sharing one weight set: separate cache namespaces, separate latency SLOs, separate output-length budgets (384K for the two deliberative levels). Mixing them on one endpoint converts your prefix cache into a miss generator.

Pin the KV cache dtype and block size deliberately. fp8 with --block-size 256 is the model card's own configuration and is what makes 1M context tractable. Changing either changes both capacity and the numerics; treat it as a versioned serving parameter. KV cache management covers the general trade.

Enable DSpark second, and measure acceptance. Seven speculative tokens with greedy draft sampling is the published default. Speculative decoding trades extra compute per step for fewer steps, so its benefit depends on the acceptance rate under your workload; speculative decoding economics has the break-even arithmetic. Because the draft comes from the same checkpoint, there is no draft-model version to track, which removes the most common source of silent skew.

Instrument expert-parallel imbalance. With 256 routed experts, 6 active, and a hash-routing component (num_hash_layers: 3, plus a tid2eid table of shape [129280, 6] per MoE layer that maps token IDs directly to expert IDs), load across expert-parallel ranks is not automatically uniform. MoE routing and load balancing and expert parallelism for inference cover the metrics to watch.

Cap context deliberately. max_position_embeddings is 1,048,576 and model_max_length in the tokenizer config matches. That is a capability, not a default: allocate the KV cache for the context you actually serve, and remember that YaRN scaling is configured with original_max_position_embeddings: 65536 and factor: 16, so the 1M window is an extrapolation from a 64K-trained base.

How to maintain it

The two shipped configs disagree, and the checkpoint settles it. The transformers-format config.json says num_nextn_predict_layers: 1; the reference inference/config.json says n_mtp_layers: 3. Counting tensors in the released index resolves it: there are 768 MTP expert w1 tensors, which is 3 blocks x 256 experts, and the MTP block indices present are [0, 1, 2]. The checkpoint carries three MTP blocks, 10.27 GB of them. If you write a converter or a memory planner off the transformers config alone, you will under-count by that amount.

No tensor in the checkpoint is named for DSpark. The dspark_* entries in both configs (dspark_block_size: 5, dspark_target_layer_ids: [40, 41, 42], dspark_markov_rank: 256, dspark_noise_token_id: 128799) are wiring parameters over the MTP blocks and the last three main-model layers, not a separate weight set. Do not go looking for a draft-model directory.

Pin everything by digest. The engine image, the model revision, and the quantized build. --trust-remote-code is in both official launch commands, which means the model repository executes code in your serving process on every start; a floating revision under that flag is a supply-chain problem, not a convenience. See container image provenance.

Re-run the encoder tests after any model or engine bump. They are four self-contained cases with no dependencies and they take under a second. They are the cheapest available detector for a prompt-format regression, which is otherwise invisible until quality drops.

Failure modes

  • apply_chat_template returns something. There is no template in the tokenizer config, so anything you get back came from a default or from a third-party repackaging, not from DeepSeek. Symptom: fluent output, degraded task performance, missing or malformed <think> blocks.
  • Effort level set at sampling time. It is a prompt prefix. Changing it after the prompt is built does nothing; changing it per request destroys prefix-cache reuse.
  • A separate draft model passed to SGLang. --speculative-draft-model-path with DSPARK is wrong; the draft weights are in the checkpoint.
  • Two GPUs. 155.4 GiB across 2x80GB leaves under 3 GiB per card. It will either OOM at load or leave no usable KV cache.
  • Expert offload behind a batched serving endpoint. Correct at batch 1, useless by batch 8, actively harmful by batch 64 when a single step reads 115 GB of expert weight.
  • MXFP4 experts on Hopper without a W4A16 kernel. Either the engine dequantizes to BF16 and the footprint triples, or the load fails. Verify the kernel path before sizing.
  • Under-counting MTP blocks. Planning from num_nextn_predict_layers: 1 misses 10.27 GB.
  • A tool-calling session that never compacts. Tool schemas disable reasoning-history dropping, so context grows with every turn until it hits the window.
  • Sampling carried over from an earlier DeepSeek release. Temperature 1.0 is the recommendation here, not 0.6.

References

Related: DeepSeek-V4 compressed sparse attention · DSpark speculative decoding · vLLM inference deployment · Serving OSS models · Expert parallelism for inference · MoE routing and load balancing · MoE kernels and expert backends · KV cache management · Prompt caching · Reasoning-effort control · Speculative decoding economics · Quantization for inference · Agentic context management · Tenant cache isolation · Container image provenance · vLLM on consumer GPUs · Glossary