Markdown

Cookbook: serve Kimi K2 with vLLM¶

Scope: a vLLM reference template for serving moonshotai/Kimi-K2-Instruct: what Kimi K2 is, why and when to use it for agentic/tool workloads, how to size the 1T-parameter MoE, how to launch the vendor-recommended 16-GPU vLLM shapes, and how to verify tool calling and latency.

Reference template. Commands and manifests are not executed here. Kimi's own deployment guide says the smallest FP8 128K deployment unit on mainstream H200/H20 platforms is 16 GPUs. Do not silently shrink this to one 8-GPU pod for production.

What it is¶

Kimi K2 Instruct is Moonshot AI's 1T-total / 32B-activated MoE language model. The model card describes it as agentic, tool-use-oriented, and long-context, with 128K context and MLA attention. The checkpoint is released in block-FP8 form under a modified MIT license. The current instruct revision is moonshotai/Kimi-K2-Instruct-0905, which extends context to 256K; for agentic reasoning, moonshotai/Kimi-K2-Thinking (1T/32B, 256K, native INT4) adds interleaved thinking and pairs --reasoning-parser kimi_k2 with the tool-call parser. Pin whichever checkpoint you validate.

Why use it¶

Use Kimi K2 when the target product is an agent rather than a simple chatbot: repository editing, terminal workflows, multi-tool orchestration, function calling, and long multi-turn tasks. The active-parameter count keeps compute manageable, but total weight memory and 128K context make the deployment larger than typical single-node vLLM services.

When to use it (and when not)¶

Use it when:

Tool calling is a first-class requirement.
The platform can reserve 16 GPUs for one replica at 128K context.
The workload benefits from long agent state and high-quality coding/tool traces.

Avoid it when:

The serving platform only has one 8-GPU node per replica.
The application only needs short Q&A or extraction.
The team cannot validate the model's modified MIT license and third-party notices.

Architecture¶

Kimi K2 is a 1T-total / 32B-activated parameter MoE model using MLA (multi-head latent attention), released in block-FP8 form under a modified MIT license. The base Kimi-K2-Instruct checkpoint has a 128K context window; the current instruct revision, Kimi-K2-Instruct-0905, extends this to 256K, and Kimi-K2-Thinking (also 1T/32B) ships as native INT4 with a 256K context and adds interleaved thinking, surfaced through --reasoning-parser kimi_k2 alongside the kimi_k2 tool-call parser. The vendor's own deployment guide puts the smallest FP8 128K deployment unit at 16 GPUs on H200/H20-class platforms, served either as pure tensor parallelism (TP16) or as data-parallel replicas with expert parallelism (DP+EP16); the sizing and launch sections below cover both shapes.

flowchart LR
  USER["Agent request"] --> API["vLLM OpenAI endpoint"]
  API --> TOOL["Tool-call parser: kimi_k2"]
  API --> MOE["1T MoE weights"]
  MOE --> EP["TP16 or DP+EP16"]
  TOOL --> RESP["Action / answer"]

How to use it¶

1. Size the replica¶

Item	Practical note
Model	`moonshotai/Kimi-K2-Instruct`
Params	1T total / 32B activated
Context	128K
Precision	block-FP8 checkpoint
License	modified MIT
Smallest vendor shape	16 GPUs for FP8 128K on H200/H20-class platforms

For an 8-GPU development node, lower --max-model-len, use a smaller Kimi variant if available, or treat the test as a non-production smoke check only. The production recipe below assumes 16 GPUs.

2. Tensor-parallel 16-GPU launch¶

Kimi's guide shows pure tensor parallelism for parallelism degree <= 16. Start a Ray cluster across two 8-GPU nodes first, then run the server from the head node.

export MODEL_PATH=/models/Kimi-K2-Instruct

vllm serve "$MODEL_PATH" \
  --port 8000 \
  --served-model-name kimi-k2 \
  --trust-remote-code \
  --tensor-parallel-size 16 \
  --distributed-executor-backend ray \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k2 \
  --max-model-len 131072

3. DP + expert-parallel 16-GPU launch¶

This shape follows Moonshot's H200 example and is the better starting point when token routing and expert locality dominate throughput.

# node 0
export MASTER_IP=<node-0-private-ip>
export PORT=13345
export MODEL_PATH=/models/Kimi-K2-Instruct

vllm serve "$MODEL_PATH" \
  --port 8000 \
  --served-model-name kimi-k2 \
  --trust-remote-code \
  --data-parallel-size 16 \
  --data-parallel-size-local 8 \
  --data-parallel-address "$MASTER_IP" \
  --data-parallel-rpc-port "$PORT" \
  --enable-expert-parallel \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.85 \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k2

# node 1
vllm serve "$MODEL_PATH" \
  --headless \
  --data-parallel-start-rank 8 \
  --port 8000 \
  --served-model-name kimi-k2 \
  --trust-remote-code \
  --data-parallel-size 16 \
  --data-parallel-size-local 8 \
  --data-parallel-address "$MASTER_IP" \
  --data-parallel-rpc-port "$PORT" \
  --enable-expert-parallel \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.85 \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k2

4. Tool-calling smoke test¶

curl -s http://<service-url>:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "kimi-k2",
    "messages": [{"role": "user", "content": "Use the get_weather tool for Athens."}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Return current weather for a city.",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto",
    "temperature": 0.6,
    "max_tokens": 512
  }'

Pass criteria:

The response includes a structured tool call, not plain text pretending to call a tool.
--tool-call-parser kimi_k2 is present in the launch command.
vLLM logs show all 16 ranks joined.
The load test records TTFT and TPOT at the configured --max-num-seqs.

How to run it in production¶

Use KubeRay, LeaderWorkerSet, or another multi-pod gang-scheduled pattern for a 16-GPU Kimi replica. A single Kubernetes Deployment with nvidia.com/gpu: 8 is not equivalent to the vendor 16-GPU shape.

Minimal scheduler constraints for each worker pod:

spec:
  hostNetwork: true
  containers:
    - name: vllm
      image: vllm/vllm-openai:<tested-vllm-tag>
      securityContext:
        capabilities:
          add: ["IPC_LOCK"]
      env:
        - { name: NCCL_DEBUG, value: INFO }
        - { name: GLOO_SOCKET_IFNAME, value: eth0 }
      resources:
        limits: { nvidia.com/gpu: 8 }
        requests: { nvidia.com/gpu: 8 }
      volumeMounts:
        - { name: dshm, mountPath: /dev/shm }
  volumes:
    - name: dshm
      emptyDir: { medium: Memory, sizeLimit: 64Gi }

How to maintain it¶

Pin the exact checkpoint revision and the vLLM image tag you validate; do not float on :latest for either. Kimi-K2-Instruct-0905 (256K) and Kimi-K2-Thinking (256K, native INT4, adds --reasoning-parser kimi_k2) are not drop-in replacements for the base Kimi-K2-Instruct (128K, no reasoning parser) launch commands above; re-check --max-model-len and the parser flags before switching checkpoints.
Re-benchmark TP16 against DP+EP16 if decode throughput regresses. The expert-parallel all-to-all path can become the bottleneck; evaluate both shapes rather than assuming one is permanently faster.
Cap --max-model-len per route and watch TTFT. 128K (or 256K on the newer checkpoints) context inflates TTFT; do not expose the full context window to every request without measuring the impact.
Re-run the tool-calling and rank-join smoke test after any upgrade (vLLM version, model revision, or GPU topology change): confirm the response still contains a structured tool_calls entry, that vLLM logs show all 16 ranks joined, and that TTFT/TPOT at the configured --max-num-seqs have not regressed.

How to develop with it¶

This page does not document a Kimi-K2-specific fine-tuning or customization recipe. General SFT/LoRA/DPO post-training practice applies to the released checkpoints (fine-tuning, SFT/LoRA); check the current Moonshot AI Hugging Face org and repo docs for any fine-tuning guidance before assuming one exists.

Failure modes¶

OOM on 8 GPUs: expected for full 128K FP8 Kimi K2; use the 16-GPU shape or reduce scope for a dev smoke test.
Tool calls emitted as text: missing --enable-auto-tool-choice or --tool-call-parser kimi_k2.
Ranks hang on startup: Ray or data-parallel RPC networking is not reachable on the private interface.
Poor decode throughput: expert-parallel backend or all-to-all path is suboptimal; evaluate DP+EP versus TP16.
Long-context SLO miss: 128K context increases TTFT; cap context per route rather than exposing max context to every request.

References¶

Kimi K2 model card: https://huggingface.co/moonshotai/Kimi-K2-Instruct
Kimi-K2-Instruct-0905 (current instruct, 256K context): https://huggingface.co/moonshotai/Kimi-K2-Instruct-0905
Kimi-K2-Thinking (256K, native INT4, reasoning parser): https://huggingface.co/moonshotai/Kimi-K2-Thinking
Kimi K2 deployment guide: https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/docs/deploy_guidance.md
vLLM expert parallel deployment: https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/
vLLM parallelism and scaling: https://docs.vllm.ai/en/latest/serving/parallelism_scaling/