Skip to content
Markdown

Cookbook: own and run an open-weight coding model

Scope: the bring-up runbook that starts where the decision ends and stops when a developer types a prompt in their editor and gets back a working diff produced by a model you run. Whether to own at all, the five deployment states, replica and fleet sizing, the per-seat economics, and the eval sizing are the prerequisite page, own or rent a coding model; read it first and arrive here with its outputs in hand. This page owns the operational sequence: pin the checkpoint, serve it, verify the endpoint (tool calling above all), put a gateway with per-developer quotas and a hard frontier budget in front of it, wire the coding harnesses, sandbox their tool execution, harden for multi-tenant use, and gate the rollout and every later upgrade on a live-endpoint eval. It delegates sandbox depth to agent sandboxing and isolation, cache isolation to tenant cache isolation, scheduling and shedding to inference QoS and admission control, Kubernetes mechanics to the vLLM deployment recipe, and eval construction to the LLM evaluation harness.

The two Python blocks are executed and asserted with the standard library on Python 3.9+. They run without a GPU because they validate endpoint and gateway contracts, not model serving. Everything else (shell, YAML, TOML) is a reference template, unexecuted here. The worked pins are model revision da6e2ed (2026-02-03), vLLM v0.24.0 (released 2026-06-29), and LiteLLM 1.91.1 as of 2026-07-10. Flags were checked against the model card and the vLLM v0.24.0 source; re-verify them when changing pins.

What it is

Six parts, bolted together in an order where each step produces the evidence the next one depends on: a checkpoint pinned to an exact revision, a serving engine with the parser flags this model needs, an endpoint verification that would have caught the parser being wrong, a gateway that turns one endpoint into N accountable seats, harnesses pointed at the gateway rather than at the engine, and an eval that runs against the live endpoint and gates every change. The deliverable is not a served model; it is a developer-visible endpoint that a coding agent can call tools through, with a budget, an owner, and a rollback path.

Why use it

  • Each step is a checkpoint you can hold. A pinned revision makes the eval result meaningful; a verified tool-call path makes the harness wiring debuggable; a gateway key makes the seat auditable. Skip a step and the failure surfaces two steps later, where it is expensive to attribute.
  • Tool calling is the actual risk. A chat endpoint that answers questions proves almost nothing about a coding agent, which lives on tool calls. Self-hosted agent setups fail at the parser boundary more than anywhere else, which is why verification here is a step with real weight and its own executed validator, not a curl and a shrug.
  • The gateway is where policy lives. Authentication, per-developer quota, the frontier escape hatch, its hard monthly budget, cache salting, and priority all belong in one place you control, not in N developer laptops.

When to use it (and when not)

  • Use it after the decision page's four gates pass and the GPUs exist. Arrive with: the deployment state you chose, the model and replica shape, the replica count, the frontier tier you will keep as a hatch, and an internal eval sized to detect the difference you care about.
  • Do not start here. If the economics or the eval are unsettled, this page will happily help you build a fleet that should not exist.
  • For one developer on one box, use running local coding agents instead; this page's gateway, tenancy, and rollout machinery only pay for themselves at team scale.
  • For choosing a model, use serving open-weight models; this page pins one concrete worked example and the sequence transfers to any coder with an OpenAI-compatible serving path.

Architecture

The request path is harness to gateway to fleet, with the frontier hatch and the eval gate as the two control loops. The gateway is deliberately the only component that holds credentials, budgets, and tenancy; the engine only ever sees salted, prioritised requests for one model.

flowchart LR
  DEV["Developer editor / CLI"] --> H["Harness: Claude Code, Codex CLI, Qwen-Code, OpenHands"]
  H --> SBX["Sandbox: scoped filesystem, default-deny egress"]
  H -->|"per-developer virtual key"| GW["Gateway (LiteLLM 1.91.1): auth, quota, budget, cache salt, priority"]
  GW -->|"model=coder"| VLLM["vLLM v0.24.0, TP=2: Qwen3-Coder-Next-FP8 @ da6e2ed"]
  GW -->|"429/5xx only, hard monthly cap"| HATCH["Frontier API hatch"]
  VLLM --> MET["Prometheus: TTFT, ITL, prefix-cache hits"]
  GW --> SEAT["Per-seat usage, escalation rate"]
  EVALG["Paired eval against the LIVE endpoint"] -->|"gates rollout and every upgrade"| VLLM

How to use it

1. Confirm the prerequisites

From the decision page: the deployment state, the model, the GPUs per replica, the replica count, the hatch tier, and the eval set. From your platform: enough GPU memory per replica for the FP8 weights, runtime overhead, and the KV cache at the selected context length; a Postgres instance for the gateway's key store; a secrets store for the vLLM, gateway, and frontier keys; and Prometheus scraping. The model card does not establish a two-80-GB-GPU launch shape, so validate the actual hardware with the chosen context and concurrency before committing a replica shape. The checkpoint is ungated on Hugging Face, so no access token is needed for the model itself.

2. Pin the model

Item Value
Repository Qwen/Qwen3-Coder-Next-FP8
Revision (pin this) da6e2ed27304dd39abadd9c82ef50e8de67bdd4c (2026-02-03)
Precision fine-grained FP8, block size 128 (BF16 sibling exists; different memory floor)
Parameters 80B total, 3B activated
Context 262,144 tokens native
License Apache-2.0
Engine floor card states vllm>=0.15.0 (this page pins v0.24.0)

Pinning the revision matters more for a coding fleet than for a chat service. A model card is a mutable branch: the vendor can edit the chat template, and the chat template is what turns tool definitions into prompt tokens, so an upstream edit can silently change tool-call markup and break every agent in the company overnight. The eval you run in step 8 is evidence about one revision only. And this card is already internally inconsistent (its prose says tensor parallel on 4 GPUs while its own command says --tensor-parallel-size 2, with no GPU memory size stated anywhere), which is exactly why you record what you validated rather than what the card says today.

# Reference template, unexecuted. Optional pre-fetch for air-gapped or multi-node pulls;
# the serve command below pins the same revision either way.
huggingface-cli download Qwen/Qwen3-Coder-Next-FP8 \
  --revision da6e2ed27304dd39abadd9c82ef50e8de67bdd4c \
  --local-dir /models/qwen3-coder-next-fp8

3. Serve it

# Reference template, unexecuted. Pins: vLLM v0.24.0 and revision da6e2ed.
pip install vllm==0.24.0

export VLLM_API_KEY='replace-with-a-secret'

vllm serve Qwen/Qwen3-Coder-Next-FP8 \
  --revision da6e2ed27304dd39abadd9c82ef50e8de67bdd4c \
  --served-model-name qwen3-coder-next \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.90 \
  --api-key "$VLLM_API_KEY" \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --port 8000

Every flag above was checked against the card or the v0.24.0 source; the ones that repay attention:

  • --tool-call-parser qwen3_coder is the card's flag. vLLM's own tool-calling docs list qwen3_xml for the Qwen3-Coder family instead. Both are right on this engine: in v0.24.0, qwen3_coder and qwen3_xml are registered aliases of the same parser (vllm/tool_parsers/__init__.py). On older engines they were distinct parsers, so if you pin anything other than v0.24.0, verify which name your version registers before trusting either document. --enable-auto-tool-choice is mandatory alongside it.
  • No reasoning parser. The card states this model supports only non-thinking mode and emits no <think> blocks. If output contains one, you are serving the wrong checkpoint or the wrong chat template; do not add --reasoning-parser to paper over it.
  • No quantization flag. The checkpoint's own config declares the FP8 scheme; passing --quantization is at best redundant.
  • Prefix caching is already on. In v0.24.0 the default is enable_prefix_caching: bool = True (vllm/config/cache.py; the docs page does not state the default, the source does). Do not add --enable-prefix-caching; confirm it remains enabled and watch the hit rate in the observability section. --no-enable-prefix-caching is the off switch.
  • --gpu-memory-utilization 0.90 is an explicit conservative value. vLLM v0.24.0 defaults to 0.92; keeping 0.90 in the template makes the KV-memory choice visible and stable.

4. Verify the endpoint before anyone depends on it

Three probes, in order, from a machine that will not be embarrassed by the answer. First, the model list and one completion:

curl -s http://vllm-coder:8000/v1/models
# expect: {"object":"list","data":[{"id":"qwen3-coder-next",...}]}

curl -s http://vllm-coder:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen3-coder-next",
  "messages": [{"role": "user", "content": "Write a Python one-liner that reverses a string."}],
  "max_tokens": 128}'
# expect: finish_reason "stop" and code in message.content

Then the probe that actually matters, a tool call:

curl -s http://vllm-coder:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen3-coder-next",
  "messages": [{"role": "user", "content": "Read src/app.py and summarise it."}],
  "tools": [{"type": "function", "function": {"name": "read_file",
    "description": "Read a file from the repository",
    "parameters": {"type": "object", "properties": {"path": {"type": "string"}},
                   "required": ["path"]}}}],
  "tool_choice": "auto", "max_tokens": 256}'

A healthy response has finish_reason: "tool_calls" and a message.tool_calls array whose function.arguments is a JSON string. The characteristic failure, and the single most common way self-hosted coding agents break, looks superficially like success: HTTP 200, but tool_calls is absent, finish_reason is "stop", and the model's tool-call markup sits in message.content as literal text such as <tool_call>...</tool_call>. That is not a model problem; it is the serve command missing --enable-auto-tool-choice or carrying the wrong --tool-call-parser, so the engine never extracted the call. A harness pointed at such an endpoint loops, pastes XML into the chat, or worse, "succeeds" by parsing the text itself with its own bugs.

Do not eyeball this; gate it. The following validator is executed here and belongs in CI against every endpoint you stand up. It rejects the wrong-parser shape by name, along with hallucinated tools, malformed arguments, truncated calls, and partial batches:

# tool_call_gate.py -- validated: reject a bad tool call before it reaches the sandbox.
# Parses an OpenAI-style chat completion and validates every tool call against a
# registry. Fail-closed: one bad call rejects the whole response. stdlib only.
import json

TOOLS = {
    "read_file": {"required": {"path": str}, "optional": {"max_bytes": int}},
    "run_tests": {"required": {"target": str}, "optional": {}},
}

def check_args(name, raw, tools):
    """Validate one call's arguments. Return (ok, reason)."""
    if not isinstance(name, str):
        return False, "unknown-tool"
    spec = tools.get(name)
    if spec is None:
        return False, "unknown-tool"
    try:
        args = json.loads(raw)
    except (json.JSONDecodeError, TypeError):
        return False, "malformed-arguments"
    if not isinstance(args, dict):
        return False, "malformed-arguments"
    allowed = {**spec["required"], **spec["optional"]}
    if any(key not in allowed for key in args):
        return False, "unexpected-argument"
    if any(key not in args for key in spec["required"]):
        return False, "missing-argument"
    for key, typ in allowed.items():
        if key in args and type(args[key]) is not typ:
            return False, "wrong-type"
    return True, "ok"

def validate(response, tools=TOOLS):
    """Full response -> (ok, reason, calls). Always fail closed."""
    if not isinstance(response, dict):
        return False, "no-choices", []
    choices = response.get("choices") or []
    if not isinstance(choices, list) or not choices:
        return False, "no-choices", []
    if len(choices) != 1:
        return False, "multiple-choices", []
    choice = choices[0]
    if not isinstance(choice, dict):
        return False, "bad-choice-shape", []
    msg = choice.get("message") or {}
    if not isinstance(msg, dict):
        return False, "bad-message-shape", []
    raw_calls = msg.get("tool_calls")
    if raw_calls is None:
        calls = []
    elif isinstance(raw_calls, list):
        calls = raw_calls
    else:
        return False, "bad-call-shape", []
    if not calls:
        content = msg.get("content")
        if isinstance(content, str) and "<tool_call>" in content:
            return False, "parser-not-configured", []
        return False, "no-tool-call", []
    if choice.get("finish_reason") != "tool_calls":
        return False, "truncated-or-mixed", []
    out = []
    for call in calls:
        if not isinstance(call, dict):
            return False, "bad-call-shape", []
        call_id = call.get("id")
        if call.get("type") != "function" or not isinstance(call_id, str) or not call_id:
            return False, "bad-call-shape", []
        fn = call.get("function")
        if not isinstance(fn, dict):
            return False, "bad-call-shape", []
        ok, reason = check_args(fn.get("name"), fn.get("arguments"), tools)
        if not ok:
            return False, reason, []
        out.append((fn["name"], json.loads(fn["arguments"])))
    return True, "ok", out

def resp(calls=None, content=None, finish="tool_calls"):
    """Build a minimal OpenAI-style response for the checks below."""
    msg = {"role": "assistant", "content": content}
    if calls is not None:
        msg["tool_calls"] = calls
    return {"choices": [{"message": msg, "finish_reason": finish}]}

def fncall(name, args, id="call_1", type="function"):
    return {"id": id, "type": type, "function": {"name": name, "arguments": args}}

# --- Happy path: a well-formed call passes and the arguments come back parsed.
ok, why, calls = validate(resp([fncall("read_file", '{"path": "src/app.py"}')]))
assert ok and why == "ok" and calls == [("read_file", {"path": "src/app.py"})]

# --- THE PARSER TRAP. With --tool-call-parser missing or wrong, the model's tool-call
# markup arrives as plain TEXT in content, tool_calls is absent, finish_reason is "stop".
# A validator that only checks "did the request succeed" admits this garbage.
wrong_parser = resp(content='<tool_call>\n{"name": "read_file", "arguments": '
                            '{"path": "src/app.py"}}\n</tool_call>', finish="stop")
assert validate(wrong_parser) == (False, "parser-not-configured", [])
# ...and a plain refusal with no markup is a different reason, so the two are separable.
assert validate(resp(content="I cannot run tools.", finish="stop"))[1] == "no-tool-call"

# --- Hallucinated tool name: rejected, not silently passed to the sandbox.
assert validate(resp([fncall("delete_repo", '{"path": "/"}')]))[1] == "unknown-tool"

# --- Malformed arguments: truncated JSON, and valid JSON that is not an object.
assert validate(resp([fncall("read_file", '{"path": "src/a')]))[1] == "malformed-arguments"
assert validate(resp([fncall("read_file", '["src/app.py"]')]))[1] == "malformed-arguments"
assert validate(resp([fncall("read_file", None)]))[1] == "malformed-arguments"

# --- Schema boundaries: missing required, unexpected extra, wrong type, bool-as-int.
assert validate(resp([fncall("read_file", '{}')]))[1] == "missing-argument"
assert validate(resp([fncall("read_file", '{"path": "a", "mode": "w"}')]))[1] == "unexpected-argument"
assert validate(resp([fncall("read_file", '{"path": 7}')]))[1] == "wrong-type"
assert validate(resp([fncall("read_file", '{"path": "a", "max_bytes": true}')]))[1] == "wrong-type"
assert validate(resp([fncall("read_file", '{"path": "a", "max_bytes": 4096}')]))[0]

# --- Fail-closed on a batch: one bad call poisons the response; the good call is NOT
# returned for execution. Partial admission is how a bad call slips through review.
mixed = resp([fncall("read_file", '{"path": "a"}'),
              fncall("nuke_prod", '{"target": "db"}', id="call_2")])
assert validate(mixed) == (False, "unknown-tool", [])

# --- Truncation: finish_reason "length" means the arguments may be cut mid-string and
# still parse (a shorter valid prefix). Reject on the finish reason, not the JSON.
assert validate(resp([fncall("read_file", '{"path": "a"}')], finish="length"))[1] == "truncated-or-mixed"

# --- Shape garbage never crashes the gate and never passes it.
assert validate({}) == (False, "no-choices", [])
assert validate({"choices": []}) == (False, "no-choices", [])
assert validate(resp([]))[1] == "no-tool-call"
assert validate(resp([{"type": "function", "function": {"name": "read_file",
                       "arguments": "{}"}}]))[1] == "bad-call-shape"   # id missing
assert validate(resp([fncall("read_file", '{"path": "a"}', type="tool")]))[1] == "bad-call-shape"
assert validate(None) == (False, "no-choices", [])
assert validate({"choices": [{}, {}]}) == (False, "multiple-choices", [])
assert validate({"choices": [{"message": {"tool_calls": "not-a-list"}}]})[1] == "bad-call-shape"
assert validate(resp([fncall(["read_file"], "{}")]))[1] == "unknown-tool"

Two details in there earn their keep in production. The bool-as-int check exists because isinstance(True, int) is True in Python, so a schema checker built on isinstance accepts {"max_bytes": true} and hands your file reader a boolean. And truncation is rejected on finish_reason, not on JSON validity, because a JSON string cut mid-way can still parse as a shorter valid prefix; {"path": "src/a fails to parse, but {"path": "src"} truncated from a longer path parses fine and reads the wrong file.

5. Front it with a gateway

Developers never talk to the engine. The gateway owns four policies: who may call (virtual keys), how much each seat may spend (quota with a reset period), where requests go (the owned fleet, with a frontier fallback), and what happens when money runs out (reject, visibly). The template pins LiteLLM 1.91.1:

# litellm-config.yaml -- reference template, unexecuted. pip install 'litellm[proxy]==1.91.1'
# Virtual keys require Postgres: export DATABASE_URL=postgresql://user:pass@host:5432/litellm
model_list:
  - model_name: coder                              # the only name developers ever see
    litellm_params:
      model: openai/qwen3-coder-next                # OpenAI-compatible upstream
      api_base: http://vllm-coder.serving.svc:8000/v1
      api_key: os.environ/VLLM_API_KEY
      use_chat_completions_api: true                # required for /v1/responses bridge
  - model_name: frontier
    litellm_params:
      model: anthropic/claude-sonnet-4-6
      api_key: os.environ/FRONTIER_API_KEY         # provider key stays server-side

router_settings:
  fallbacks: [{"coder": ["frontier"]}]             # generic fallback; filter 4xx before routing
  num_retries: 2

litellm_settings:
  request_timeout: 120

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY        # admin credential, never a developer's

Issue one virtual key per developer, scoped to coder, with a monthly budget that resets:

curl -s https://llm-gw.internal/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"models": ["coder"], "max_budget": 25, "budget_duration": "30d",
       "metadata": {"developer": "ana"}}'

A key over its budget is rejected with HTTP 429 and error type budget_exceeded (message text typically ExceededBudget: ...); a key without budget_duration never resets. LiteLLM generic fallbacks cover all remaining errors, not only 429 and 5xx. This YAML does not prevent a 4xx from reaching the frontier. Enforce the 429/5xx-only policy with a pre-routing status classifier or equivalent gateway middleware, and test it against validation failures.

The hard monthly cap on the hatch itself lives in two layers: a spend alert on the gateway's escalation counter for early warning, and a hard spend limit on the frontier provider key in the vendor's own console, so that exhausting the hatch turns the hatch off while the fleet keeps serving. What the combined machine must guarantee is easy to state and easy to get wrong, so it is stated here as executed code. This is the contract, with the properties proven: reservations count against quota so concurrent requests cannot double-claim headroom, settlement is exactly-once so a retry cannot double-spend, client errors never escalate, the cap counts in-flight holds so it cannot be overrun, and exhaustion rejects rather than degrades:

# gateway_budget.py -- validated: per-developer quota, escape hatch, hard budget.
# Reserve-then-settle accounting: a request holds its worst case until it settles,
# so concurrent in-flight requests can never overrun a cap. Rejects; never degrades.

import math

def is_money(value):
    return type(value) in (int, float) and math.isfinite(value) and value >= 0

class Gateway:
    def __init__(self, dev_quota_tokens, hatch_cap_usd):
        assert type(dev_quota_tokens) is int and dev_quota_tokens > 0
        assert is_money(hatch_cap_usd)
        self.quota = dev_quota_tokens
        self.cap = hatch_cap_usd
        self.used = {}
        self.held = {}
        self.hatch_spent = 0.0
        self.hatch_held = {}
        self.done = set()                      # settled request ids, never reused

    def admit(self, dev, req, max_tokens):
        """Reserve max_tokens against quota, or reject the whole request."""
        assert type(max_tokens) is int and max_tokens > 0
        if req in self.held or req in self.done:
            return "reject-duplicate"
        in_flight = sum(t for d, t in self.held.values() if d == dev)
        if self.used.get(dev, 0) + in_flight + max_tokens > self.quota:
            return "reject-quota"
        self.held[req] = (dev, max_tokens)
        return "fleet"

    def escalate(self, req, fleet_status, est_usd):
        """Fleet failed. Only 429 and 5xx may use the hatch."""
        assert req in self.held and req not in self.done
        assert is_money(est_usd)
        if req in self.hatch_held:
            return "hatch"                         # idempotent retry
        if not (fleet_status == 429 or 500 <= fleet_status <= 599):
            return "reject-client-error"
        if self.hatch_spent + sum(self.hatch_held.values()) + est_usd > self.cap:
            return "reject-hatch-budget"
        self.hatch_held[req] = est_usd
        return "hatch"

    def settle(self, req, tokens, usd=0.0):
        """Settle exactly once; all checks precede mutation."""
        assert req not in self.done, "double settle"
        assert req in self.held, "unknown request"
        assert type(tokens) is int and tokens >= 0
        assert is_money(usd)
        dev, reserved = self.held[req]
        assert tokens <= reserved, "settled above reservation"
        assert usd <= self.hatch_held.get(req, 0.0), "settled above hatch reservation"
        del self.held[req]
        self.hatch_held.pop(req, None)
        self.used[dev] = self.used.get(dev, 0) + tokens
        self.hatch_spent += usd
        self.done.add(req)

    def new_month(self):
        assert not self.held and not self.hatch_held, "reset with requests in flight"
        self.used, self.hatch_spent = {}, 0.0

# Each throwaway Gateway is discarded right after construction, so the sentinel
# "unreachable" message (not just AssertionError) is what proves the real check fired,
# rather than the object being built and merely never used again.
for bad_quota in (0.5, True, -1, 0):
    try:
        Gateway(dev_quota_tokens=bad_quota, hatch_cap_usd=10.0)
        assert False, "unreachable"
    except AssertionError as e:
        assert str(e) != "unreachable", f"quota check let {bad_quota!r} through"
for bad_cap in (-1.0, True, "10"):
    try:
        Gateway(dev_quota_tokens=1000, hatch_cap_usd=bad_cap)
        assert False, "unreachable"
    except AssertionError as e:
        assert str(e) != "unreachable", f"hatch_cap_usd check let {bad_cap!r} through"

gw = Gateway(dev_quota_tokens=1_000_000, hatch_cap_usd=100.0)
for bad in (-1, 0.5, True):
    try:
        gw.admit("ana", "bad", bad); assert False, "unreachable"
    except AssertionError:
        pass
assert gw.admit("ana", "negative-tokens", 100) == "fleet"
for bad_tokens in (-1, 0.5, True):
    try:
        gw.settle("negative-tokens", bad_tokens); assert False, "unreachable"
    except AssertionError:
        pass
assert gw.used.get("ana", 0) == 0
gw.settle("negative-tokens", 0)
assert gw.admit("ana", "negative-usd", 100) == "fleet"
try:
    gw.settle("negative-usd", 0, usd=-0.01); assert False, "unreachable"
except AssertionError:
    pass
gw.settle("negative-usd", 0)

# --- Happy path: admit, serve on the fleet, settle the actual usage.
assert gw.admit("ana", "r1", 40_000) == "fleet"
gw.settle("r1", 31_000)
assert gw.used["ana"] == 31_000

# --- NO DOUBLE-SPEND. Settling the same request twice must fail loudly, and the
# replayed id must not be admitted again either (retries reuse the request id).
try:
    gw.settle("r1", 31_000); assert False, "unreachable"
except AssertionError as e:
    assert "double settle" in str(e)
assert gw.used["ana"] == 31_000                      # charged exactly once
assert gw.admit("ana", "r1", 1) == "reject-duplicate"

# --- Quota is enforced against RESERVATIONS, not just settled spend: two concurrent
# requests cannot each claim the same remaining headroom.
assert gw.admit("ana", "r2", 900_000) == "fleet"     # 31k used + 900k held = 931k
assert gw.admit("ana", "r3", 100_000) == "reject-quota"   # 931k + 100k > 1M
assert gw.admit("ana", "r4", 69_000) == "fleet"      # 931k + 69k = 1M exactly: admitted
assert gw.admit("ana", "r5", 1) == "reject-quota"    # one token over: rejected
gw.settle("r2", 100_000)                             # actual far below the hold
assert gw.used["ana"] == 131_000                     # spend ACCUMULATES across requests
assert gw.admit("ana", "r5", 1) == "fleet"           # freed headroom returns
gw.settle("r4", 0); gw.settle("r5", 0)

# --- A rejection must not mutate state (reject-then-retry would otherwise leak quota).
before = (dict(gw.used), dict(gw.held))
assert gw.admit("ana", "r6", 10_000_000) == "reject-quota"
assert (dict(gw.used), dict(gw.held)) == before

# --- Settling above the reservation is a contract violation, not a bigger bill: the
# gateway sets the request's max_tokens to the reservation, so this cannot happen
# unless the accounting is wrong. Fail loudly, mutate nothing.
assert gw.admit("bob", "r7", 10_000) == "fleet"
snap = (dict(gw.used), dict(gw.held), gw.hatch_spent)
try:
    gw.settle("r7", 10_001); assert False, "unreachable"
except AssertionError as e:
    assert "above reservation" in str(e)
assert (dict(gw.used), dict(gw.held), gw.hatch_spent) == snap    # failed settle: no-op
gw.settle("r7", 10_000)                                          # at the bound: fine

# --- Charging hatch dollars on a request that never escalated is the same bug class:
# there is no reservation to charge against, so the settle must refuse.
assert gw.admit("bob", "r8", 1_000) == "fleet"
try:
    gw.settle("r8", 500, usd=0.50); assert False, "unreachable"
except AssertionError as e:
    assert "hatch reservation" in str(e)
gw.settle("r8", 500)

# --- THE ESCAPE HATCH: 429 and 5xx escalate; 4xx never does. A malformed request
# that escalated would burn frontier budget on garbage, forever, on every retry.
gw2 = Gateway(dev_quota_tokens=1_000_000, hatch_cap_usd=10.0)
assert gw2.admit("ana", "h1", 50_000) == "fleet"
assert gw2.escalate("h1", 400, est_usd=1.0) == "reject-client-error"
assert gw2.escalate("h1", 429, est_usd=4.0) == "hatch"
gw2.settle("h1", 20_000, usd=3.10)
assert gw2.hatch_spent == 3.10

# --- HARD BUDGET, counted with in-flight holds: 3.10 spent + 6.00 held caps out.
assert gw2.admit("bob", "h2", 50_000) == "fleet"
assert gw2.escalate("h2", 503, est_usd=6.0) == "hatch"          # 3.10 + 6.00 <= 10
assert gw2.escalate("h2", 503, est_usd=999.0) == "hatch"  # repeated escalation is idempotent
assert gw2.admit("cyn", "h3", 50_000) == "fleet"
assert gw2.escalate("h3", 503, est_usd=1.0) == "reject-hatch-budget"  # 9.10 + 1.0 > 10
gw2.settle("h2", 10_000, usd=0.40)                              # actual: only $0.40
assert gw2.escalate("h3", 503, est_usd=1.0) == "hatch"          # freed: 3.50 + 1.0 <= 10
gw2.settle("h3", 10_000, usd=1.0)
assert gw2.hatch_spent == 4.50 <= gw2.cap
# The cap is inclusive: a reservation landing EXACTLY on it is admitted, not rejected.
assert gw2.admit("dan", "h4", 1_000) == "fleet"
assert gw2.escalate("h4", 503, est_usd=5.50) == "hatch"         # 4.50 + 5.50 == 10.00
gw2.settle("h4", 0, usd=5.50)
assert gw2.hatch_spent == 10.0 == gw2.cap

# --- REJECT, DON'T DEGRADE: exhaustion is a refusal the developer can see, never a
# silent downgrade. Sweep: whatever the arrival order, total hatch spend never
# exceeds the cap and every over-cap request was rejected, not shrunk.
gw3 = Gateway(dev_quota_tokens=10_000_000, hatch_cap_usd=20.0)
outcomes = []
for i, cost in enumerate([7.0, 7.0, 7.0, 7.0, 7.0]):
    rid = f"s{i}"
    gw3.admit("dev", rid, 1000)
    verdict = gw3.escalate(rid, 503, est_usd=cost)
    outcomes.append(verdict)
    gw3.settle(rid, 500, usd=cost if verdict == "hatch" else 0.0)
assert outcomes == ["hatch", "hatch", "reject-hatch-budget", "reject-hatch-budget",
                    "reject-hatch-budget"]
assert gw3.hatch_spent == 14.0 <= gw3.cap            # 2 * 7.0; never 21.0

# --- Month reset restores admission, and refuses to run with requests in flight.
gw4 = Gateway(dev_quota_tokens=100, hatch_cap_usd=0.0)
assert gw4.admit("ana", "m1", 100) == "fleet"
try:
    gw4.new_month(); assert False, "unreachable"
except AssertionError as e:
    assert "in flight" in str(e)
gw4.settle("m1", 100)
assert gw4.admit("ana", "m2", 1) == "reject-quota"
gw4.new_month()
assert gw4.admit("ana", "m1", 1) == "reject-duplicate"  # request IDs remain unique across resets
assert gw4.admit("ana", "m3", 100) == "fleet"        # fresh month, fresh quota
# A zero-dollar cap means the hatch never opens: the fleet-only configuration.
assert gw4.escalate("m3", 503, est_usd=0.01) == "reject-hatch-budget"

Whatever gateway product you run, hold it to those assertions. The sweep at the end is the whole policy in one line: five identical $7.00 escalations against a $20.00 cap admit exactly two and settle at $14.00, and the three that did not fit were refused, not shrunk, not queued onto the fleet that had just failed, and not billed.

6. Wire the coding harnesses

Every harness points at the gateway with the developer's virtual key, and every harness asks for the same served name, coder. The gateway speaks three dialects on one port: OpenAI /v1/chat/completions, Anthropic /v1/messages, and OpenAI /v1/responses, so each harness uses its native one.

# Qwen-Code (OpenAI-compatible; variable names per its auth docs)
export OPENAI_BASE_URL=https://llm-gw.internal/v1
export OPENAI_API_KEY=sk-...        # the developer's virtual key
export OPENAI_MODEL=coder

# OpenHands CLI (LiteLLM-style provider prefix on the model name). The CLI ignores
# these in favor of ~/.openhands/agent_settings.json unless launched with the flag below.
export LLM_MODEL=openai/coder
export LLM_BASE_URL=https://llm-gw.internal/v1
export LLM_API_KEY=sk-...
# openhands --override-with-envs

# Claude Code (Anthropic Messages format; the gateway's /v1/messages translates)
export ANTHROPIC_BASE_URL=https://llm-gw.internal
export ANTHROPIC_AUTH_TOKEN=sk-...  # bearer credential; ANTHROPIC_API_KEY is the x-api-key variant
export ANTHROPIC_MODEL=coder

# Codex CLI: ~/.codex/config.toml (config reference, current as of 2026-07-10)
model = "coder"
model_provider = "owned"

[model_providers.owned]
name = "Owned coding fleet"
base_url = "https://llm-gw.internal/v1"
env_key = "OWNED_API_KEY"     # export OWNED_API_KEY=<the developer's virtual key>
wire_api = "responses"        # "responses is the only supported value" in the current reference

The version-dependent facts, stated with their versions: Codex's current config reference documents responses as the only supported wire_api, so Codex requires the gateway's /v1/responses endpoint; LiteLLM bridges Responses API calls onto chat-completions backends when use_chat_completions_api: true is set for the OpenAI-compatible upstream. Claude Code accepts a custom model name through the gateway configuration path, but the gateway must accept and translate coder; do not assume an arbitrary name works after a client update. Anthropic documents gateways for Claude models, so routing Claude Code to a non-Claude model remains outside that documented support surface and requires an endpoint test on every release.

7. Sandbox what the agent executes

The model writes code and the harness runs it, so from here on the endpoint is an execution path into whatever the harness can touch. Full depth is in agent sandboxing and isolation and prompt-injection defense; the concrete minimum for one seat is:

# Reference template, unexecuted: one seat, one container, pinned harness image.
docker run --rm -it \
  --user 1000:1000 --cap-drop ALL --security-opt no-new-privileges \
  --network agent-egress \
  -v "$PWD":/workspace -w /workspace \
  -e OPENAI_BASE_URL -e OPENAI_API_KEY -e OPENAI_MODEL \
  ghcr.io/example/harness:<pinned-tag>
  • Filesystem scope: the working copy only, mounted read-write; nothing else writable, no home directory, no ~/.ssh, no Docker socket. Treat the checkout as disposable and reviewable (a branch, not main).
  • Network egress: default-deny. The named network above must be enforced by something that actually filters (a Kubernetes NetworkPolicy or nftables on the host; a plain Docker bridge does not), with an allowlist of exactly the gateway and your package mirrors. A coding agent with open egress plus a hostile string in a README is an exfiltration primitive.
  • Secrets: the virtual key is the only credential inside the sandbox, it is spend-capped and revocable per developer, and it is injected as an environment variable at launch, never baked into the image and never committed as harness config. Cloud credentials and signing keys stay outside.

8. Gate the rollout on the acceptance test

Run the internal paired eval from the decision page against the live endpoint, through the gateway, with a dedicated CI key, before the first developer is invited. Offline eval results (the model loaded in a notebook) do not transfer: the live path adds the chat template as served, the tool-call parser, the gateway translation, and the context limit, and any of the four can be the thing that is broken. The gate is mechanical:

  1. The tool-call validator above passes on a sample of live responses (it is cheap; run hundreds).
  2. The paired eval clears its pre-registered threshold: exact McNemar p-value on the observed discordant counts, plus the no-regression floors, per the LLM evaluation harness and agent evaluation.
  3. TTFT and ITL under a replayed concurrent load meet the SLOs you set from inference serving SLOs.

Record the revision, engine version, gateway version, harness versions, and the eval's b, c, and p in the rollout ticket. That tuple is what "the endpoint works" means from now on. Then roll out in rings: one pilot team behind its own gateway keys, then the fleet.

How to develop with it

Day-to-day change lands in three places, and all three go through the step 8 gate rather than around it. New tools: the tool registry (the TOOLS dict pattern above) is an interface; adding a tool means adding its schema to the validator, a task exercising it to the eval set, and only then the tool to the harness. Prompt and harness changes: a system-prompt edit or a harness version bump changes agent behaviour as much as a model change does, so it rides the same paired eval, and the pinned triple (model revision, engine version, harness version) moves together or not at all. Adaptation: when recurring house-specific failures survive harness and context work, the decision page's ladder points to LoRA on your merged pull requests; serve per-team adapters on the same base via multi-LoRA serving rather than forking the fleet.

How to run it in production

Multi-tenant hardening. Two request fields, both injected by the gateway so no developer can opt out. First, cache_salt: vLLM's prefix cache is shared across everyone hitting the replica, which is a cache-timing side channel between teams; the v0.24.0 chat-completions request schema carries a cache_salt field folded into the block hash, so requests with different salts never share cached blocks. Set one salt per team (a trust group that keeps the caching win) or per developer (maximum isolation, lower hit rate); the attack surface and the trade-off live in tenant cache isolation. Second, priority: launch the engine with --scheduling-policy priority and have the gateway stamp interactive completions priority 0 and background agent loops priority 10; in vLLM lower means earlier handling, the default is 0, and a non-zero priority on a server not launched with priority scheduling raises an error, so the flag and the field deploy together. Admission and shedding policy beyond that is inference QoS and admission control.

Deployment shape. Run the engine as the vLLM deployment recipe Deployment with the image pinned to vllm/vllm-openai:v0.24.0 and the step 3 args verbatim; the recipe owns probes, rollout, and autoscaling. The gateway is stateless apart from Postgres and scales horizontally in front.

Observability. Metric names per the v0.24.0 source (vllm/v1/metrics/loggers.py); vLLM documents counters without the _total suffix that the Prometheus client appends at exposition, so PromQL uses _total:

Signal Where Metric Measures
Per-developer tokens and spend gateway key-level usage from the LiteLLM DB cost
Prefix-cache hit rate engine vllm:prefix_cache_hits / vllm:prefix_cache_queries (_total in PromQL) cost
TTFT engine vllm:time_to_first_token_seconds cost (latency SLO)
Inter-token latency (TPOT) engine vllm:inter_token_latency_seconds, per-request vllm:request_time_per_output_token_seconds cost (latency SLO)
Escalation rate to the hatch gateway fallback events per hour capability gap
Diff acceptance rate harness telemetry accepted diffs / proposed diffs value

Only the last row measures value; everything above it measures what the value costs. A fleet whose token counts rise while acceptance rate falls is a fleet developers are fighting, not using, and the escalation rate is the early sign that the capability gate is drifting: if the hatch share climbs month over month, the owned model is falling behind the work. Wire the export through GenAI observability.

How to maintain it

  • Upgrade nothing in place. A new engine version, model revision, or harness release goes to a canary replica behind the gateway first, gets the step 8 gate (validator, paired eval, latency under load), and is promoted only on a pass. The qwen3_coder/qwen3_xml aliasing is the cautionary tale: parser names and behaviour move between vLLM releases, so the tool-call smoke test re-runs on every engine bump even when the release notes look unrelated.
  • Keep rollback one command. The previous (image, revision) pair stays deployable; on Kubernetes that is kubectl rollout undo plus the pinned args, and the gateway config is versioned next to it. A rollback that needs a rebuild is not a rollback.
  • Re-measure what the economics rest on. After any engine change, re-measure per-replica prefill and decode throughput and re-run the decision page's marginal-cost criterion; an engine upgrade can move the binding phase and, through it, the fleet size and the rent-vs-own answer itself.
  • Budget hygiene. Every key has budget_duration set (a budget without one never resets), the hatch's provider-side spend limit matches the cap you told finance, and the escalation counter has an alert threshold below the cap, not at it.
  • Re-baseline the eval from recent merged pull requests on a schedule, so the gate keeps measuring your work rather than what leaked into training corpora.

Failure modes

  • Tool calls come back as prose. HTTP 200, finish_reason: "stop", <tool_call> markup in message.content, empty tool_calls. Missing --enable-auto-tool-choice or wrong --tool-call-parser; the validator's parser-not-configured is this exact signature.
  • Agents break overnight with no deploy. An unpinned model reference picked up an upstream card edit; a chat-template change moved the tool-call markup. The fix is the --revision pin; the tell is that yesterday's eval no longer reproduces.
  • Engine bump silently changes tool calling. Parser registrations move between vLLM releases (two names alias one parser in v0.24.0; they were distinct earlier). The canary gate plus the tool-call smoke test catches it; upgrading in place does not.
  • <think> blocks appear in output. Wrong checkpoint or wrong chat template; this model is non-thinking by its card. Fix the artefact; do not add a reasoning parser to strip symptoms.
  • OOM on load after a "harmless" precision swap. The BF16 sibling uses substantially more memory, and the model card does not establish a two-80-GB-GPU shape. Re-derive the weight, runtime, and KV-cache memory floor before changing precision or context length, per the decision page.
  • Interactive TTFT collapses during work hours. Background agent loops are competing at equal priority. Deploy --scheduling-policy priority with gateway-stamped priorities; confirm with the TTFT histogram split by priority label.
  • One team's prompts warm another team's cache. Shared prefix cache without cache_salt is a timing side channel; salted requests never share blocks. Symptom: cross-tenant latency correlation, per tenant cache isolation.
  • Frontier bill spikes mid-month. The hatch is escalating client errors (4xx) or retrying into it; escalation must trigger on 429/5xx only, and the block above rejects the rest. Check the gateway's fallback log for the originating status codes.
  • A developer is locked out and stays locked out. Their key hit max_budget and has no budget_duration, so it will never reset; the rejection is HTTP 429 with error type budget_exceeded.
  • Rising vllm:num_preemptions_total, thrashing agents. KV pressure at 262K context under concurrent loops; lower --max-model-len or add a replica, per the KV-cache OOM runbook.
  • The offline eval passed and the live agent fails. The notebook bypassed the served chat template, the parser, and the gateway translation. Acceptance runs against the live endpoint or it is not acceptance.
  • Claude Code starts erroring on gateway responses after an update. New request fields outpaced the gateway's translation layer (the symptom is 400s naming unrecognised fields). Re-verify the /v1/messages path on every harness release; the wiring is outside Anthropic's support surface.

References

  • Qwen3-Coder-Next-FP8 model card (fine-grained FP8, 80B total / 3B active, 262,144 context, Apache-2.0, vllm serve ... --tensor-parallel-size 2 --enable-auto-tool-choice --tool-call-parser qwen3_coder, vllm>=0.15.0; its prose says 4 GPUs while its command says TP 2): https://huggingface.co/Qwen/Qwen3-Coder-Next-FP8
  • vLLM tool calling (parser catalogue; lists qwen3_xml for the Qwen3-Coder family; --enable-auto-tool-choice requirement): https://docs.vllm.ai/en/latest/features/tool_calling.html
  • vLLM v0.24.0, vllm/tool_parsers/__init__.py (qwen3_coder and qwen3_xml registered as aliases of the same parser): https://github.com/vllm-project/vllm/blob/v0.24.0/vllm/tool_parsers/init.py
  • vLLM v0.24.0, vllm/config/cache.py (enable_prefix_caching: bool = True, the default the docs page does not state): https://github.com/vllm-project/vllm/blob/v0.24.0/vllm/config/cache.py
  • vLLM automatic prefix caching (what caching accelerates and what it does not): https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html
  • vLLM v0.24.0, chat-completions request schema (priority: lower is earlier, errors without priority scheduling; cache_salt): https://github.com/vllm-project/vllm/blob/v0.24.0/vllm/entrypoints/openai/chat_completion/protocol.py
  • vLLM production metrics (counter names; _total appended at exposition): https://docs.vllm.ai/en/latest/usage/metrics.html
  • LiteLLM: vLLM provider (hosted_vllm/ prefix, api_base): https://docs.litellm.ai/docs/providers/vllm
  • LiteLLM: virtual keys (/key/generate, Postgres and master-key requirements): https://docs.litellm.ai/docs/proxy/virtual_keys
  • LiteLLM: budgets (max_budget, budget_duration, HTTP 429 budget_exceeded on limit, no-reset default): https://docs.litellm.ai/docs/proxy/users
  • LiteLLM: fallbacks and reliability (generic fallbacks cover all remaining errors; enforce any 429/5xx-only policy outside the generic setting): https://docs.litellm.ai/docs/proxy/reliability
  • LiteLLM: Anthropic-format /v1/messages endpoint: https://docs.litellm.ai/docs/anthropic_unified
  • LiteLLM: Responses API endpoint and chat-completions bridge (use_chat_completions_api): https://docs.litellm.ai/docs/response_api
  • Claude Code: connect to an LLM gateway (ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN vs ANTHROPIC_API_KEY): https://code.claude.com/docs/en/llm-gateway-connect
  • Claude Code: model configuration (ANTHROPIC_MODEL and custom model names behind a gateway): https://code.claude.com/docs/en/model-config
  • Claude Code: LLM gateways overview (third-party gateways not endorsed or audited; routing to non-Claude models is not a supported configuration): https://code.claude.com/docs/en/llm-gateway
  • Claude Code: gateway protocol reference (the feature pass-through table; request fields and headers change release to release): https://code.claude.com/docs/en/llm-gateway-protocol
  • Codex CLI configuration reference ([model_providers], wire_api = "responses" as the only supported value): https://developers.openai.com/codex/config-reference
  • Qwen-Code authentication (OPENAI_BASE_URL, OPENAI_API_KEY, OPENAI_MODEL): https://qwenlm.github.io/qwen-code-docs/en/users/configuration/auth/
  • OpenHands: CLI command reference (LLM_MODEL, LLM_BASE_URL, LLM_API_KEY, and the --override-with-envs flag they require): https://docs.openhands.dev/openhands/usage/cli/command-reference
  • OpenHands: connecting to an OpenAI-compatible proxy (openai/ prefix on a custom model name): https://docs.openhands.dev/openhands/usage/llms/openai-llms

Related: Own or rent a coding model · Serving open-weight models · Running local coding agents · vLLM inference deployment recipe · Tenant cache isolation · Inference QoS and admission control · Inference serving SLOs · GenAI observability · LLM evaluation harness · Agent evaluation · Agent sandboxing and isolation · Prompt-injection defense · Multi-LoRA serving · KV-cache OOM runbook · Glossary