Skip to content
Markdown

Cookbook: an OpenRouter-style inference platform on your own GPUs

Scope: the end-to-end solution architecture for the product surface OpenRouter made familiar (one API key, a catalog of models behind POST /v1/chat/completions, per-token billing) built on GPU nodes you operate yourself. It covers the gateway and its admission pipeline, tenant isolation on a shared KV cache, the pool-per-model serving plane, the model registry and its onboarding gate, token metering and settlement, autoscaling, and the SRE layer, in enough detail to implement each part and to know what evidence gates each milestone. It deliberately delegates depth it does not own: engine-on-Kubernetes mechanics to the vLLM deployment recipe, the cache side-channel research and salting internals to tenant cache isolation, scheduling and shedding theory to inference QoS and admission control, SLO construction to inference serving SLOs, fleet sizing to GPU capacity planning, adapter serving to multi-LoRA serving, and model selection to serving open-weight models.

The four Python blocks are executed and asserted with the standard library on Python 3.10+. They run without a GPU because they validate the platform's contracts (capacity arithmetic, salt derivation and allowlist reconstruction, settlement idempotency, the autoscaler control law), not model serving. Everything else (YAML, SQL, PromQL) is a reference template, unexecuted here. Engine specifics were checked against vLLM documentation as of 2026-07: re-verify flag names, metric names, and cache_salt endpoint coverage against the exact engine version you pin.

What it is

One deliberate difference from OpenRouter itself, then the same product surface. OpenRouter is an aggregator: it fronts other providers' endpoints under shared upstream credentials, so its isolation and billing properties are inherited, not owned. This architecture serves from your own engines on your own GPUs, which makes you responsible for every property an aggregator inherits: cross-tenant cache isolation, fair scheduling, honest metering, and capacity. The public audit literature shows why that responsibility is real: CacheProbe used OpenRouter as its case study, and found its default shared-credential mode collapses per-account prompt-cache isolation across the upstream providers tested (OpenRouter's BYOK mode, which forwards each user's own provider credentials, was shown to restore isolation).2

The system is six cooperating parts:

  1. An OpenAI-compatible gateway, stateless and horizontally replicated, that owns policy: authentication, admission (quota before balance before routing), tenant isolation stamping, streaming relay, and usage capture. It never re-implements inference semantics; the engine already speaks the OpenAI wire format.
  2. A serving plane of engine pools, one pool per (model, tier, hardware class), each pool a set of identical vLLM replicas with a pinned image, a validated launch shape, and a metrics sidecar that splits observability from the inference port.
  3. A model registry that is the single source of truth for what exists, what it costs, and how it launches: name, upstream revision hash, weight digests, serving profile, pricing, lifecycle state. /v1/models renders from it; pools reference it; nothing ships without passing its onboarding gate.
  4. A metering and settlement path that turns every request (including aborted streams) into exactly one usage record, delivered to billing effectively once, with an arrears ledger so a bill that exceeds a drained balance is recorded rather than lost.
  5. A two-layer autoscaler: pool replicas track engine queue depth under a damped control law; node capacity follows pending GPU pods.
  6. An SRE layer with per-pool, per-tier SLIs, burn-rate alerts, GPU-fleet health telemetry, and a continuously running isolation canary.

Why use it

  • Utilization is the margin. Renting a whole GPU to one tenant leaves it idle between their requests. A shared, metered endpoint amortizes a fixed hourly GPU cost across every tenant's tokens; the capacity arithmetic below shows a single 80 GB GPU serving ~44 concurrent 8K-context sequences of an 8B model at fp16 KV, and ~88 at fp8. The same GPUs sold per token clear more revenue than the same GPUs sold per hour at low duty cycle.
  • Isolation is a sellable feature, not a compliance checkbox. Timing attacks against shared prefix caches are real and peer-reviewed, not theoretical: on a controlled testbed, token-by-token system-prompt reconstruction reaches 89% accuracy at roughly 113 queries per recovered token under favorable conditions, and the same research independently confirms several production services support the underlying exploitable caching behavior.1 A platform that closes the cache channel by construction, and can prove it with a canary, has a differentiator most hosted APIs do not document.
  • The OpenAI surface is the integration point. Every agent framework, SDK, and IDE already speaks it. Owning a compatible endpoint means every existing client is your client, and fine-tuned or in-house models get a serving path with zero client-side work.
  • Every property is testable. The design below insists on evidence gates: measured capacity instead of formula trust, a reproduced timing oracle instead of an isolation claim, kill-tested settlement instead of "exactly-once" on a slide.

When to use it (and when not)

  • Use it when you have (or can rent) a GPU fleet, more than a handful of tenants, and a reason to sell tokens rather than hours: an internal platform team serving many product teams, a regional provider monetizing surplus capacity, or a company productizing its fine-tunes.
  • Do not build it for one tenant or one model. A single-tenant deployment needs none of the isolation, quota, or settlement machinery; use the vLLM deployment recipe directly. A coding-assistant fleet for one engineering org is the smaller sibling of this page: own and run an open-weight coding model.
  • Do not build the aggregator variant casually. Proxying to third-party providers under your credentials re-imports the shared-credential trap CacheProbe documents, and your isolation story stops at your gateway. This page is the own-supply variant only.
  • Scope v1 to dense, single-node models. Most current open-weight flagships are MoE and want expert or data parallel serving shapes; those multiply the serving-plane complexity before the gateway, billing, and isolation core is proven. Launch on capable dense models, add MoE pools as a second phase. This is a product decision to make explicitly, not an implementation detail.

Architecture

The gateway is the only component that holds tenant identity, quotas, and salts; engines only ever see rebuilt, salted, prioritized requests arriving from it. All durable state lives in one control store (Postgres), so gateway replicas stay stateless and any replica can serve any tenant.

flowchart TB
    sdk["Tenant app or agent<br/>any OpenAI SDK"] --> edge["Edge proxy<br/>TLS, host routing, long-stream timeouts"]
    edge --> gw["Inference gateway, stateless xN<br/>auth, admission, salt stamp,<br/>routing, SSE relay, usage capture"]
    gw --> store[("Control store (Postgres)<br/>tenants, trust groups, quotas,<br/>usage journal, registry, arrears")]
    gw -->|"rebuilt request + cache_salt + priority"| poolA["Engine pool: model A<br/>vLLM replicas + metrics sidecar"]
    gw -->|"rebuilt request + cache_salt + priority"| poolB["Engine pool: model B"]
    ctl["Pool controller"] -->|"reconciles, scales replicas"| poolA
    ctl --> poolB
    ctl -->|"pending GPU pods"| nodes["Node autoscaler<br/>provisions GPU nodes"]
    poolA -->|"weights at startup"| obj["Object store weights mirror<br/>digest-pinned, node NVMe cache"]
    gw -->|"batched idempotent usage records"| bill["Billing service<br/>balance, settlement, arrears"]
    prom["Metrics stack"] -.->|"scrape sidecar port only"| poolA

The request lifecycle, end to end:

sequenceDiagram
    participant C as Client SDK
    participant G as Gateway
    participant B as Billing
    participant V as Engine replica

    C->>G: POST /v1/chat/completions stream=true
    G->>G: authN, mint server request_id (UUIDv7)
    G->>G: quota gate RPM/TPM/streams (cheap, first)
    G->>B: GetBalance (fresh read)
    B-->>G: balance
    G->>G: resolve model+price, clamp max_tokens, 402 if floor unmet
    G->>G: rebuild request from allowlist, stamp cache_salt + priority
    G->>V: proxied request on the inference port
    V-->>G: SSE deltas with running usage counts
    G-->>C: SSE relay (usage stripped unless client opted in)
    Note over G,V: client abort cancels upstream and the last usage seen settles
    G->>G: checkpoint usage record to journal
    G->>B: batched ingest, idempotent on request_id
    B->>B: deduct to zero, remainder to arrears ledger

How to use it

1. Write the invariants down before any code

Every component below exists to satisfy one of these. Post them where design reviews happen; every shortcut proposal will collide with one of them.

  1. No cross-tenant KV or prefix-cache reuse, ever. Every request carries a server-derived, per-trust-group, unguessable cache_salt folded into the engine's prefix-cache block hash. The tenant can never choose or see it.
  2. Engines are reachable only from the gateway. The inference port admits gateway traffic only; metrics are re-exposed on a separate port by a sidecar, because stock vLLM serves /metrics on the same listener as the OpenAI API and a scraper that can reach that port can also run unsalted inference.
  3. Admission before work. Quota checks run before balance checks, model resolution, or any allocation, so floods are rejected at minimal cost, with a 429 naming the tripped cap.
  4. Reject, never silently degrade. Exhaustion returns a typed error. The platform never swaps the model, truncates context, or downgrades a tier silently.
  5. Every billable token is captured and settled effectively once. At-most-once capture (bounded loss on crash), at-least-once delivery, idempotent apply keyed on a server-minted request id.
  6. Priority is not cache protection. Engine scheduling priority moves queue position; KV eviction remains global LRU. Protection for latency-critical tenants is structural: admission caps and reserved or dedicated pools.
  7. No GPU sharing across trust boundaries. Time-slicing and MPS provide no cross-tenant memory isolation; one GPU serves one engine process. Hardware-partitioned sharing exists only via MIG and only within a documented trust story per multi-tenant GPU security.
  8. Prompt and completion text never persists, not in logs, metrics labels, traces, or billing records; neither does the salt. Enforce with an executable log-content test, not a convention.
  9. No published number without a measurement behind it. Capacity, SLOs, and isolation claims ship with their evidence or not at all.
  10. Reuse your platform primitives; add the minimum. The net-new surface should be confined to the gateway, one pool controller, a registry schema, a usage-ingest contract, and one autoscaling signal.

2. Pin the API contract

The engine already implements the OpenAI wire format; the gateway owns policy and passes controlled extensions through. Version 1 needs exactly four public endpoints:

Endpoint Semantics
POST /v1/chat/completions OpenAI Chat Completions; stream: true yields SSE deltas; stream_options.include_usage honored
POST /v1/completions legacy text completions (vLLM serves both natively)
GET /v1/models the tenant-visible catalog, rendered offline from the registry, deterministic, no engine calls
GET /v1/models/{id} single-model metadata

Auth is a bearer API key resolved to (tenant, key id, scopes); embeddings, batch, and responses APIs are later phases with their own pool classes.

Rebuild requests from an allowlist; never strip a blocklist. The gateway deserializes the client body into a known struct and re-serializes only supported fields into the engine request. This is the only robust defense against a client smuggling cache_salt (or an engine alias of it) through casing tricks or nested extension objects, because the engine's own parser accepts unknown fields silently. One consistent rule for everything else: a reserved server field or an unknown field in the client body is a 400, never a silent drop. Rejection surfaces both probing attempts and honest client bugs (a mistyped parameter name that silent-drop would hide forever). The field policy in one table:

Field Policy Note
messages/prompt, temperature, top_p, stop, stream, seed, response_format, tools, tool_choice, logprobs, logit_bias, penalties pass through verify per engine pin
max_tokens pass through, clamped registry cap and affordability, step 8
n, best_of capped by max_n; reject best_of < n; bill on fanout = max(n, best_of); best_of > 1 settles non-streaming fan-out multiplies cost
stream_options pass through; gateway forces running usage upstream stripped from relay unless the client opted in
cache_salt, priority, request ids reject 400 server-controlled
anything else reject 400 allowlist reconstruction

Errors are OpenAI-shaped JSON: 401 bad key, 403 scope, 404 unknown or unowned model (never 403 for someone else's private adapter, which would confirm its existence), 400 invalid request, 402 insufficient balance, 429 quota with Retry-After and the cap name, 503 pool saturated with Retry-After.

3. Build the admission pipeline in this exact order

Order is load-bearing: cheap, price-free checks run before expensive or price-dependent ones, so a flood spends almost nothing before rejection.

  1. Authenticate and mint. Resolve the key, mint a UUIDv7 request_id. A client-sent request id header is a trace tag only, never the billing key; otherwise a client controls billing dedup and retries become free.
  2. Resolve the tenant: trust group, tier, salt generation, quota profile. One cached read, TTL at most 60 s, push-invalidated on key revocation and salt rotation so revocation is not merely TTL-bounded.
  3. Price-free quota gate: fixed-window RPM, concurrent-streams gauge, and TPM estimated as fanout x max_tokens plus a prompt estimate, enforced per tenant across all their keys against a shared atomic store (Postgres upserts), not per-replica counters. Per-replica counters are wrong the moment a tenant's keys land on two replicas. Accept the fixed-window boundary burst (2x for one window) as documented slack. The prompt estimate needs a real per-model tokenizer in the gateway; a bytes-divided-by-four heuristic mis-admits CJK and code by integer factors.
  4. Balance gate: read the balance fresh from billing on the healthy path. The only exception is an explicit billing-outage degraded mode with a stale-bounded balance and a hard per-tenant grace budget, logged per admission, as an amendment you write down, not a cache you slip in.
  5. Resolve the model: registry lookup for pool, price, caps, tier availability, and adapter ownership.
  6. Price-dependent gate: enforce the monthly budget cap and clamp max_tokens to affordability (step 8 below). Stamp the resolved price onto the pending usage record so settlement and invoices reconstruct without a second price lookup.
  7. Stamp isolation: inject the derived cache_salt and the tier-derived engine priority during allowlist reconstruction.
  8. Route (step 7 of this guide).
  9. Relay and checkpoint: SSE pass-through, abort propagation, incremental usage checkpoints to the journal.
  10. Settle: journal to billing, batched and idempotent (step 8 of this guide).

Shed load before the engine queue grows. The shed watermark reads the same per-pool gauges the autoscaler consumes (vllm:num_requests_waiting, vllm:kv_cache_usage_perc), and shedding engages only when the pool is at max replicas or KV usage is past ~0.9. Tying shed and scale to one signal source stops them fighting; see inference QoS and admission control for per-class shedding order. Shed over-cap tenants first, then the lowest tier at the pool watermark. The watermark state must be shared across gateway replicas: a per-replica watermark admits replicas x past the inflection point.

4. Make cross-tenant cache reuse impossible by construction

This is the differentiating subsystem, and it is grounded in public, reproducible research. The threat model, condensed (the tenant cache isolation page carries the full treatment):

# Threat Vector Answer
T1 Cross-tenant prompt reconstruction prefix-cache timing oracle; ~113 queries per recovered token at 89% accuracy, on a controlled testbed under favorable conditions1 per-trust-group cache_salt
T2 Cache residency probing only whole blocks are cached, so hits confirm block-aligned prefixes same salt
T3 Noisy-neighbor KV eviction a flood of unique prefixes evicts a quiet tenant's warm blocks; LRU has no priority awareness admission caps; reserved/dedicated tiers
T4 Compute noisy neighbor long prompts inflate everyone's latency watermark shedding, tier priority
T5 Shared-credential collapse the aggregator trap2 structurally avoided: own supply; the equivalent duty is T1 at your gateway
T6 Packet-timing on encrypted streams topic inference often above 98% AUPRC (17 of 28 tested models; range 71-100%) from SSE packet sizes and timing3 not closed by cache isolation; offer a padded or non-streaming mode
T7 GPU co-residency time-slicing/MPS share memory without isolation one engine per GPU across trust boundaries
T8 Billing evasion client-set idempotency keys, abort-before-settlement server-minted ids, prompt-inclusive abort billing
T9 cached_tokens as a residency oracle a seat inside a multi-seat account reads exact cached-token counts to confirm a co-worker's warm prefixes default trust group = one API key, not the account
T10 Salt overwrite a writable data-plane path re-salts a victim to an attacker-known value rotation is control-plane-only, audited
T11 Silent salt drop an engine endpoint that ignores unknown fields returns 200 with cross-tenant sharing startup assertion per endpoint plus a live canary

Salting mechanics. Derive, never store: salt = HKDF(master_secret, trust_group_id || generation). The master secret lives in your KMS and is the isolation single point of failure; treat its custody, audit, and rotation as seriously as a signing key. The database stores only an integer generation per trust group; every stateless gateway replica derives the identical salt on demand, and rotation is a control-plane increment that cold-starts exactly one group's cache. Storage-based alternatives fail concretely: a hashed salt cannot be injected (a hash is one-way), and per-replica lazy minting splits one tenant's cache across replicas. The default trust group is the authenticated API key, not the account, because billed cached_tokens inside a shared-salt account is a noise-free residency oracle (T9); account-wide sharing is an explicit opt-in. vLLM folds cache_salt into the first block's hash, and every later block inherits it through the parent-hash chain, so one field isolates the whole prefix tree.4

Why salting and not a dedicated engine per tenant: blanket physical isolation destroys the cross-request cache reuse that makes a shared endpoint cheap, and the measured cost of full isolation is large (SafeKV reports cutting full isolation's extra TTFT overhead from 50.41% to 11.74% on Qwen3-235B-A22B and from 118% to 34.28% on DeepSeek-R1, and improving throughput 1.36x-2.66x over per-user cache isolation across its evaluated workloads).5 Salting preserves intra-tenant hits (an agent re-sending its own system prompt) while making cross-tenant hits impossible. See prompt caching for how providers price exactly this distinction.

The two contract properties, executed and asserted:

# isolation_contract.py -- validated: salt derivation and allowlist reconstruction.
# The two properties every multi-tenant gateway must hold by construction:
# (1) the cache salt is DERIVED per trust group from one master secret, so N stateless
#     replicas agree without storing secrets per tenant, and rotation is an integer bump;
# (2) the engine request is REBUILT from an allowlist, so a client can never smuggle
#     a salt (or any reserved field) past the gateway in any casing or nesting.
# stdlib only.
import base64
import hashlib
import hmac
import uuid
from typing import Any

def hkdf_sha256(key: bytes, info: bytes, length: int = 32) -> bytes:
    # RFC 5869 extract-then-expand, empty salt, single block is enough for 32 bytes.
    prk = hmac.new(b"\x00" * 32, key, hashlib.sha256).digest()
    return hmac.new(prk, info + b"\x01", hashlib.sha256).digest()[:length]

def derive_cache_salt(master_secret: bytes, trust_group_id: str, generation: int) -> str:
    info = f"cache-salt|{trust_group_id}|{generation}".encode()
    return base64.b64encode(hkdf_sha256(master_secret, info)).decode()

# Fields the client may set; everything else in the body is a 400, never a silent drop.
ALLOWED = {"model", "messages", "prompt", "temperature", "top_p", "max_tokens", "stop",
           "stream", "stream_options", "seed", "response_format", "tools", "tool_choice",
           "logprobs", "logit_bias", "n", "best_of"}
RESERVED = {"cache_salt", "priority", "request_id"}    # server-controlled, in any casing

class RejectedRequest(Exception):
    pass

def rebuild_for_engine(client_body: dict[str, Any], salt: str, priority: int) -> dict[str, Any]:
    for key in client_body:
        if key.lower().replace("-", "_") in RESERVED:
            raise RejectedRequest(f"400: reserved server field: {key}")
        if key not in ALLOWED:      # exact match: "Temperature" is a 400, not a silent drop
            raise RejectedRequest(f"400: field not allowed: {key}")
    extra = client_body.get("stream_options") or {}
    if isinstance(extra, dict) and any(k.lower() in RESERVED for k in extra):
        raise RejectedRequest("400: reserved field nested in stream_options")
    engine_req = {k: v for k, v in client_body.items() if k in ALLOWED}
    engine_req["cache_salt"] = salt          # stamped, never client-influenced
    engine_req["priority"] = priority
    engine_req["request_id"] = str(uuid.uuid4())   # server-minted billing key
    return engine_req

MASTER = b"kms-held-master-secret-never-in-env-or-logs"

# --- Replica agreement: two independent derivations (two stateless gateway replicas)
# produce the identical salt for the same trust group, with nothing stored per tenant.
replica_a = derive_cache_salt(MASTER, "key-7f3a", generation=1)
replica_b = derive_cache_salt(MASTER, "key-7f3a", generation=1)
assert replica_a == replica_b

# --- Distinct principals derive distinct salts: this is the structural check the
# continuous canary samples in production (two tenants must never share a salt).
assert derive_cache_salt(MASTER, "key-7f3a", 1) != derive_cache_salt(MASTER, "key-9c1d", 1)

# --- Rotation is a control-plane integer bump: the new salt shares nothing usable
# with the old one, so a suspected leak cold-starts exactly one trust group's cache.
rotated = derive_cache_salt(MASTER, "key-7f3a", generation=2)
assert rotated != replica_a

# --- The salt is not the master secret and does not reveal it: derivation is one-way.
assert base64.b64decode(replica_a) != MASTER and len(base64.b64decode(replica_a)) == 32

# --- Honest client: allowlisted fields pass through, server fields are stamped.
req = rebuild_for_engine({"model": "llama-3.1-70b", "messages": [], "max_tokens": 128},
                         salt=replica_a, priority=10)
assert req["cache_salt"] == replica_a and req["priority"] == 10 and "request_id" in req

# --- Smuggling attempts: exact name, casing games, header-style spelling, nesting.
for hostile in ({"model": "m", "cache_salt": "attacker-chosen"},
                {"model": "m", "CACHE_SALT": "attacker-chosen"},
                {"model": "m", "Cache-Salt": "attacker-chosen"},
                {"model": "m", "priority": 0},
                {"model": "m", "stream_options": {"cache_salt": "attacker-chosen"}},
                {"model": "m", "Temperature": 2.0},   # wrong casing: reject, never drop
                {"model": "m", "made_up_field": 1}):
    try:
        rebuild_for_engine(hostile, salt=replica_a, priority=10)
        raise AssertionError(f"hostile body admitted: {hostile}")
    except RejectedRequest:
        pass

# --- A client-supplied request id is trace-only: the billing key is always a fresh
# server mint, so a "retry" with a reused id can never dedup away a real charge.
r1 = rebuild_for_engine({"model": "m"}, replica_a, 10)
r2 = rebuild_for_engine({"model": "m"}, replica_a, 10)
assert r1["request_id"] != r2["request_id"]

print("isolation contract: all asserts pass")

Isolation tiers productize the same mechanisms. Shared: per-key salt in a shared pool. Reserved: salt plus a guaranteed admission share, with pool sizing that includes the tenant's cache working set (an undersized reservation only protects capacity, not residency: the tenant's own blocks then evict within the reserved region). Dedicated: a tenant-pinned pool on its own GPUs. Stock engines expose no reserved-KV-partition knob, so the honest protections on a shared tier are admission control and physical separation, with provably bounded fairness (a virtual-token-counter scheduler with a proven worst-case gap in service between two continuously-backlogged clients) as the upgrade path at the gateway.6

What you must not claim. Scope every customer-facing statement to cache-channel confidentiality, never "prompt privacy": the packet-timing channel on encrypted streams (T6) infers conversation topic at often above 98% AUPRC (17 of 28 tested models; range 71-100%) against a perfectly isolated engine, and it affects every streaming provider tested.3 The mitigation (padding, token batching, or a non-streaming mode) lives at your gateway; offer it to topic-sensitive tenants from day one if isolation is your sales posture.

5. Shape the serving plane

One pool per (model, tier, hardware class); one pod per engine replica; one engine per GPU set. Realize the pool as a small CRD plus controller if you run Kubernetes, or as an equivalent declarative unit elsewhere. Two design rules matter more than the tooling:

The serving profile is rich, not flat. Real open-weight models need materially different launch shapes, and the differences fail silently when wrong: a missing tool-call parser leaks tool markup into content with no error; an unsupported quantization on older silicon falls back to slow kernels without complaint; an MoE model may serve data-parallel with expert parallelism where a dense model wants plain tensor parallelism (inference parallelism strategies maps the families). So every load-bearing engine flag is a first-class, validated pool field, never an opaque extraArgs escape hatch that hides it from the capacity model and the onboarding gate.

# pool-llama-3-1-70b.yaml -- reference template, unexecuted. One pool per
# (model, tier, hardware class). Field names are yours; the shape is the point.
apiVersion: inference.example.com/v1
kind: ModelPool
metadata:
  name: llama-3-1-70b-shared
spec:
  modelRef: llama-3.1-70b-instruct     # registry key; weights + revision pinned there
  tier: shared                          # shared | reserved | dedicated
  engine:
    image: vllm/vllm-openai@sha256:...  # digest-pinned, upgraded per pool via the gate
    servedModelName: llama-3.1-70b-instruct   # == registry id; gateway never rewrites
    parallelism: { tensor: 4, pipeline: 1, data: 1, expert: false }
    quantization: null                  # silicon-gated; eval-gated at onboarding
    kvCacheDtype: fp8                   # halves KV bytes, doubles the ceiling; eval-gated
    maxModelLen: 8192
    gpuMemoryUtilization: 0.92
    maxNumSeqs: 51                      # from the MEASURED capacity bench, not the formula
    maxNumBatchedTokens: 8192           # the ITL/TTFT lever; first-class, tuned per pool
    toolCallParser: null                # wrong parser leaks tool markup into content
    reasoningParser: null
    speculativeConfig: null             # acceptance rate measured at onboarding
    trustRemoteCode: false              # security-relevant, so first-class
    schedulingPolicy: priority          # enables per-request tier priority
    enablePromptTokensDetails: true     # REQUIRED or prompt_tokens_details is omitted
                                        # and the cached-input discount is unbillable
  probes:
    startupBudgetMinutes: 30            # large weights load slowly; readiness gates routing
  gpu:
    count: 4                            # tensor x pipeline x data, validated by the controller
    interconnect: nvlink                # shards must not straddle the NVLink domain
  replicas: { min: 1, max: 8 }
  scaling: { targetWaitingPerReplica: 5, scaleDownStabilizationSeconds: 300 }
  drain: { maxStreamSeconds: 900 }      # preStop blocks until running requests reach 0

Split the metrics port with a sidecar. Stock vLLM serves /metrics on the same HTTP listener as the OpenAI API, and Kubernetes NetworkPolicy filters on container ports, so a Service port remap separates nothing. Run a tiny sidecar in the engine pod that scrapes localhost:8000/metrics over pod loopback and re-exposes only /metrics on its own port. Then two default-deny NetworkPolicies finish the job: the engine port admits only the gateway; the sidecar port admits only your metrics collector and the pool controller.

# networkpolicy-engine.yaml -- reference template, unexecuted.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: engine-inference-port, namespace: inference }
spec:
  podSelector: { matchLabels: { app: model-pool } }
  policyTypes: [Ingress]
  ingress:
    - from: [{ podSelector: { matchLabels: { app: inference-gateway } } }]
      ports: [{ port: 8000 }]           # OpenAI API + engine /metrics: gateway only
    - from: [{ podSelector: { matchLabels: { app: metrics-collector } } }]
      ports: [{ port: 9410 }]           # sidecar: /metrics only, cannot reach the API

Size pools with KV arithmetic, then distrust it. The concurrency ceiling of a dense-attention engine is KV-pool math, not weight fit, and the two diverge violently. The executed model below reproduces the published worked anchors; the discipline that survives architecture changes (MLA, sparse attention, speculative decoding) is measure maxNumSeqs with a load test at onboarding, because the arithmetic below is dense-attention-specific even though the method is general. Depth in KV cache management and KV cache inference speedup.

# capacity_model.py -- validated: size a pool from KV arithmetic, then distrust it.
# The concurrency ceiling of a dense-attention engine is KV-pool math, not weight fit.
# Every input below is public: model configs from the HF config.json of each model,
# 80 GB HBM per H100-class GPU. Numbers are decimal GB (1e9), matching vendor specs.
from dataclasses import dataclass

GB = 1e9

@dataclass(frozen=True)
class ModelShape:
    name: str
    layers: int
    kv_heads: int
    head_dim: int
    param_bytes: float          # weights footprint at serving precision

@dataclass(frozen=True)
class PoolShape:
    gpus: int                   # tensor_parallel x pipeline_parallel x data_parallel
    hbm_per_gpu: float
    gpu_mem_util: float         # the engine's gpu-memory-utilization knob
    reserve_per_gpu: float      # activations, CUDA graphs, allocator slack
    kv_dtype_bytes: int         # 2 = fp16/bf16 KV, 1 = fp8 KV
    max_model_len: int

def kv_bytes_per_token(m: ModelShape, dtype_bytes: int) -> int:
    # K and V, per layer, per kv-head, per head-dim element.
    return 2 * m.layers * m.kv_heads * m.head_dim * dtype_bytes

def max_concurrent_seqs(m: ModelShape, p: PoolShape) -> int:
    budget = p.gpus * (p.gpu_mem_util * p.hbm_per_gpu - p.reserve_per_gpu) - m.param_bytes
    if budget <= 0:
        return 0                # the model does not even FIT at this shape
    per_seq = kv_bytes_per_token(m, p.kv_dtype_bytes) * p.max_model_len
    return int(budget / per_seq)

# Public config.json values: Qwen3-8B (36 layers, 8 kv heads, head_dim 128, ~8.2B params bf16),
# Llama-3.1-70B-Instruct (80 layers, 8 kv heads, head_dim 128, ~70.55B params bf16).
QWEN3_8B = ModelShape("qwen3-8b", 36, 8, 128, 16.4 * GB)
LLAMA31_70B = ModelShape("llama-3.1-70b", 80, 8, 128, 141.1 * GB)

def h100_pool(gpus: int, kv_dtype_bytes: int = 2, max_len: int = 8192) -> PoolShape:
    return PoolShape(gpus, 80 * GB, 0.92, 4 * GB, kv_dtype_bytes, max_len)

# --- Anchor 1: one H100 serves Qwen3-8B at 8K context at ~44 concurrent sequences
# with fp16 KV, and fp8 KV doubles that. This is why kv-cache dtype is a pool field.
q_fp16 = max_concurrent_seqs(QWEN3_8B, h100_pool(1, kv_dtype_bytes=2))
q_fp8 = max_concurrent_seqs(QWEN3_8B, h100_pool(1, kv_dtype_bytes=1))
assert q_fp16 == 44, q_fp16
assert q_fp8 == 88 and q_fp8 == 2 * q_fp16, q_fp8

# --- Anchor 2: "fits" and "serves" are different claims. Llama-3.1-70B at TP=2 does
# not even fit under this budget (weights exceed the post-reserve pool), while TP=4
# serves ~51 sequences: same weights, 4 GPUs, a >12x swing from KV headroom alone.
l_tp2 = max_concurrent_seqs(LLAMA31_70B, h100_pool(2))
l_tp4 = max_concurrent_seqs(LLAMA31_70B, h100_pool(4))
assert l_tp2 == 0, l_tp2
assert l_tp4 == 51, l_tp4

# --- Context length is a divisor, not a detail: doubling max_model_len halves the
# ceiling (floor rounding aside), which is why the pool pins max-model-len rather
# than serving whatever the model card advertises.
l_tp4_32k = max_concurrent_seqs(LLAMA31_70B, h100_pool(4, max_len=32768))
assert l_tp4_32k == 12 and l_tp4_32k <= l_tp4 // 4 + 1, l_tp4_32k

# --- Monotonicity holds across the whole grid: more GPUs never serves fewer
# sequences, cheaper KV dtype never serves fewer, longer context never serves more.
for m in (QWEN3_8B, LLAMA31_70B):
    for g in (1, 2, 4, 8):
        assert max_concurrent_seqs(m, h100_pool(g)) <= max_concurrent_seqs(m, h100_pool(g + 1))
        assert max_concurrent_seqs(m, h100_pool(g, 2)) <= max_concurrent_seqs(m, h100_pool(g, 1))
        assert max_concurrent_seqs(m, h100_pool(g, 2, 16384)) <= max_concurrent_seqs(m, h100_pool(g, 2, 8192))

# --- Degenerate shapes return 0, never a negative ceiling: a scheduler consuming
# this number must see "cannot serve", not an exception or a nonsense value.
assert max_concurrent_seqs(LLAMA31_70B, h100_pool(1)) == 0        # weights alone overflow
assert max_concurrent_seqs(QWEN3_8B, h100_pool(0)) == 0           # no GPUs, no pool

print("capacity model: all asserts pass",
      {"qwen3-8b fp16": q_fp16, "fp8": q_fp8, "llama70b tp2": l_tp2, "tp4": l_tp4, "tp4@32k": l_tp4_32k})

Placement constraints the scheduler must not violate: a tensor-parallel pod lands with all its GPUs in one NVLink domain (a shard group split across a non-NVLink boundary collapses throughput); small-model pools pack several to a node while a 70B takes a whole node; new replicas prefer nodes that already hold the model's weights in a digest-keyed NVMe cache (a warm node starts in seconds, a cold one pays the full object-store fetch plus load); and rolling updates with maxSurge 1 / maxUnavailable 0 cost a whole spare replica's GPUs per step, so tie them to your warm-node headroom or accept a stalled (safe) rollout on scarce supply. Drain is real work: set terminationGracePeriodSeconds at or above the max stream duration and block the preStop hook until vllm:num_requests_running reaches zero, because plain SIGTERM aborts every in-flight stream.

6. Run models through a registry and an onboarding gate

The registry is the single source of truth: name, upstream repo plus exact revision hash and weight digests (a model card is a mutable branch, and a silent chat-template edit changes tool-call markup under your users), license gate, per-tier availability, pricing, the validated serving profile per hardware class, and lifecycle state (staging, live, deprecated, retired). Weights mirror once from the hub into your object store, digest-verified, and engine pods load from the mirror through the node cache; the serving path has no dependency on the upstream hub.

flowchart LR
    reg["Registry entry<br/>staging: repo + revision SHA,<br/>license, serving profile"] --> gate["Onboarding gate<br/>mirror, launch shape, capacity bench,<br/>tool-call + quant + reasoning checks"]
    gate --> pool["Pool spec applied<br/>Provisioning: nodes added"]
    pool --> warm["Warming<br/>weights fetch, engine load,<br/>startup probe (minutes)"]
    warm --> live["Live<br/>enters the routing table;<br/>alias flip makes it visible"]
    live -.->|"new revision:<br/>same gate, champion vs challenger"| gate
    live -->|"retire: stop admission,<br/>drain, scale to zero"| ret["Retired"]

A staging entry becomes live only after a frozen acceptance gate, in order:

  1. Mirror weights, digest-verified at copy time and at pod load.
  2. Launch-shape validation on the target hardware with the full serving profile, before any generic wrapping.
  3. Capacity bench: TTFT, time-per-output-token, and max-concurrency curves under load; the measured maxNumSeqs and shed watermark ship, not the formula's.
  4. Correctness gates, each a real silent-failure class: quantization eval-delta versus the upstream reference, and quant-versus-silicon compatibility (an unsupported format falls back to slow kernels without erroring; see quantization for inference); tool-call structured output (the parser produces valid tool-call JSON and leaks no markup into content); reasoning-field split for thinking models; speculative decoding acceptance rate if configured; required companion kernels present in the pinned image.
  5. Price and tier set, then the alias flips to live.

Champion/challenger discipline for revisions: the new revision runs its own pool through the same gate, and the registry alias flips atomically only when it wins; rollback is flipping the alias back, no pod churn. Model visibility is a registry flip, never a redeploy. Adding a model that fits an existing deployment shape is a registry entry plus a pool spec, zero gateway code; a genuinely new shape (a new parallelism family, a multimodal DAG per vLLM-Omni) is a controller change. State that boundary honestly rather than promising "zero code for any model." See experiment tracking and model registry for the alias-flip pattern.

7. Route requests without conflating the two hash layers

Routing answers three questions in order: which model, which gateway replica, which engine replica.

  • Model selection is client-explicit in v1: the model field names a registry entry, which resolves to a pool. Semantic routing (picking the model from the prompt) is a real later-phase product surface with a concrete reference design in the vLLM Semantic Router and the cost-quality evidence in LLM request routing; name it now so your logs record what a router would need, and defer it.
  • Gateway-replica selection happens at the edge before authentication, so it cannot hash on tenant identity (the tenant is not resolved yet). Hash on the API key for best-effort cache locality, and make nothing correctness-critical depend on stickiness: quotas, in-flight spend, and watermarks all live in the shared store precisely so that any replica can serve any request.
  • Engine-replica selection is where prefix-cache locality is won or lost. Make tenant affinity primary, not a tiebreak: route each request to its rendezvous-hash endpoint on the trust group id (where its salted prefix blocks live), and spill to the second affine candidate only when the primary's live outstanding count exceeds k x the pool mean (power-of-two-choices bounded to affine candidates). A pure least-loaded tiebreak routes non-stickily under load and collapses the intra-tenant hit rate.
  • Discovery and the load signal must be named, not hand-waved: the gateway watches endpoint readiness for each pool (an endpoint enters the table only when the engine has loaded weights and reports ready, and drops on not-ready immediately), and "outstanding count" is the gateway's own dispatched-but-unfinished counter per endpoint, reconciled against engine-reported vllm:num_requests_running so drift self-corrects.
  • The SSE relay is the hot path: zero-buffer frame forwarding (buffering destroys time-to-first-token), abort propagation in both directions (client disconnect cancels the engine request and frees KV; an upstream error surfaces as an SSE error event, never a silent truncation), bounded per-stream backpressure so one stuck consumer cannot pin an engine slot, and edge-proxy timeouts tuned for streams of ten minutes and longer, validated end to end before launch.

Exact prefix-aware and KV-aware routers (routing to the engine holding the longest cached prefix) exist in the vLLM production-stack router and in disaggregated designs; adopt one only after you measure the hit-rate ceiling of simple tenant affinity under rolling updates, which remap roughly 1/N of tenants per pod replacement. Cross-region routing is its own page: region-replica routing.

8. Meter every token, settle effectively once, and keep an arrears ledger

Metering is where multi-tenant inference platforms quietly lose money or trust. The load-bearing choices:

  • Force running usage onto every stream chunk (vLLM: stream_options: {include_usage: true, continuous_usage_stats: true} injected upstream, or the engine-side force flag), so the last frame seen before an abort carries authoritative token counts including the prompt. Strip usage from the client relay unless the client asked for it.
  • Abort accounting is prompt-inclusive. Prefill is work the engine performed; the prompt cost settles whether or not any output streamed. Without this, "huge prompt, abort before the first token" is free compute forever.
  • The usage journal is a shared table, not a per-replica volume. Stateless replicas checkpoint in-flight usage every few seconds; a leader-elected sweeper drains rows whose owner lease expired; a dead replica loses at most one checkpoint increment.
  • Enable prompt-token details in the engine (--enable-prompt-tokens-details) or the whole prompt_tokens_details object is omitted from the response and your cached-input discount is unbillable.
  • Settlement is effectively-once, stated honestly: at-most-once capture, at-least-once delivery, idempotent apply keyed on the server-minted request id. Billing validates ingested counts against registry caps, because a negative or absurd count would mint credits.
  • When a bill exceeds a drained balance, deduct to zero and book the remainder in a dedicated append-only arrears table, idempotent on request id, drained on the next top-up, with an explicit write-off state. Do not overload your billing system's transaction-type enum with a new value its readers do not know; a closed enum that maps unknown values to a default will silently misreport arrears as something else. The affordability clamp below keeps arrears the exception rather than the norm, and the worst-case exposure per tenant is one window of max_concurrent_streams x fanout x max_tokens x price, provided in-flight spend is held in the shared store rather than per replica.
# settlement_contract.py -- validated: effectively-once billing with an arrears floor.
# The honest decomposition of "exactly-once" for token billing: capture at-most-once
# (a crash loses at most one checkpoint increment), deliver at-least-once (retries),
# apply idempotently (dedup on the server-minted request id). Plus the two abuse
# closures: prompt-inclusive abort accounting and the fanout-aware affordability clamp.
# stdlib only.
from dataclasses import dataclass, field

@dataclass
class UsageRecord:
    request_id: str
    prompt_tokens: int = 0
    completion_tokens: int = 0

@dataclass
class BillingLedger:
    balance: float
    settled: dict[str, float] = field(default_factory=dict)   # request_id -> charged
    arrears: dict[str, float] = field(default_factory=dict)   # request_id -> owed

    def apply(self, rec: UsageRecord, price_in: float, price_out: float) -> str:
        if rec.request_id in self.settled:
            return "duplicate"                     # idempotent apply: retry is a no-op
        if rec.prompt_tokens < 0 or rec.completion_tokens < 0:
            raise ValueError("negative token count")   # a negative count mints credits
        bill = rec.prompt_tokens * price_in + rec.completion_tokens * price_out
        take = min(bill, self.balance)             # balance never goes negative...
        self.balance -= take
        if bill > take:
            self.arrears[rec.request_id] = bill - take   # ...and the remainder is never lost
        self.settled[rec.request_id] = bill
        return "settled"

# --- At-least-once delivery, idempotent apply: redelivering every record changes nothing.
ledger = BillingLedger(balance=10.0)
records = [UsageRecord("req-1", 1000, 500), UsageRecord("req-2", 200, 100)]
for r in records + records + records:              # sweeper retries after lost ACKs
    ledger.apply(r, price_in=1e-3, price_out=2e-3)
assert ledger.settled == {"req-1": 2.0, "req-2": 0.4}
assert abs(ledger.balance - 7.6) < 1e-9 and not ledger.arrears

# --- Abort accounting is prompt-inclusive: prefill was performed even if the client
# disconnects before the first output token, so the prompt settles unconditionally.
# Without this, "huge prompt, abort instantly" is free compute forever.
abort = UsageRecord("req-3", prompt_tokens=120_000, completion_tokens=0)
ledger.apply(abort, 1e-3, 2e-3)
assert ledger.settled["req-3"] == 120.0

# --- The arrears floor: that abort billed 120.0 against 7.6 of balance. Balance
# clamps at zero and the unpaid remainder lands in the ledger, visible and idempotent,
# instead of silently vanishing (or driving the balance negative).
assert ledger.balance == 0.0
assert abs(ledger.arrears["req-3"] - 112.4) < 1e-9
ledger.apply(abort, 1e-3, 2e-3)                    # replay: no double-arrears
assert len(ledger.arrears) == 1

# --- Poisoned record: a negative count would DEDUCT negative, i.e. mint credits.
try:
    ledger.apply(UsageRecord("req-4", -5, 0), 1e-3, 2e-3)
    raise AssertionError("negative token count admitted")
except ValueError:
    pass

# --- Capture bound: mid-stream checkpoints every K tokens mean a gateway crash loses
# at most one increment of completion tokens, never the prompt and never the stream.
K = 64
def last_checkpoint(tokens_streamed: int) -> int:
    return (tokens_streamed // K) * K
for streamed in (0, 1, K - 1, K, K + 1, 5 * K + 17):
    lost = streamed - last_checkpoint(streamed)
    assert 0 <= lost < K, (streamed, lost)

# --- The affordability clamp, fanout-aware. best_of candidates are all generated
# server-side, so cost scales with max(n, best_of), not n. best_of < n is invalid.
def clamp_max_tokens(requested: int, model_cap: int, balance: float, prompt_cost: float,
                     n: int, best_of: int, price_out: float) -> int:
    if best_of < n:
        raise ValueError("best_of must be >= n")
    fanout = max(n, best_of)
    affordable = int((balance - prompt_cost) / (fanout * price_out))
    return max(0, min(requested, model_cap, affordable))

# 10.0 of balance, 1.0 prompt cost, 2e-3 per output token: 4500 affordable at fanout=1.
assert clamp_max_tokens(8192, 8192, 10.0, 1.0, n=1, best_of=1, price_out=2e-3) == 4500
# fanout=4 divides that by 4: the clamp is what bounds worst-case exposure per stream.
assert clamp_max_tokens(8192, 8192, 10.0, 1.0, n=1, best_of=4, price_out=2e-3) == 1125
# The model cap and the request still bind when they are smaller.
assert clamp_max_tokens(256, 8192, 10.0, 1.0, 1, 1, 2e-3) == 256
assert clamp_max_tokens(8192, 2048, 10.0, 1.0, 1, 1, 2e-3) == 2048
# A drained balance clamps to zero (reject with 402 upstream), never negative.
assert clamp_max_tokens(8192, 8192, 0.5, 1.0, 1, 1, 2e-3) == 0
try:
    clamp_max_tokens(8192, 8192, 10.0, 1.0, n=4, best_of=2, price_out=2e-3)
    raise AssertionError("best_of < n admitted")
except ValueError:
    pass

print("settlement contract: all asserts pass")

Price per model is three numbers (input, output, cached input per million tokens) stamped onto the usage record at admission, so a price change affects only requests admitted after it and every invoice reconstructs from records alone. The control-store schema, condensed:

-- inference schema, DDL sketch -- reference template, unexecuted.
-- Identity keys are TEXT (your IdP's subject), never invented UUIDs; only the
-- server-minted request_id is a UUID. Salts are DERIVED, never stored (step 4).
CREATE TABLE tenants        (tenant_id TEXT PRIMARY KEY, tier TEXT NOT NULL,
                             status TEXT NOT NULL DEFAULT 'active');
CREATE TABLE trust_groups   (trust_group_id TEXT PRIMARY KEY,
                             tenant_id TEXT NOT NULL REFERENCES tenants,
                             salt_generation INT NOT NULL DEFAULT 1);
CREATE TABLE api_key_trust  (key_id TEXT PRIMARY KEY,
                             tenant_id TEXT NOT NULL REFERENCES tenants,
                             trust_group_id TEXT NOT NULL REFERENCES trust_groups);
CREATE TABLE quota_profiles (tenant_id TEXT REFERENCES tenants,  -- NULL = plan default
                             rpm INT, tpm BIGINT, max_concurrent_streams INT,
                             max_context_tokens INT, max_n INT,
                             monthly_budget NUMERIC(20,8));
CREATE TABLE models         (name TEXT PRIMARY KEY, lifecycle TEXT, capabilities JSONB);
CREATE TABLE model_revisions(name TEXT, revision TEXT, upstream_ref TEXT,
                             weight_digests JSONB, serving_profile JSONB,
                             PRIMARY KEY (name, revision));
CREATE TABLE model_prices   (name TEXT, pricing_version INT,
                             price_input NUMERIC, price_output NUMERIC,
                             price_cached_input NUMERIC, effective_from TIMESTAMPTZ,
                             PRIMARY KEY (name, pricing_version));
-- shared per-tenant gauges: authoritative on any gateway replica
CREATE TABLE tenant_windows (tenant_id TEXT, window_start BIGINT,   -- floor(now/60s)
                             rpm_used INT DEFAULT 0, tpm_used BIGINT DEFAULT 0,
                             PRIMARY KEY (tenant_id, window_start));
CREATE TABLE tenant_inflight(tenant_id TEXT PRIMARY KEY,
                             streams INT NOT NULL DEFAULT 0,
                             est_spend NUMERIC NOT NULL DEFAULT 0);
CREATE TABLE usage_journal  (request_id UUID PRIMARY KEY, tenant_id TEXT, key_id TEXT,
                             model TEXT, adapter TEXT, prompt_tokens INT,
                             completion_tokens INT, cached_tokens INT, fanout INT,
                             price_snapshot JSONB, tier TEXT, ttft_ms INT,
                             duration_ms INT, prompt_tokens_estimated BOOL DEFAULT false,
                             state TEXT NOT NULL DEFAULT 'open',  -- open|sealed|settled
                             owner_lease TEXT, checkpoint_at TIMESTAMPTZ);
CREATE TABLE arrears        (request_id UUID PRIMARY KEY REFERENCES usage_journal,
                             tenant_id TEXT NOT NULL, amount NUMERIC(20,8) NOT NULL,
                             status TEXT NOT NULL DEFAULT 'owed', -- owed|drained|written_off
                             resolved_at TIMESTAMPTZ);

9. Autoscale on queue depth under a damped control law

Two independent layers. Pool replicas scale on engine queue depth (vllm:num_requests_waiting per replica against a target of ~5), never on GPU utilization, which reads 100% on a healthy saturated engine and says nothing about queueing. Node capacity follows pending GPU pods through whatever cluster autoscaler you run, with a warm-node floor per model class (a single global warm figure cannot bound cold start for both an 8B and a 70B). Scale-down stabilization of at least 300 s, because a model cold start is minutes of node provisioning plus weight loading.

The control law is normative, not an implementation detail, because replica lead time is minutes while the queue moves per second. A naive proportional controller becomes a relaxation oscillator; the executed simulation shows the failure and the fix:

# scaler_control_law.py -- validated: why the damping rules are normative, not taste.
# Replica lead time (node provision + weight load) is minutes while the queue signal
# moves per second. A naive proportional law turns that mismatch into a relaxation
# oscillator: ramp to max, drain the backlog, watch "desired" evaporate, tear down
# mid-burst, rebuild from scratch, and pay a cold start on every lap. The damped law
# counts in-flight supply, caps the step, takes max() of a queue signal and a rate
# signal, and holds scale-down for a stabilization window. stdlib only.
from dataclasses import dataclass, field

SERVICE = 20          # requests one ready replica completes per tick
TARGET = 5            # target waiting-per-replica (the queue-depth signal)
LEAD = 8              # ticks between ordering a replica and it turning Ready
MIN_R, MAX_R = 2, 16
STEP = 2              # damped: max replicas ordered or retired per tick
HOLD = 20             # damped: low-queue ticks required before one down-step

@dataclass
class Fleet:
    ready: int = MIN_R
    pending: list[int] = field(default_factory=list)   # ticks until Ready
    waiting: float = 0.0
    provisioned: int = 0                                # cold starts paid
    history: list[int] = field(default_factory=list)

    def step_world(self, arrivals: float) -> None:
        self.waiting = max(0.0, self.waiting + arrivals - self.ready * SERVICE)
        self.pending = [t - 1 for t in self.pending]
        self.ready += sum(1 for t in self.pending if t <= 0)
        self.pending = [t for t in self.pending if t > 0]
        self.history.append(self.ready)

    def order(self, k: int) -> None:
        self.pending += [LEAD] * max(0, k)
        self.provisioned += max(0, k)

def queue_desired(waiting: float) -> int:
    return max(MIN_R, min(MAX_R, -(-int(waiting) // TARGET)))

def naive_controller(f: Fleet, _arrivals: float) -> None:
    # No in-flight discount, no step cap, no hold: tears down the moment the
    # backlog reads zero, even though the burst is still arriving.
    desired = queue_desired(f.waiting)
    if desired > f.ready:
        f.order(min(desired - f.ready, MAX_R - f.ready - len(f.pending)))
    elif desired < f.ready:
        f.ready = desired

def make_damped_controller():
    ema, low_ticks = 30.0, 0
    def controller(f: Fleet, arrivals: float) -> None:
        nonlocal ema, low_ticks
        ema = 0.3 * arrivals + 0.7 * ema               # observed arrival rate
        rate_floor = -(-int(ema) // int(SERVICE * 0.8))  # keep 20% headroom
        desired = max(queue_desired(f.waiting), min(MAX_R, max(MIN_R, rate_floor)))
        supply = f.ready + len(f.pending)              # in-flight discount
        if desired > supply:
            f.order(min(STEP, desired - supply, MAX_R - supply))
            low_ticks = 0
        elif desired < f.ready:
            low_ticks += 1                             # hold: is this dip real?
            if low_ticks >= HOLD:
                f.ready -= min(STEP, f.ready - desired)
                low_ticks = 0                          # re-arm after each down-step
        else:
            low_ticks = 0
    return controller

def run(controller, ticks: int = 220) -> Fleet:
    f = Fleet()
    for t in range(ticks):
        arrivals = 100.0 if 10 <= t < 80 else 30.0     # one sustained burst
        f.step_world(arrivals)
        controller(f, arrivals)
    return f

def flaps(history: list[int]) -> int:
    deltas = [b - a for a, b in zip(history, history[1:]) if b != a]
    return sum(1 for a, b in zip(deltas, deltas[1:]) if (a > 0) != (b > 0)) + (1 if deltas else 0)

naive = run(naive_controller)
damped = run(make_damped_controller())

# --- Steady-state need is 100/20 = 5 replicas. The naive law overshoots to max,
# drains the backlog, collapses to min MID-BURST (the queue momentarily reads zero,
# so "desired" evaporates), then rebuilds from scratch, over and over.
assert max(naive.history) == MAX_R
assert min(naive.history[30:80]) == MIN_R              # collapsed during the burst
assert flaps(naive.history) >= 6                       # the oscillator signature

# --- The damped law never collapses mid-burst (the rate floor holds while the
# queue reads zero) and changes direction a handful of times, not every drain.
assert min(damped.history[30:80]) >= 5
assert flaps(damped.history) <= 4

# --- Cold starts are the bill for oscillation: every naive lap re-provisions the
# capacity it just tore down. Weight loading makes each of those laps minutes long.
assert naive.provisioned >= 2 * damped.provisioned, (naive.provisioned, damped.provisioned)

# --- Neither controller strands the queue, and both scale back after the burst.
assert naive.waiting == 0.0 and damped.waiting == 0.0
assert naive.history[-1] == MIN_R and damped.history[-1] <= MIN_R + STEP

print("control law: all asserts pass",
      {"naive cold starts": naive.provisioned, "damped": damped.provisioned,
       "naive flaps": flaps(naive.history), "damped flaps": flaps(damped.history)})

On this simulation the naive law pays 84 replica provisions and changes direction 12 times across one 70-tick burst; the damped law pays 14 and changes direction twice. Add the KV-pressure signal (vllm:kv_cache_usage_perc sustained above 0.9, or any preemptions) as a second scale-up trigger combined with max(), never summed. Capacity planning above the autoscaler follows GPU capacity planning: demand forecast over measured per-replica throughput at ~0.65 target utilization with ~20% headroom, N+1 spare, and growth in the order exhaust one GPU, shrink KV bytes, size tensor parallelism by KV headroom, add replicas, and disaggregate prefill from decode last and only when the rate-matching math says so.

10. Gate each milestone on evidence

Ship in phases, each ending on a gate with banked evidence, dates unset until the gate passes:

  • Phase 0, spike: one manually deployed pool. Gates: capacity model within 20% of measured; cache_salt behavior confirmed per served endpoint on the pinned engine (same-salt hit, cross-salt miss, timing-verified) together with the full flag compatibility matrix (KV dtype x attention backend x chunked prefill x prefix caching, priority policy, usage-stream flags, metric names); a ten-minute SSE stream survives the edge path; a billing round-trip prototype including the arrears path.
  • Phase 1, MVP: gateway with the full admission pipeline, pool controller with static replicas, registry with two models, real settlement. Gates: unmodified OpenAI SDK end-to-end green (stream and non-stream, both endpoints); effectively-once settlement proven under kill-testing (crash before checkpoint, journaled-but-not-sent, sent-but-ACK-lost, billing-outage replay); the isolation evidence test in CI per endpoint; seven days of synthetic plus dogfood traffic for an SLO baseline and the intra-tenant cache-hit rate measured under rolling update.
  • Phase 2, elasticity: autoscaling live under the damped law, scale-to-zero for long-tail models with published cold-start numbers, LoRA adapters with ownership checks and controller-owned load/replay (adapters are runtime state that does not survive an engine restart; see multi-LoRA serving). Gate: burst tests hold SLO without oscillation; adapter isolation re-verified (salt and adapter-ID hash interplay, cross-tenant adapter access returns 404).
  • Phase 3, tiers and fairness: reserved and dedicated tiers, weighted-token fairness at the gateway, public SLOs only now, after two phases of measurement.
  • Phase 4, advanced: KV-aware routing if the measured hit-rate ceiling demands it, prefill/decode disaggregation where rate-matching pays, alternate hardware pools behind their own validation playbook, padding for the packet-timing channel.

How to develop with it

  • Keep the gateway a small, auditable service. The admission pipeline is security-critical and hot-path; it wants its own deployable unit with its own scaling and failure domain, not a module inside your management API. The management plane (key CRUD, usage rollups, registry admin) stays a separate, boring service that talks to the same store.
  • Develop against a fake engine. A stub that speaks the OpenAI SSE wire format (including running usage frames, abort behavior, and error frames) lets you test the relay, checkpointing, and settlement without GPUs. The executed blocks above are exactly the contracts such a stub must honor.
  • Pin SDK conformance in CI. Integration tests run unmodified OpenAI SDKs (Python and JS, pinned versions) against a staging gateway: streaming, non-streaming, usage opt-in, aborts, and every typed error. The field-policy matrix from step 2 is the test parameterization, so a new engine pin that changes field behavior fails loudly.
  • Make the log-content test executable. Drive normal, malformed, panic, and abort paths in CI and assert that no prompt text and no salt appears in gateway or engine logs; engine request logging stays disabled in the pool template.
  • Schema migrations belong to the gateway service, one owner, one migration chain; the management plane reads through views or RPCs. Two writers to one schema is how quota tables drift.
  • Treat the registry as code review surface. Price and kill-switch fields are revenue-critical (a zero output price gives inference away; an absurd one makes the affordability clamp reject everything, a self-inflicted outage). Registry changes go through two-person review, delta sanity bounds, and audited rows.

How to run it in production

Per pool and per tier, never fleet-averaged (averages hide a starved class):

SLI Source Note
TTFT vllm:time_to_first_token_seconds histogram; product SLI measured at the gateway engine-side is the diagnostic, gateway-side includes admission and routing
Time per output token vllm:request_time_per_output_token_seconds the decode-side latency SLI
Availability and error ratio gateway 5xx over all, excluding 429/402 engine success counters miss requests that never reached an engine
Saturation vllm:num_requests_waiting, vllm:kv_cache_usage_perc, vllm:num_preemptions_total, per-tenant shed rate preemptions rising means the KV pool is undersized and the engine is silently re-paying prefill
Prefix-cache hit rate vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total a standing SLI, not a launch gate: the cached-input discount is priced off it
Goodput tokens served within latency SLO the north-star efficiency metric; see goodput for AI systems
Isolation health scheduled cross-tenant canary plus a structural salt-derivation sample both are metric series with alert rules, not prose
Billing integrity journal-to-settlement lag, unsettled-record age, ingest failure rate lag p99 over 5 minutes pages

Reference SLO targets, all to-be-validated against your own measurement before any external commitment: TTFT p99 at or under 1 s (interactive), time per output token p99 at or under 50 ms, availability 99.5% over 28 days initially. Construction and burn-rate alerting follow inference serving SLOs, the SLO/SLI catalog, and multi-window burn-rate rules (14.4x and 6x page, 3x and 1x ticket). A sketch of the two most load-bearing rules:

# alert-rules.yaml -- reference template, unexecuted.
groups:
  - name: inference-slo
    rules:
      - alert: TTFTBurnRateFast
        expr: |
          (1 - (sum(rate(vllm:time_to_first_token_seconds_bucket{le="1.0"}[1h])) by (pool)
              / sum(rate(vllm:time_to_first_token_seconds_count[1h])) by (pool)))
            > 14.4 * 0.005
        for: 2m
        labels: { severity: page }
        annotations: { runbook_url: ".../runbook-inference-slo-breach" }
      - alert: KVPreemptionThrash
        expr: rate(vllm:num_preemptions_total[5m]) > 0
        for: 10m
        labels: { severity: page }
        annotations: { runbook_url: ".../runbook-inference-kv-cache-oom" }

Underneath the service SLIs, run a GPU-fleet health layer (DCGM exporter DaemonSet: profiling-class utilization fields, ECC and Xid faults, thermal throttle, PCIe replay), because the SLO-breach runbook's first fork is "rule out the GPU underneath." The exporter has sharp wiring edges (privileged capability for profiling fields, a custom CSV that replaces rather than extends the default set, the kubelet pod-resources socket for pod attribution); the DCGM exporter manifest page carries them, and observability and monitoring the alert catalog. Adopt the OpenTelemetry GenAI semantic conventions for spans and metrics so tooling interoperates, with content-carrying attributes left at their default of off; see GenAI observability with OTel.

The two on-call runbooks to write before launch, not after: inference SLO breach (split prefill-bound from decode-bound, roll back first on deploy correlation, always rule out throttle and ECC underneath) and KV-cache OOM and preemption thrash (distinguish crash-OOM from steady preemption; cordon and drain before mutating a replica). Capacity adds follow the capacity-add runbook: burn in cordoned, verify health telemetry scrapes the new node, then uncordon.

What is never observed, enforced by the CI log test: prompt or completion text anywhere (logs, metric labels, traces, usage records), the salt anywhere, tenant ids on engine-side high-cardinality metrics.

How to maintain it

  • Engine upgrades are per-pool canaries, never in-place. Pin version plus image digest per pool; every bump re-runs the onboarding gate and the compatibility matrix from Phase 0, because parser behavior, flag defaults, and metric names change between engine versions (the KV usage gauge was renamed across a major engine rewrite, and batched-token defaults have drifted; both would silently break SLO rules and capacity models pinned to the old behavior).
  • Model revision bumps are champion/challenger through the same gate, alias-flipped, with the old pool kept warm until the new one has evidence.
  • Salt rotation is the cache-compromise kill switch: control-plane increment, audited, rate-limited, push-invalidated; effect is bounded by cache propagation, which the runbook states honestly. Rotating a group costs that group a cold cache, so an anomaly alert watches for rotation abuse.
  • Master-secret custody is an incident-response artifact: documented rotation procedure, audited access, never in an environment variable, log line, or crash dump. A leak computes every tenant's salt.
  • Price changes are effective-dated registry rows behind change control; in-flight requests keep the price their clamp was computed against.
  • Re-verify the isolation canary and the field-policy matrix on every engine or SDK pin change. The canary is continuous; the matrix re-run is the cheap insurance against an engine quietly accepting a field it used to reject.
  • Watch the boundary between registry data and code. The moment a "new model" needs a controller change (new parallelism family, new pool class), it is a design change with its own review, not a registry row.

Failure modes

Failure Designed behavior Note
Billing service unreachable explicit degraded mode: stale-bounded balance, hard per-tenant grace budget, every grace admission logged; blocked tenants stay blocked from the last snapshot a written amendment to "always read fresh", never a silent cache
Bill exceeds drained balance deduct to zero, remainder to the arrears ledger, tenant blocked at admission never silently lost; never a negative balance
Engine pod crash in-flight requests fail to the client; readiness gates re-admission; preemption alerts fire before OOM in the steady-thrash case client retry is a new server request id
Hot-model flood per-tenant caps trip first, then pool watermark shed with Retry-After, then autoscaling ordered so one tenant cannot trigger fleet-wide shedding
Huge-prompt abort abuse prompt cost settles unconditionally; per-tenant abort rate is an abuse signal closes the free-prefill hole
Cold-start stampede on a new replica single-flight weight fetch per node keyed by digest; readiness only after load the weights cache turns a mirror outage into a scale-out risk, not a serving outage
Object-store mirror down running pods unaffected (weights are node-local); new replicas stall and alert registry mutations blocked
Control store down gateway serves from its last-good in-memory registry and tenant snapshot with bounded staleness; admission continues; mutations blocked quota gauges degrade to per-replica with the honest replicas x exposure bound, logged
Gateway replica loss stateless; journal rows with expired owner leases are swept by the leader loses at most one usage checkpoint increment
Rolling update on scarce GPUs surge pod waits Pending and the rollout stalls safely, or the warm pool absorbs it never violate max-unavailable zero during drain
Slow or stuck SSE client bounded per-stream buffer, then treat as disconnected and abort upstream one consumer cannot pin an engine slot
Metric-name drift on engine upgrade compatibility matrix re-run per pin; SLO rules break loudly in staging, not silently in production treat a metric rename as a breaking change

References

  • vLLM, OpenAI-compatible server documentation: https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html
  • vLLM, automatic prefix caching and block hashing design: https://docs.vllm.ai/en/latest/design/prefix_caching.html
  • vLLM PR #17045, "[Core] Prevent side-channel attacks via cache salting": https://github.com/vllm-project/vllm/pull/17045
  • vLLM RFC #16016, cache-salt design discussion: https://github.com/vllm-project/vllm/issues/16016
  • vLLM, production metrics reference: https://docs.vllm.ai/en/latest/usage/metrics.html
  • vLLM production-stack, router and prefix-aware routing: https://github.com/vllm-project/production-stack
  • Kubernetes Gateway API Inference Extension (InferencePool, endpoint picker): https://github.com/kubernetes-sigs/gateway-api-inference-extension
  • llm-d, Kubernetes-native distributed inference: https://github.com/llm-d/llm-d
  • NVIDIA Dynamo, KV-aware routing and disaggregated serving: https://github.com/ai-dynamo/dynamo
  • Song, Pang, Wang et al., "The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems," IEEE TIFS: https://arxiv.org/abs/2409.20002
  • Fahey, "CacheProbe: Auditing Prompt Cache Isolation in Gateway APIs": https://arxiv.org/pdf/2605.30613
  • McDonald and Bar Or, "Whisper Leak: A Side-Channel Attack on Large Language Models": https://arxiv.org/abs/2511.03675
  • Chu et al., "Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference" (SafeKV): https://arxiv.org/abs/2508.08438
  • Sheng et al., "Fairness in Serving Large Language Models" (VTC), OSDI 2024: https://arxiv.org/abs/2401.00588
  • Krawczyk and Eronen, RFC 5869, "HMAC-based Extract-and-Expand Key Derivation Function (HKDF)": https://www.rfc-editor.org/rfc/rfc5869
  • OpenAI API reference, chat completions and streaming options: https://platform.openai.com/docs/api-reference/chat
  • OpenRouter documentation (the product surface this page reproduces on owned supply): https://openrouter.ai/docs
  • NVIDIA DCGM exporter: https://github.com/NVIDIA/dcgm-exporter
  • OpenTelemetry, GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/

Related: tenant cache isolation covers the side-channel research and salting internals this page builds on; inference QoS and admission control the shedding and priority theory; the vLLM deployment recipe the engine-on-Kubernetes mechanics; inference serving SLOs and the SLO/SLI catalog the SLO construction; KV cache management and KV cache inference speedup the capacity arithmetic; GPU capacity planning fleet sizing; multi-LoRA serving adapter multiplexing; multi-tenant GPU security the co-residency trust model; LLM request routing and the vLLM Semantic Router the model-routing evolution; serving open-weight models model selection; continuous batching internals the scheduler this platform rides on; own and run an open-weight coding model the single-org sibling of this build; and inference serving and optimization the broader serving map.

  1. Song et al., "The Early Bird Catches the Leak" (IEEE TIFS): on the authors' own local testbed, an incremental search recovers a victim's system prompt token by token, averaging 112.72 queries per recovered token at 89.0% accuracy under a fine-tuned attacker predictor, a best-case condition; the same paper reports 5,231 queries per token at only 69% accuracy with a generic predictor. Separately, the paper confirms several production services support the exploitable caching behavior and demonstrates a distinct real attack against one production API (inferring whether a document was uploaded, not prompt reconstruction) at 89% accuracy. https://arxiv.org/abs/2409.20002 ↩↩

  2. Fahey, "CacheProbe": found OpenRouter's default (shared-credential) mode leaks cross-account cache signals across all three upstream providers tested, from a 100% cache-hit rate on Groq down to a low single-digit percentage the paper calls minor on OpenAI; OpenRouter's BYOK mode was shown to restore isolation. https://arxiv.org/pdf/2605.30613 ↩↩

  3. McDonald and Bar Or, "Whisper Leak": packet-size and timing metadata on encrypted streaming responses classify conversation topic with "near-perfect classification (often >98% AUPRC)" across 28 models from major providers; padding, token batching, and packet injection each reduce but do not eliminate the leak. https://arxiv.org/abs/2511.03675 ↩↩

  4. vLLM PR #17045, "[Core] Prevent side-channel attacks via cache salting," merged April 30, 2025: a request-level cache_salt folded into the first prefix block's hash, inherited by all descendant blocks. https://github.com/vllm-project/vllm/pull/17045 ↩

  5. Chu et al., "SafeKV": selective isolation versus complete per-request isolation cuts the extra TTFT overhead from 50.41% to 11.74% on Qwen3-235B-A22B and from 118% to 34.28% on DeepSeek-R1 (multi-turn workload), and improves throughput 1.36x-2.66x over per-user cache isolation across evaluated workloads, quantifying the cost of over-isolating. https://arxiv.org/abs/2508.08438 ↩

  6. Sheng et al., "Fairness in Serving Large Language Models," OSDI 2024: the Virtual Token Counter scheduler bounds the absolute gap in weighted-token service between any two continuously-backlogged clients, a guarantee the paper proves is within 2x of the best any work-conserving scheduler can achieve (not a claim that one client's service is capped at 2x another's); the reference design for gateway-level fairness beyond static caps. https://arxiv.org/abs/2401.00588 ↩