Markdown

Inference system map: runtimes, scheduling, parallelism, and production serving¶

Scope: reference and decision map for open-weight foundation-model inference on accelerator fleets, from a versioned model artifact and request contract through rendering, scheduling, prefill, decode, KV-cache management, parallel execution, streaming, routing, admission control, autoscaling, and production SLOs. It covers text generation in depth and locates embeddings, rerankers, multimodal encoders, diffusion, and audio in the same serving planes. Focused implementation details live in the linked pages.

The terms in this map are not peers. A model architecture defines the computation and state; a checkpoint stores tensors and metadata; an inference engine loads them and executes optimized kernels; a scheduler forms work from concurrent requests; an API server exposes inference over a protocol; a router or gateway chooses and protects endpoints; an orchestrator places and replaces replicas. Quantization, continuous batching, KV paging and prefix caching, speculative decoding, parallelism, and prefill/decode disaggregation modify different parts of that system. A production deployment selects several of these coordinates at once.

This is a reference page. It defines boundaries, dependencies, and routing, but its examples are real launch and verification paths rather than configuration calculators. The llama.cpp path was compiled and exercised against a live local server. The pinned vLLM and SGLang images and commands were verified against their tagged source and registry manifests; they require a compatible GPU host and were not hardware-executed on the CPU-only validation host.

Focused pages¶

Question	Focused page
Which serving engines and optimization families exist?	Inference serving and optimization
Which efficiency lever matches the measured bottleneck?	LLM inference efficiency and roofline analysis
Which open-weight model and deployment recipe fit the hardware?	Serving open-weight models and the vLLM deployment recipe
How are weights selected, transformed, fused, and loaded?	Model weight loading
How does token-level continuous batching work?	Continuous batching internals
What is stored in the KV cache?	KV-cache fundamentals
How are KV blocks paged, reused, evicted, and compressed?	KV-cache management, token eviction, and KV compression
How should tensor, pipeline, data, and expert parallelism be selected?	Inference parallelism strategies and expert parallelism
When should prefill and decode use separate pools?	Disaggregated inference and rate matching
How does KV state cross a pool boundary?	NIXL KV transfer and centralized KV placement
Which weight, activation, or KV precision should be used?	Quantization for inference and NVFP4 precision
When does a draft model or n-gram drafter help decode?	Speculative decoding and its economics
How are JSON schemas and grammars enforced?	Constrained decoding
How are many adapters served over one base model?	Multi-LoRA serving
How should requests be routed across models and replicas?	LLM request routing and vLLM Semantic Router
How are overload, priorities, and fairness handled?	Inference QoS and admission control
Which latency and availability signals define the service?	Inference serving SLOs and the SLO/SLI catalog
How is a multi-tenant inference product assembled?	OpenRouter-style inference platform and tenant cache isolation
How are failures triaged under live traffic?	Inference SLO breach and KV-cache OOM runbooks
How does inference fit inside online RL?	Post-training system map and rollout fleet sizing

One system, several orthogonal decisions¶

Each axis answers a different engineering question. A product name may implement several axes, but it does not collapse them.

Decision axis	Question	Main choices	Output or state
Workload	What result is requested?	Autoregressive generation, embedding, reranking, classification, speech, diffusion, multimodal generation	Request and response contract
Model architecture	What computation and recurrent state exist?	Dense or MoE; MHA, MQA, GQA, MLA, local or hybrid attention; encoder, decoder, diffusion, SSM	Graph, state shape, supported kernels
Artifact	Which exact model is loaded?	Safetensors checkpoint, packed quantized checkpoint, GGUF, TensorRT engine, base plus LoRA	Versioned tensors and metadata
Runtime	Who owns model execution?	vLLM, SGLang, TensorRT-LLM, Transformers, llama.cpp, another specialized engine	Loaded workers and runtime API
Numerical representation	How are weights, activations, KV, and collectives represented?	BF16, FP16, FP8, INT8, INT4, FP4 and mixed combinations	Capacity, kernels, accuracy envelope
Kernel and compiler	How is each operator executed?	Prefill attention such as FlashAttention; decode or paged-KV attention; GEMM or grouped GEMM; fused MoE; CUDA Graphs; `torch.compile`; TensorRT	Device work and captured graphs
Scheduler	How is concurrent work formed and ordered?	Continuous batching, token budget, chunked prefill, priorities, preemption, deadlines	Waiting and running batches
Cache policy	Which request state is retained and shared?	Paged KV, exact-prefix reuse, tenant salt, eviction, CPU or remote tier, quantized KV	KV block tables and cache index
Intra-replica parallelism	How does one model instance span devices?	Tensor, pipeline, expert, context, attention data, or decode-context parallelism	One logical replica across ranks
Replica parallelism	How does capacity scale out?	Independent data-parallel replicas, cache-aware routing, regional replicas	More throughput and fault domains
Phase placement	Do request stages share workers?	Aggregated; prefill/decode; encoder/prefill/decode; multimodal stage splits	KV or embedding transfer boundary
Protocol and server	How do callers express and receive work?	OpenAI-compatible HTTP, Anthropic-compatible HTTP, gRPC, KServe, engine-native tokens-in/tokens-out	Authenticated stream or response
Fleet routing	Which model, adapter, region, and replica handles the request?	Static pools, semantic routing, SLO and cost routing, cache affinity, least-load	Endpoint selection
Orchestration	How are replicas placed and replaced?	Kubernetes, Slurm, Ray, managed endpoint	Pods or jobs, services, rollout state
Operations	How is safe service maintained?	Admission, load shedding, observability, autoscaling, canaries, rollback, incident response	SLO goodput and controlled failure

The resulting configurations are combinations, not competing labels:

SGLang + BF16 weights + FP8 KV + TP=8 + radix cache describes one runtime replica.
vLLM + AWQ INT4 + TP=4 + DP=2 describes two independent four-GPU replicas, not one eight-GPU cache.
TensorRT-LLM + backend=tensorrt + FP8 + inflight batching + CUDA Graphs describes a built TensorRT-engine path; release v1.2.1 otherwise defaults to backend=pytorch.
Dynamo + SGLang + NIXL + separate prefill/decode pools describes a distributed serving layer around an engine and a KV transport.
OpenAI-compatible gateway + cache-aware routing + vLLM replicas describes a protocol and fleet layer, not another model runtime.
llama.cpp + GGUF Q4_K_M + CPU/GPU offload describes a compact local or edge path, not the normal datacenter scale-out topology.

Concept dependency map¶

The model artifact and request contract enter different sides of the system. They meet inside the runtime after routing and admission.

flowchart TD
    ART["Model artifact and tokenizer"] --> LOAD["Loader, conversion, and warmup"]
    LOAD --> ENGINE["Inference engine"]
    ENGINE --> KERNEL["Kernels and captured graphs"]
    ENGINE --> SCHED["Continuous-batching scheduler"]
    SCHED --> KV["Paged KV and prefix cache"]
    KERNEL --> EXEC["Prefill and decode execution"]
    KV --> EXEC

    REQ["Authenticated request"] --> GATE["Gateway: validate, admit, and choose model pool"]
    GATE --> RENDER["Template, tokenizer, media processor"]
    RENDER --> ROUTE["Cache- and load-aware endpoint selection"]
    ROUTE --> SCHED
    EXEC --> SAMPLE["Sampling, grammar, and parsers"]
    SAMPLE --> STREAM["Detokenize, stream, and account"]

    INTRA["TP, PP, EP, and context parallelism"] -.-> ENGINE
    FLEET["DP replicas and phase pools"] -.-> GATE
    FLEET -.-> EXEC
    OPS["SLOs, metrics, autoscaling, and rollout"] -.-> GATE
    OPS -.-> ENGINE

This map prevents common category errors:

An engine is not a fleet manager. vLLM or SGLang can execute and serve a model; a gateway, Kubernetes controller, or Dynamo layer owns cross-replica routing and replacement.
Paged KV is not prefix caching. Paging allocates request state in blocks. Prefix caching additionally shares completed blocks for an identical token prefix and compatible model context.
Data parallelism is not tensor parallelism. DP adds independent model copies and KV pools. TP shards one copy and pays per-layer communication.
Chunked prefill is not less prefill work. It divides scheduling work so decode can interleave; it does not remove the prompt's attention operations.
Structured output is not semantic validation. A grammar can guarantee syntactic JSON while the fields remain false or unsafe.
An OpenAI-compatible endpoint is not full behavioral equivalence. Chat rendering, tool and reasoning parsers, logprobs, streaming usage, errors, cancellation, and extension fields vary by engine and release.

High-level theory¶

Start with the request and artifact contracts¶

Inference correctness begins before a GPU kernel. The request fixes a model or alias, messages or input media, sampling parameters, output limits, tools or response schema, tenant identity, priority, deadline, and stream mode. The artifact fixes the model and tokenizer revisions, config.json, generation defaults, chat template, tensor files, quantization metadata, optional model code, adapters, and engine-specific parsers. A mismatch can produce fluent but wrong output with no crash.

Pin and promote the composite, not only the tensor files:

Contract element	Failure when omitted
Model repository and immutable revision	A reload may fetch different weights or Python modeling code
Tokenizer files and revision	Token IDs, stop conditions, cache keys, and billable counts can change
Chat template and tool/reasoning parser	Roles, tool calls, or thinking content are rendered or split incorrectly
Quantization config and calibration lineage	The loader may choose an unsupported or slow kernel, or quality may regress
Base and LoRA adapter revisions	The adapter may target missing modules or poison prefix-cache identity
Engine image digest and launch configuration	Defaults, metrics, scheduling, and API behavior can drift
GPU architecture, driver, CUDA, and collective libraries	A valid artifact can fail to load or fall back to a slower path
Held-out quality and load-test result	A file that loads has not yet proved correct or serviceable

Treat trust_remote_code as code execution, not as a model-format switch. Mirror and review required code, pin the revision, run it under the same supply-chain controls as the serving image, and disable outbound network access from the runtime where possible. Safetensors avoids pickle execution for tensor storage, but it does not make remote modeling code, tokenizers, media decoders, or custom kernels trusted.

Normalize, render, and tokenize before scheduling¶

The gateway should authenticate, cap request size and fanout, reject unsupported fields, attach tenant and trace identity, and select a model pool before GPU admission. Rendering then applies the exact chat template, validates multimodal placeholders, resolves stop tokens, and tokenizes. Tool schemas and images can make this CPU path large enough to become the bottleneck. Scale API and tokenizer workers separately when engine support permits, but keep one authoritative rendering contract so replicas do not disagree.

Never reconstruct usage-critical input from visible text after generation. Preserve token IDs, template revision, accepted request fields, selected adapter, cache salt, and sampling configuration with the request record. The gateway's token count is useful for early admission; the engine's accepted token count remains the execution record and must be reconciled for billing.

Remote media adds a separate trust boundary. Restrict schemes and destinations, block link-local and private address ranges unless explicitly required, cap bytes, pixels, frames, and decode time, and fetch through a controlled service. A model server should not become an unrestricted SSRF proxy.

Prefill and decode are regimes, not universal bottleneck labels¶

For an autoregressive decoder, prefill evaluates all prompt tokens and materializes their KV state. It exposes parallel work across tokens and is commonly compute-heavy. Decode evaluates the newest token for every active sequence, reads the weights and prior KV repeatedly, and is commonly limited by memory traffic at low or moderate batch sizes. Large decode batches can become compute-bound; very long prefill can hit attention-memory limits. Measure the target model, batch, context distribution, and GPU instead of treating either label as a law.

The phase distinction produces separate service objectives. Caller-visible SLOs normally start at gateway receipt; admission-based variants need different metric names.

Queue time: admission completion to scheduling start. This reveals overload and priority behavior.
Time to first token (TTFT): gateway receipt to the first streamed token. It includes admission, queueing, rendering, cache lookup, and prefill.
Inter-token latency (ITL): the distribution of gaps between successive streamed tokens, excluding the TTFT interval.
Time per output token (TPOT): for a request with more than one output token, (end-to-end latency - TTFT) / (output tokens - 1).
End-to-end latency: gateway receipt to the final response byte or terminal stream event.
Request goodput: completed requests per second satisfying every declared per-request latency and correctness objective. Report output-token goodput separately. Raw throughput can rise while goodput falls.

Embeddings and rerankers normally execute a bounded encoder or pooling pass and have no autoregressive decode KV. Diffusion repeatedly denoises latent state rather than appending tokens. Multimodal generation adds one or more encoders before or alongside language prefill. They reuse the artifact, runtime, batching, routing, and operations planes, but their state and SLO decomposition need task-specific metrics.

Account for the complete memory ledger¶

Weight fit is only the first gate. A replica consumes:

model weights and scales;
KV or other recurrent state for admitted sequences;
transient activations and logits;
CUDA Graph capture pools and compiler workspaces;
attention, GEMM, MoE, and sampling scratch space;
NCCL, DeepEP, NIXL, or other communication buffers;
tokenizer, multimodal processor, and prefix-index state on host or device;
allocator fragmentation and recovery headroom.

For standard decoder attention, a useful lower bound for KV bytes is 2 * layers * kv_heads * head_dim * cached_tokens * bytes_per_element. The factor two is K plus V. GQA and MQA reduce kv_heads; MLA, sliding-window attention, cross-attention, recurrent or Mamba state, hybrid layers, speculative draft state, and multimodal caches require architecture-specific accounting. Multiply by the number of independent DP replicas only after computing one replica. PP partitions layers and their KV across stages. TP may shard or replicate attention KV depending on the KV-head count, backend, and rank mapping. DP replicas still own independent KV pools.

Set maximum context and maximum concurrent sequences from this ledger, then measure the actual high-water mark. A checkpoint advertising a 128K context does not prove that the deployment has capacity, acceptable TTFT, correct position handling, or retained model quality at 128K.

The runtime binds model code to kernels¶

The loader reads model metadata, selects an implementation, maps checkpoint keys, converts or fuses tensors, chooses quantized layouts, and places ranks. Startup may also compile kernels, autotune shapes, capture CUDA Graphs, allocate KV, and run warmup prompts. Readiness must remain false until all mandatory ranks and the request path are usable.

FlashAttention is an exact attention algorithm rather than an approximate attention rule. It tiles computation to reduce HBM traffic and intermediate storage, but floating-point results need not be bitwise identical. It primarily helps prefill and long-sequence attention. Decode often uses single-query or paged-KV kernels. CUDA Graphs reduce repeated CPU launch overhead for compatible shapes; they add capture time and memory and need eager fallback for uncaptured or dynamic paths. Neither technique reduces model FLOPs, model bytes, or the KV capacity formula by itself.

MoE replaces some dense feed-forward work with token-to-expert routing. Only selected experts compute per token, but all or most expert weights still have to reside across the replica. Expert parallelism adds dispatch and combine communication plus load imbalance. Monitor per-expert token counts, dropped or rerouted tokens, all-to-all time, and hot-rank memory rather than relying on the model's active-parameter count for capacity.

Scheduling turns requests into device work¶

Iteration-level continuous batching admits and retires sequences between iterations. Separately configured scheduler policies enforce token and KV budgets, may chunk and mix prefills with decode, and select priority and preemption behavior. Continuous batching improves utilization under concurrency but does not choose a fair or SLO-safe policy automatically.

The production scheduler contract includes:

maximum queued and running requests, input tokens, output tokens, fanout, and total cached tokens;
class, tenant, age, or deadline priority with starvation bounds;
a chunked-prefill policy that trades TTFT against decode interference;
preemption and recomputation behavior when KV allocation fails;
cancellation that reaches the engine and frees KV, draft, media, and transfer state;
overload behavior that rejects or degrades work before every request misses its SLO.

Closed-loop clients that wait for a response before sending the next request hide queue collapse. Capacity testing must use an open-loop or replayed arrival process with the production input-length, output-length, prefix-reuse, adapter, modality, sampling, and cancellation distributions.

Paged KV, prefix reuse, and tenant identity solve different problems¶

Paged KV allocates fixed-size logical blocks instead of one contiguous maximum-length buffer per request. It reduces fragmentation and permits block sharing, but partial blocks, allocator metadata, and architecture-specific state remain. It does not remove linear KV growth or the attention traffic of decode.

Prefix caching reuses completed KV blocks for an exact tokenized prefix. A safe cache identity covers at least model and tokenizer revisions, token IDs, adapter, multimodal inputs, KV dtype and attention layout, parallel mapping, and a tenant isolation salt. It saves repeated prefill work. It does not speed the unique suffix or decode, and cache-aware routing is needed when DP replicas own independent caches.

Cross-tenant cache sharing can leak prefix membership through timing. Salt or physically isolate cache identity according to the threat model, then verify same-tenant hits and cross-tenant misses on the exact engine release. Metrics endpoints and cache-management APIs also expose sensitive fleet state and should not be public merely because generation requires an API key.

Sampling, constraints, parsing, and streaming finish the request¶

After each model step, the runtime applies logits processors, penalties, and any grammar mask before token selection. A speculative path verifies proposals against the processed target distribution and commits accepted or correction tokens. Detokenization follows commitment. Reasoning and tool parsers then split model-specific markup into API fields. Each layer needs compatibility tests because a correct kernel can still produce an invalid wire response.

Constrained decoding can enforce a choice, regex, JSON schema, or grammar. It guarantees only the accepted syntax. Application invariants still need a validator after generation. Grammar compilation time, tokenizer alignment, mask generation, and fallback behavior belong in latency and failure tests.

Streaming normally uses Server-Sent Events for OpenAI-style HTTP APIs. The gateway must forward frames without buffering, propagate client disconnects upstream, bound slow-consumer buffers, preserve terminal errors, and reconcile usage. Retry only before response commitment unless the application has an explicit resumable protocol. Silently replaying a partially streamed stochastic request can duplicate visible text, tool calls, and charges.

Parallelism maps one logical replica onto hardware¶

Choose parallelism from the state or compute that fails to fit, then follow the physical fabric. Every added axis introduces communication, synchronization, and failure coupling.

Axis	What is partitioned	Main benefit	Communication and placement boundary
Tensor parallelism (TP)	Matrices and attention or feed-forward work within each layer	Makes one layer fit; can lower single-request latency when collectives are fast	Collectives occur throughout every layer; keep TP inside NVLink or another fast, predictable domain where possible
Pipeline parallelism (PP)	Contiguous layer stages	Spans hosts when a model does not fit one node; lowers weight memory per rank	Activations cross stages and bubbles add latency; failures invalidate the whole pipeline
Expert parallelism (EP)	MoE expert weights and token assignments	Makes large sparse expert sets fit and raises expert throughput	Dispatch and combine all-to-all, expert skew, dropped work, and hot ranks dominate
Data parallelism (DP)	Independent full serving replicas	Scales aggregate throughput and isolates failures	No per-token collective across replicas, but routing fragments prefix locality and each replica pays its own weights and KV
Context or decode-context parallelism	One request's context, attention, or KV work across ranks	Makes extreme context lengths or decode state fit	Attention communication can exceed saved compute; semantics and support are engine-specific
Attention data parallelism	Attention weights are replicated while requests are assigned to attention DP ranks	Lets MoE attention and expert phases use different scaling shapes	Requires compatible request-to-rank and expert-parallel mappings
Hybrid expert tensor parallelism	Experts are distributed with EP and each expert is further sharded with TP	Fits or accelerates large experts when EP alone is insufficient	Adds both token dispatch and tensor collectives with a larger failure unit

For a shape written TP=4, PP=2, DP=3, one replica spans eight GPUs and there are three replicas, for 24 GPUs total. Capacity is three times the measured eight-GPU replica capacity. It is not one 24-GPU KV pool. Network design should keep frequent TP and EP traffic on the fastest links, place PP boundaries across the next tier only when needed, and use ordinary service routing across DP replicas.

Do not assume more shards mean faster service. Small models often lose latency to collectives; PP can reduce throughput through bubbles; EP can stall on one popular expert. Establish a one-GPU or one-node baseline, then add the minimum axis required by capacity or a measured latency target.

Disaggregation adds a state-transfer system¶

Aggregated P/D serving runs prefill and decode in the same execution worker or pool. Rendering and media preprocessing may still run in separate API or preprocessing processes. Prefill/decode disaggregation sends a request to a prefill worker, transfers its KV state and metadata, then resumes on a decode worker. Multimodal systems may also separate encoders; diffusion and omni-model systems may split additional stages.

Separate pools can isolate phase interference and use different parallelism, GPU types, and replica ratios. They also add routing, KV serialization, transport registration, layout compatibility, backpressure, cancellation cleanup, and another failure boundary. A first-order break-even check is:

transfer_service_time = KV_bytes / effective_link_bandwidth + setup + metadata + layout_conversion

The exposed transfer time is the measured non-overlapped portion on the critical path and lies between zero and the transfer service time. Use disaggregation only when the SLO or throughput benefit from isolation and independent scaling exceeds exposed transfer and coordination cost. Long prompts increase the possible prefill benefit and the KV volume to transfer. Match the prefill-to-decode ratio to measured arrival rate, input length, output length, cache reuse, and service times. A fixed 1P:1D recipe is not a capacity model.

The transfer contract must pin model revision, KV dtype and layout, layer and head partitioning, block size, parallel mapping, request and tenant identity, and ownership. A cancellation or decode failure must release the remote lease and both pools' blocks. A transport that moves bytes correctly can still corrupt inference if the source and destination interpret those bytes under different TP, KV, or model layouts.

Optimization levers attach to different bottlenecks¶

Apply one material change at a time and rerun quality plus the complete load curve. The useful order is usually: make the artifact correct and fit; establish a scheduler and KV baseline; tune limits; then evaluate lower precision, speculative decoding, extra parallelism, caching, or disaggregation against a named bottleneck.

Lever	Pressure it targets	Evidence that it is working	Common counter-cost
Continuous batching	Idle device gaps and fixed-batch head-of-line blocking	More SLO-compliant work at the same arrival trace	Queueing, fairness, and isolated-request latency
Chunked prefill	Long-prefill interference with decode	Lower ITL tail or higher mixed-workload goodput	More scheduling overhead and sometimes higher TTFT
Paged KV	Contiguous reservation and fragmentation	More admitted cached tokens before OOM	Partial blocks and allocator bookkeeping remain
Prefix caching	Repeated exact prefixes	Hit tokens, avoided prefill time, and lower TTFT for matched requests	Cache index cost, eviction, replica affinity, and timing side channels
FlashAttention or specialized attention	Attention HBM traffic and intermediates	Lower prefill latency and memory at the same outputs	Backend and shape compatibility; no reduction in stored KV
CUDA Graphs	CPU launch overhead and short repeated kernels	Higher graph-hit rate with lower decode overhead	Capture memory, startup time, shape buckets, eager fallback
Weight/activation quantization	Weight capacity, bandwidth, or tensor-core throughput	Lower HBM plus faster end-to-end service at acceptable quality	Calibration, repacking, unsupported slow kernels, numerical drift
KV quantization	Per-token state capacity and bandwidth	More concurrency or context within the same SLO	Long-context and generation-quality drift
Speculative decoding	Serial memory-bound decode steps	More accepted tokens per target verification and lower TPOT	Draft cost, rejection, verification, extra state; can lose at high batch
TP, PP, or EP	Per-device fit or one-replica latency	The model fits or target latency improves after communication	Collective traffic, bubbles, imbalance, larger blast radius
DP replicas	Fleet throughput	Higher SLO goodput at the same per-replica envelope	Replicated weight memory, fragmented cache, routing and warmup
P/D disaggregation	Phase interference or independently varying rates	Better TTFT and ITL goodput after transfer cost	KV transport, rate mismatch, more failure states

Speculative decoding deserves a precise boundary. A drafter proposes tokens and the target verifies several positions in one pass. Strict rejection sampling with matching sampling semantics can preserve the target distribution within hardware numerical behavior. At least one token can commit per verification, but the draft and verification overhead can still make performance worse than ordinary decode. Measure accepted tokens per verification, draft time, target time, batch size, and extra memory. Synthetic acceptance or relaxed methods have different quality contracts.

Cross-feature compatibility is part of the release contract¶

Features that pass independently can fail together. Maintain a tested compatibility matrix for the exact engine, model, GPU, and container digest.

Combination	What must be revalidated
Quantized weights + attention or MoE backend	Kernel availability, fallback path, tensor layout, accuracy and actual latency
Quantized KV + maximum context	Long-context quality, cache capacity, transfer layout, prefix reuse
Speculative decoding + structured output	Sampling equivalence, grammar masking, accepted-token accounting, stop conditions
LoRA + prefix cache	Adapter identity in the cache key, adapter paging, mixed-adapter batching
Prefix cache + multiple tenants	Tenant salt or physical isolation, cache-aware routing, timing canaries
CUDA Graphs + dynamic or multimodal shapes	Capture coverage, memory overhead, eager fallback, media processor behavior
DP + cache-aware routing	Load balance versus locality, hot replicas, failure and rebalance behavior
TP/PP/EP + disaggregation	Compatible KV semantics and a tested repartition or layout-conversion contract when mappings differ; conversion cost and byte interpretation at both endpoints
Chunked prefill + priority scheduling	Starvation bounds, TTFT/ITL trade, preemption behavior
Streaming + proxy + billing	No buffering, disconnect propagation, terminal errors, prompt-inclusive accounting

Do not turn a current incompatibility into a timeless statement. Runtime matrices move quickly. Pin the version in the test result and re-run the matrix on every engine, driver, kernel, model, adapter, or quantization upgrade.

Engine and serving-layer selection¶

Select by model support, target hardware, required API semantics, and measured workload. Cross-project throughput claims usually use different models, kernels, lengths, batching, and SLOs and should not substitute for an on-hardware bake-off.

Component	Category and strongest fit	Main operational boundary
vLLM	General-purpose inference engine and OpenAI-compatible server with broad model coverage, paged KV, continuous batching, parallelism, quantization, speculative decoding, and extensible serving	Fast-moving feature combinations and model implementations require a pinned compatibility and quality suite
SGLang	Inference engine and server with RadixAttention, structured generation, broad parallelism, multimodal paths, and large-scale disaggregation support	CLI, kernel, cache, and backend behavior moves quickly; prefix-heavy advantages depend on real reuse and routing
TensorRT-LLM	NVIDIA-specific runtime, kernels, and OpenAI-compatible `trtllm-serve`; release `v1.2.1` defaults to the PyTorch backend, while `backend=tensorrt` serves built engines	Tight hardware and version coupling; conversion/build time and a separate engine-artifact lifecycle apply to the TensorRT backend, not every `trtllm-serve` deployment
Transformers serving and continuous batching	Compatibility-first path close to upstream model code; useful for day-zero model validation and a common training/evaluation codebase	Usually not the final high-throughput fleet choice until its measured envelope meets the SLO
llama.cpp	GGUF runtime and OpenAI-compatible server for CPU, workstation, edge, and partial GPU offload	Different artifact ecosystem and scale model from datacenter-native distributed engines
Hugging Face TGI	Existing deployments with continuous batching, streaming, TP, metrics, and tracing	The upstream project entered maintenance mode and was archived in March 2026; use it for maintained estates, not as the default new engine
NVIDIA Dynamo	Distributed inference framework around vLLM, SGLang, or TensorRT-LLM, adding routing, planning, KV-aware and disaggregated serving, and Kubernetes integration	It is not a replacement model kernel engine; backend pins and distributed feature maturity must be read from the selected release
llm-d / Kubernetes Inference Gateway integrations	Kubernetes-native distributed-serving patterns, endpoint selection, and validated deployment recipes around engines	Adds controllers, CRDs, and routing dependencies; confirm API maturity and upgrade path before platform adoption
Managed endpoints	Provider-owned image, autoscaling, rollout, and service integration	Reduced systems ownership in exchange for provider limits, less kernel control, and another pricing and data boundary

TGI's status illustrates why the category matters: its useful architecture did not become invalid when maintenance stopped, but it is no longer the best default dependency for a new long-lived platform. Likewise, Dynamo can coordinate three engines without replacing their model loaders and kernels.

An engine bake-off should test the same immutable model, tokenizer, template, quantization, GPU topology, input/output trace, sampling, cache state, concurrency, and SLO. Compare correctness first, then TTFT, ITL, end-to-end latency, goodput, HBM high-water mark, startup time, and recovery. Reject any result that silently changes context length, output limits, precision, parser, or quality to win throughput.

Architecture: end-to-end request and lifecycle¶

The request path and model lifecycle meet only after a candidate has passed quality, load, and warmup gates.

flowchart TD
    REG["Registry: model, tokenizer, template, quantization, lineage"] --> STAGE["Stage weights and verify integrity"]
    STAGE --> LOAD["Load ranks, compile, capture, and warm"]
    LOAD --> READY{"Correctness and readiness gate"}
    READY -->|"fail"| QUAR["Quarantine and diagnose"]
    READY -->|"pass"| POOL["Versioned serving pool"]

    CLIENT["Client request or stream"] --> EDGE["TLS, authentication, limits, and trace"]
    EDGE --> ADMIT{"Quota, deadline, and load admission"}
    ADMIT -->|"reject"| ERROR["Typed error and retry guidance"]
    ADMIT -->|"accept"| POOLSEL["Model, adapter, region, and version pool"]
    POOLSEL --> RENDER["Render, tokenize, and process media"]
    RENDER --> ENDPOINT["Cache- and load-aware replica or phase routing"]
    POOL --> ENDPOINT
    ENDPOINT --> SCHED["Continuous-batching scheduler"]
    SCHED --> PREFILL["Prefix lookup and prefill"]
    PREFILL --> KV["KV ownership and optional transfer"]
    SCHED --> DECODE["Decode, sample, and constrain"]
    KV --> DECODE
    DECODE -->|"next iteration"| SCHED
    DECODE -->|"committed token or finish"| OUT["Detokenize, parse, stream, and account"]
    OUT --> CLIENT

    CANCEL["Deadline or disconnect"] -.-> SCHED
    CANCEL -.-> KV
    TELE["Metrics, traces, logs, and GPU telemetry"] -.-> EDGE
    TELE -.-> SCHED
    TELE -.-> PREFILL
    TELE -.-> DECODE

The system has four planes with different owners and scaling signals:

Artifact and lifecycle plane: registry, conversion, integrity, staging, loading, compilation, warmup, canary, draining, and rollback.
Request and policy plane: TLS, authentication, schema validation, quota, routing, cache identity, deadlines, streaming relay, usage, and audit.
Execution and state plane: tokenizer and media processors, scheduler, kernels, parallel ranks, KV, draft state, and transfer.
Operations plane: SLOs, load tests, metrics, traces, GPU and fabric health, autoscaling, capacity, incident response, and change control.

Distributed topology¶

The following topology keeps fleet routing separate from rank-level execution and shows optional phase disaggregation without implying that every deployment needs it.

flowchart LR
    GW["Replicated gateway and admission"] --> POOLSEL["Model and version pool selection"]
    POOLSEL --> RENDER["Render, tokenize, and process media"]
    RENDER --> ROUTER["Cache- and load-aware endpoint router"]

    subgraph POOL["One versioned model pool"]
        subgraph PREFILL["Independent prefill pool"]
            P1["Prefill replica 1"]
            P2["Prefill replica N"]
        end
        XFER["KV transfer and layout contract"]
        subgraph DECODE["Independent decode pool"]
            D1["Decode replica 1"]
            D2["Decode replica M"]
        end
        subgraph AGG["Optional aggregated pool"]
            A1["Aggregated replica"]
        end

        ROUTER --> P1
        ROUTER --> P2
        ROUTER --> D1
        ROUTER --> D2
        ROUTER --> A1
        P1 --> XFER
        P2 --> XFER
        XFER --> D1
        XFER --> D2
    end

    STORE["Weight and cache storage"] -.-> P1
    STORE -.-> P2
    STORE -.-> D1
    STORE -.-> D2
    STORE -.-> A1
    CONTROL["Registry, rollout, autoscaler, and drain control"] -.-> POOL
    OBS["Metrics, traces, logs, GPU and fabric telemetry"] -.-> POOL
    D1 --> GW
    D2 --> GW
    A1 --> GW

Within each box labeled replica, TP, PP, or EP ranks form one failure unit. The router sees replicas or phase endpoints, not individual tensor ranks. A worker loss inside a sharded replica normally removes that replica from service; sending the next token to an arbitrary surviving rank cannot recover its KV or collective state.

Model replica lifecycle¶

Readiness is a state transition, not proof that a process owns a port.

stateDiagram-v2
    [*] --> Staged
    Staged --> Loading: integrity and compatibility pass
    Loading --> Warming: all ranks loaded
    Warming --> Ready: generation and metric gates pass
    Ready --> Draining: rollout, scale-down, or maintenance
    Draining --> Stopped: in-flight work reaches policy boundary
    Loading --> Failed: load, rank, or memory failure
    Warming --> Failed: compile, graph, or correctness failure
    Ready --> Failed: health or output corruption
    Failed --> Loading: bounded retry after remediation
    Stopped --> [*]

Use a cheap liveness probe to decide whether the process is stuck, a readiness probe to decide whether it may receive traffic, and a startup probe or generous initial deadline for weight loading and graph capture. Add a synthetic generation gate during deployment and periodically out of band. Running a generation on every high-frequency readiness probe consumes capacity and can turn the probe itself into an outage.

Runnable serving paths¶

The examples below exercise actual model servers and OpenAI-compatible requests. They are not throughput calculators or mocked handlers. Each artifact is pinned to a release or immutable model revision. Re-resolve container digests when intentionally changing platforms because an OCI index can select different architecture-specific manifests.

Executed control-path proof: build and serve with llama.cpp¶

This path was executed on an x86-64 host without a CUDA device. It proves the artifact, startup, health, model-discovery, chat-generation, usage-accounting, and malformed-request control paths with a real model. It does not validate GPU kernels, GPU memory sizing, or distributed behavior.

The server was built from llama.cpp tag b10069, commit 178a6c44937154dc4c4eff0d166f4a044c4fceba. The model file is SmolLM2-135M-Instruct-Q4_K_M.gguf from Hugging Face revision 9e6855bc4be717fca1ef21360a1db4b29d5c559a, with SHA-256 ed5fa30c487b282ec156c29062f1222e5c20875a944ac98289dbd242e947f747.

git clone --filter=blob:none \
  https://github.com/ggml-org/llama.cpp.git llama.cpp-b10069
cd llama.cpp-b10069
git switch --detach b10069

cmake -S . -B build \
  -DGGML_NATIVE=OFF \
  -DLLAMA_BUILD_TESTS=OFF \
  -DLLAMA_BUILD_EXAMPLES=OFF \
  -DLLAMA_BUILD_SERVER=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j 8

GGUF_REVISION=9e6855bc4be717fca1ef21360a1db4b29d5c559a
GGUF_FILE=SmolLM2-135M-Instruct-Q4_K_M.gguf
GGUF_SHA256=ed5fa30c487b282ec156c29062f1222e5c20875a944ac98289dbd242e947f747

curl --fail --location --output "$GGUF_FILE" \
  "https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/resolve/$GGUF_REVISION/$GGUF_FILE"
printf '%s  %s\n' "$GGUF_SHA256" "$GGUF_FILE" | sha256sum --check -

build/bin/llama-server \
  --model "$GGUF_FILE" \
  --alias unsloth/SmolLM2-135M-Instruct-GGUF:Q4_K_M \
  --host 127.0.0.1 \
  --port 18080 \
  --ctx-size 512 \
  --parallel 2

Run the verifier from a second shell. It rejects a server that merely opens a port, advertises the wrong model, returns a non-chat object, emits no generated tokens, or accepts malformed JSON.

set -euo pipefail

INFERENCE_BASE=http://127.0.0.1:18080
MODEL_ID=unsloth/SmolLM2-135M-Instruct-GGUF:Q4_K_M

HEALTH_JSON=$(curl -fsS "$INFERENCE_BASE/health")
test "$(jq -r '.status' <<<"$HEALTH_JSON")" = ok
printf 'READY status=%s\n' "$(jq -r '.status' <<<"$HEALTH_JSON")"

LOADED_MODEL=$(curl -fsS "$INFERENCE_BASE/v1/models" | jq -r '.data[0].id')
test "$LOADED_MODEL" = "$MODEL_ID"
printf 'MODEL %s\n' "$LOADED_MODEL"

RESPONSE_JSON=$(curl -fsS "$INFERENCE_BASE/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "unsloth/SmolLM2-135M-Instruct-GGUF:Q4_K_M",
    "messages": [{"role": "user", "content": "Reply with exactly: READY"}],
    "temperature": 0,
    "max_tokens": 8
  }')
test "$(jq -r '.object' <<<"$RESPONSE_JSON")" = chat.completion
test "$(jq -r '.usage.completion_tokens' <<<"$RESPONSE_JSON")" -gt 0
printf 'GENERATE object=%s prompt_tokens=%s completion_tokens=%s\n' \
  "$(jq -r '.object' <<<"$RESPONSE_JSON")" \
  "$(jq -r '.usage.prompt_tokens' <<<"$RESPONSE_JSON")" \
  "$(jq -r '.usage.completion_tokens' <<<"$RESPONSE_JSON")"

BAD_BODY=$(mktemp)
trap 'rm -f "$BAD_BODY"' EXIT
BAD_CODE=$(curl -sS -o "$BAD_BODY" -w '%{http_code}' \
  "$INFERENCE_BASE/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  --data-binary '{"messages":[')
test "$BAD_CODE" -ge 400
test "$BAD_CODE" -lt 600
jq -e '.error.message | length > 0' "$BAD_BODY" >/dev/null
printf 'REJECT malformed_json http=%s\n' "$BAD_CODE"
printf 'ALL LIVE SERVER CHECKS PASSED\n'

Executed output:

READY status=ok
MODEL unsloth/SmolLM2-135M-Instruct-GGUF:Q4_K_M
GENERATE object=chat.completion prompt_tokens=37 completion_tokens=8
REJECT malformed_json http=500
ALL LIVE SERVER CHECKS PASSED

The 500 is the observed llama-server behavior at this tag, not a recommended public API contract. A gateway can normalize malformed client input to 400, but conformance tests should retain the backend result so an engine upgrade cannot change failure semantics unnoticed.

GPU reference template: start SGLang and generate¶

The following template starts SGLang v0.5.15.post1 with a public 0.6B checkpoint. The OCI index digest and CLI flags were checked against the release image and tagged source on 2026-07-20. The host used for this page had no NVIDIA device, so this block still requires execution on the target driver and GPU.

set -euo pipefail

export INFERENCE_API_KEY
INFERENCE_API_KEY=$(openssl rand -hex 32)
SGLANG_IMAGE=docker.io/lmsysorg/sglang@sha256:00c53fe4c31bf22d7b37537f28bbdfd924c02de13cdfb4bff7378c9c34d75ab2
MODEL_REVISION=c1899de289a04d12100db370d81485cdf75e47ca

docker volume create hf-model-cache >/dev/null
docker run --detach --name sglang-qwen3 \
  --gpus all \
  --ipc=host \
  --shm-size=16g \
  -p 127.0.0.1:30000:30000 \
  --mount type=volume,src=hf-model-cache,dst=/root/.cache/huggingface \
  "$SGLANG_IMAGE" \
  python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-0.6B \
  --revision "$MODEL_REVISION" \
  --served-model-name qwen3-0.6b \
  --host 0.0.0.0 \
  --port 30000 \
  --api-key "$INFERENCE_API_KEY" \
  --enable-metrics \
  --mem-fraction-static 0.80 \
  --max-running-requests 16

SGLang /health and /health_generate both execute a one-token request by default in this release. Set SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION=false only when a cheap process check is required; /health_generate continues to test generation. Neither route is a substitute for a deployment gate that checks the expected model and response schema.

set -euo pipefail

test -n "$INFERENCE_API_KEY"
INFERENCE_BASE=http://127.0.0.1:30000
DEADLINE=$((SECONDS + 300))

until curl -fsS "$INFERENCE_BASE/health_generate" >/dev/null; do
  if (( SECONDS >= DEADLINE )); then
    docker logs sglang-qwen3
    exit 1
  fi
  sleep 2
done

MODEL=$(curl -fsS "$INFERENCE_BASE/v1/models" \
  -H "Authorization: Bearer $INFERENCE_API_KEY" |
  jq -r '.data[0].id')
test "$MODEL" = qwen3-0.6b

RESPONSE=$(curl -fsS "$INFERENCE_BASE/v1/chat/completions" \
  -H "Authorization: Bearer $INFERENCE_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3-0.6b",
    "messages": [{"role": "user", "content": "Reply with exactly: READY"}],
    "temperature": 0,
    "max_tokens": 16
  }')
test "$(jq -r '.object' <<<"$RESPONSE")" = chat.completion
test "$(jq -r '.usage.completion_tokens' <<<"$RESPONSE")" -gt 0
curl -fsS "$INFERENCE_BASE/metrics" | grep -q '^sglang:'
printf 'SGLang generated %s token(s) with model %s\n' \
  "$(jq -r '.usage.completion_tokens' <<<"$RESPONSE")" "$MODEL"

Remove only this named container after the check:

docker rm -f sglang-qwen3

Production qualification must add the real tokenizer and chat template, context and output boundary cases, multiple sampling modes, tool and structured-output cases when exposed, disconnect cancellation, overload, replica loss, and a trace-shaped load test. SGLang's API is OpenAI-compatible at selected routes; it is not proof of complete OpenAI behavioral equivalence.

GPU reference template: start vLLM with the same model¶

Using the same model revision separates engine behavior from artifact drift. The vLLM image has ENTRYPOINT ["vllm", "serve"], so arguments begin with the model rather than another vllm serve.

set -euo pipefail

export INFERENCE_API_KEY
INFERENCE_API_KEY=$(openssl rand -hex 32)
VLLM_IMAGE=docker.io/vllm/vllm-openai@sha256:e4f88a835143cd22aee2397a26ec6bb80b3a4a6fe0c882bcbc63822904766089
MODEL_REVISION=c1899de289a04d12100db370d81485cdf75e47ca

docker volume create hf-model-cache >/dev/null
docker run --detach --name vllm-qwen3 \
  --gpus all \
  --ipc=host \
  -p 127.0.0.1:8000:8000 \
  --mount type=volume,src=hf-model-cache,dst=/root/.cache/huggingface \
  "$VLLM_IMAGE" \
  Qwen/Qwen3-0.6B \
  --revision "$MODEL_REVISION" \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name qwen3-0.6b \
  --generation-config vllm \
  --api-key "$INFERENCE_API_KEY" \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 16

--generation-config vllm prevents a checkpoint generation_config.json from silently overriding server sampling defaults. The ordinary vLLM server's /health checks engine health but does not generate a token and has no distinct readiness route. Validate /v1/models and run a generation before adding the replica to service.

set -euo pipefail

test -n "$INFERENCE_API_KEY"
INFERENCE_BASE=http://127.0.0.1:8000
DEADLINE=$((SECONDS + 300))

until curl -fsS "$INFERENCE_BASE/health" >/dev/null; do
  if (( SECONDS >= DEADLINE )); then
    docker logs vllm-qwen3
    exit 1
  fi
  sleep 2
done

curl --no-buffer -fsS "$INFERENCE_BASE/v1/chat/completions" \
  -H "Authorization: Bearer $INFERENCE_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3-0.6b",
    "messages": [{"role": "user", "content": "Count from one to four."}],
    "temperature": 0,
    "max_tokens": 32,
    "stream": true
  }'

A client benchmark should run from the matching vLLM release and drive the public route through the same gateway used in production. The tokenizer snapshot below must be staged from Qwen revision c1899de289a04d12100db370d81485cdf75e47ca. The thresholds are example test objectives, not universal SLOs.

set -euo pipefail

test -n "$GATEWAY_API_KEY"
TOKENIZER_SNAPSHOT=/srv/models/qwen3-0.6b-c1899de
test -f "$TOKENIZER_SNAPSHOT/tokenizer.json"

vllm bench serve \
  --backend openai-chat \
  --base-url https://inference.example.net \
  --endpoint /v1/chat/completions \
  --header "Authorization=Bearer $GATEWAY_API_KEY" \
  --model Qwen/Qwen3-0.6B \
  --served-model-name qwen3-0.6b \
  --tokenizer "$TOKENIZER_SNAPSHOT" \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --num-prompts 1000 \
  --num-warmups 16 \
  --request-rate 4 \
  --max-concurrency 32 \
  --goodput ttft:500 tpot:50 \
  --save-result \
  --save-detailed

Run both open-loop arrival tests and trace replay. A closed-loop client that waits for each completion suppresses queue growth and can make an overloaded server appear stable.

Production fleet and policy plane¶

An optimized engine is one component of an inference service. The fleet also needs an admission point, a queue owner, endpoint selection, model lifecycle control, autoscaling, failure isolation, and an observability contract. A replicated HTTP proxy in front of GPU Pods is necessary but insufficient if it assigns a request before it knows queue, KV, adapter, and readiness state.

Admission, queue ownership, and routing¶

Apply policy before committing scarce backend state:

Authenticate the caller and resolve tenant, model alias, adapter, and policy.
Validate schema, maximum body size, prompt and media limits, sampling limits, and supported features.
Attach an immutable request ID, deadline, priority class, and cache-isolation identity.
Reject over-quota or already-expired work before tokenization or queue insertion.
Select a versioned model, adapter, region, and rollout pool.
Render and tokenize the request, process media, and compute the security-scoped prefix identity.
Late-bind to a ready replica or phase endpoint using exact prefix, queue, KV, adapter, and topology evidence.
Propagate cancellation and the remaining deadline to the engine.

A plain Kubernetes Service selects an endpoint without LLM scheduler context. Requests can become trapped behind one replica's long queue while another is idle. A central queue can delay the binding decision, but it becomes a critical availability and fairness component. Bound its decision latency, replicate it, and define whether stale metrics fail open, fail closed, or shed the request.

An InferencePool and Endpoint Picker do not by themselves provide central queueing. With Gateway API Inference Extension Flow Control disabled, which is the current default in release v1.5.0, non-sheddable requests are dispatched to model-server-local queues. Central late-binding queues, strict priority, and same-priority fairness require the flowControl feature gate in the alpha config.x-k8s.io/v1alpha1 EndpointPickerConfig. Qualify that alpha configuration separately from the stable InferencePool API.

The Gateway API Inference Extension provides a stable inference.networking.k8s.io/v1 InferencePool API. The project's released getting-started guide still labels the overall implementation experience experimental, so API stability must not be read as production maturity for every gateway, Endpoint Picker, or upgrade path. This reference manifest assumes the CRD, a conformant gateway, and an Endpoint Picker Service already exist:

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: qwen3-vllm
  namespace: inference
spec:
  targetPorts:
    - number: 8000
  selector:
    matchLabels:
      app.kubernetes.io/name: qwen3-vllm
      app.kubernetes.io/version: v0.25.1
  endpointPickerRef:
    name: qwen3-vllm-epp
    port:
      number: 9002
    failureMode: FailOpen

The v1 API reference defines FailOpen as forwarding to an endpoint selected by the parent when the Endpoint Picker does not respond; FailClose drops the request. FailOpen preserves reachability but loses EPP-owned queueing, priority, fairness, cache affinity, load-aware admission, and overload defense. Authentication, authorization, mandatory quota, and hard request limits must remain upstream so fallback cannot bypass them. Use FailOpen only when parent-selected routing at fallback load is safe, then test the timeout, fallback selection, and recovery behavior.

Routing policy should account for the following state without creating unbounded work:

Decision	Required evidence	Failure guard
Model and version	Resolved alias, capability, region, rollout cohort	Reject unknown versions; never fall through to another model
Adapter	Loaded adapters, load cost, signed adapter identity	Bound dynamic loads; separate tenant and adapter cache keys
Prefix locality	Exact rendered-token hash, cache ownership, reusable length	Salt by security domain; cap locality bias to avoid hotspots
Load	Central and local queue age, running sequences, KV pressure, predicted service	Shed before every replica is saturated
Priority	Authenticated class, deadline, fairness identity	Age low-priority work and cap high-priority share
Topology	Replica health, GPU/rank membership, phase endpoint, transfer reachability	Remove the entire failed sharded replica
Rollout	Version weight, cohort, quality and SLO gates	Keep rollback capacity warm until the gate closes

Fairness requires a declared unit. Round-robin requests are not fair when one tenant sends 64K-token prompts and another sends 256-token prompts. Useful policies meter admitted prompt tokens, reserved output tokens, actual generated tokens, concurrent KV bytes, or measured GPU time. Gateway and engine priority number direction can differ, so translate named classes at one boundary rather than forwarding an unchecked integer.

Capacity and autoscaling¶

Capacity is workload-specific. At minimum, measure request arrivals (\lambda), input and output length distributions, prefix hit rate, sampling and constraint mix, model and quantization, per-phase token rates, service time (S), KV bytes per live sequence, cold-start time, and failure headroom.

Little's Law gives a concurrency check:

[ L = \lambda W ]

where (L) is the average number of requests in the system and (W) includes queueing and service. It does not size token capacity by itself. Aggregated replicas share GPUs across prefill and decode, so size them from full-trace SLO goodput:

[ R_{\text{agg}} \ge \left\lceil \frac{\lambda}{C_{\text{trace,SLO}}} \right\rceil ]

(C_{\text{trace,SLO}}) is the measured per-replica request goodput for the production trace, not peak throughput. For disaggregated pools, calculate phase bounds separately:

[ R_P \ge \left\lceil \frac{\lambda E[T_{\text{in,miss}}]}{C_P} \right\rceil,\qquad R_D \ge \left\lceil \frac{\lambda E[T_{\text{out}}]}{C_D} \right\rceil ]

(C_P) and (C_D) are measured per-replica token capacities under the target SLO. Apply (M_{\text{live KV}}/C_{\text{KV,safe}}) as another replica lower bound to every pool that owns live KV. (M_{\text{live KV}}) includes allocator overhead and fragmentation, while (C_{\text{KV,safe}}) reserves memory for weights, workspaces, graphs, and transient peaks. Add failure, rollout, and forecast headroom after these lower bounds.

Scale on demand indicators that lead saturation:

Signal	Use	Trap
Central queue size and oldest age	Primary scale-up and overload evidence	Missing metrics can look like zero demand
Arrival rate and admitted token rate	Forecast near-term replicas from the measured service curve	Request rate alone ignores length
Local waiting/running sequences	Detect imbalance and scheduler pressure	Aggregation can hide one hot replica
KV utilization, allocation failures, and preemption	Catch memory-limited capacity	High HBM use can be a healthy full cache
TTFT/ITL SLO burn	User-visible backstop	Tail latency reacts after saturation
Pending Pods and GPU provisioning latency	Prevent repeated scale-up and show supply lag	HPA desired replicas are not ready replicas

GPU utilization is a weak primary scaling signal. Decode can be memory-bound at modest compute utilization, prefix-cache hits can lower prefill work without lowering traffic, and an empty queue can coexist with a long-running batch. Keep minimum warm capacity where startup time exceeds the allowed queue delay. Scale-to-zero is appropriate only for traffic that tolerates model download, rank formation, compilation, and warmup.

This autoscaling/v2 reference template applies only when one Deployment Pod is one independently routable model replica. The custom metrics adapter must expose one inference_pending_requests sample per target Pod; HPA computes the average for AverageValue. Multi-Pod TP, PP, or EP replicas require a group-aware workload controller and scale target so an autoscaler cannot add or remove individual ranks. The target and policies require workload-specific calibration:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: qwen3-vllm
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: qwen3-vllm
  minReplicas: 2
  maxReplicas: 16
  metrics:
    - type: Pods
      pods:
        metric:
          name: inference_pending_requests
        target:
          type: AverageValue
          averageValue: "2"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
        - type: Pods
          value: 2
          periodSeconds: 60
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Pods
          value: 1
          periodSeconds: 120

Kubernetes HPA calculates each configured metric and uses the largest desired replica count. Missing metrics can block a scale-down. KEDA's Prometheus scaler requires its query to return one element; ignoreNullValues defaults to true, which can hide a lost metrics path. Set it deliberately and alert on scaler errors. KEDA cooldown governs scale-to-zero behavior, while HPA stabilization and policies govern ordinary scale-down.

SLO and observability contract¶

Define timestamp ownership before publishing a latency number:

Signal	Definition and decision
Availability	Eligible requests received at the caller-visible gateway that reach the documented successful terminal response, divided by eligible requests. Exclude invalid, unauthorized, and caller-cancelled work only by explicit policy; count capacity rejection, routing failure, timeout, and server failure unless a separate acceptance SLO says otherwise
Queue delay	Admission completion to engine scheduling start; break out gateway and engine queues
TTFT	Caller-visible gateway receipt to the first response token. Give admission-based TTFT a different metric name
ITL	Distribution of gaps between successive streamed tokens, excluding the initial TTFT interval
TPOT	For (N_{\text{out}}>1), ((\text{E2E}-\text{TTFT})/(N_{\text{out}}-1)). Report one-token requests separately and give another denominator a distinct metric name
End-to-end latency	Gateway receipt to final byte or terminal stream event
Request goodput	Completed requests per second satisfying every per-request latency and correctness threshold. Name output-token goodput separately and state how eligibility and output length weight its numerator
Cache	Prefix lookups, matched tokens, hit ratio, evictions, KV allocation, fragmentation, offload and transfer bytes
Scheduler	Waiting and running sequences, preemptions, chunked-prefill work, batch token count, priority and rejection reason
Lifecycle	Stage, load, compile, warm, ready, drain and rollback durations by immutable artifact
GPU and fabric	HBM, SM activity, power, clocks, ECC/Xid, PCIe/NVLink/RDMA throughput and errors

Use histograms with buckets around actual SLO thresholds. Counters need bounded labels such as engine release, model alias, version, route, region, status class, and rejection reason. Do not label metrics with raw tenant IDs, request IDs, prompts, adapter paths, or arbitrary error strings. High-cardinality dimensions belong in sampled traces or access logs under the data-retention policy.

Trace the gateway, queue, tokenizer, scheduler, prefill, KV lookup or transfer, decode, and stream relay. A streaming span must finish on final token, disconnect, deadline, or terminal error, not when response headers are written. Link engine spans and rank failures to the request ID without logging prompt or output content by default.

An SLO dashboard should pair user symptoms with saturation and supply:

p50, p95, and p99 TTFT, ITL, and end-to-end latency split by model, version, route, priority, and input/output buckets.
Offered, admitted, rejected, cancelled, completed, and goodput rates.
Central and per-replica queue age, running sequences, KV occupancy, preemptions, and cache reuse.
Ready, warming, draining, failed, desired, and pending replicas with GPU provisioning latency.
Generation errors, empty outputs, parser failures, non-finite logits, repetition, and canary-quality drift.
GPU, rank, interconnect, storage, and Endpoint Picker health.

Rollout, draining, and recovery¶

A safe serving change is a model, tokenizer, template, engine, kernel, driver, CUDA, quantization, scheduler, topology, and policy change set. Changing only one image tag can change several of these.

Resolve and record immutable weights, tokenizer, template, processor, adapter, engine image, configuration, and expected capability manifest.
Verify hashes and format compatibility before allocating a GPU.
Load all ranks, run compile or graph capture, and warm the intended shape buckets.
Run deterministic conformance, numerical drift, structured-output, tool, streaming, cancellation, and boundary tests.
Add the replica to a shadow or zero-weight pool and verify telemetry.
Canary a caller cohort or request percentage. A request percentage is not a token, GPU-time, or cost percentage.
Compare quality, TTFT, ITL, goodput, error, cache, HBM, and cost gates under the same traffic mix.
Increase traffic in bounded steps while retaining rollback capacity.
Stop new assignments before termination, propagate the unready state, and drain existing streams to a documented deadline.
Remove artifacts only after traffic, rollback, retention, and audit windows close.

Automatic retry is allowed only before any response headers or body bytes have been committed to the caller, and only when the request is known idempotent or the original attempt is known not to have been applied. Generation POSTs normally require a durable idempotency and result ledger. After any committed SSE event or body byte, return a terminal stream error instead of replaying on another replica. Bound retries with an absolute deadline, retry budget, jittered backoff, and Retry-After; one layer should own retries. A request ID enables correlation, not durable idempotency by itself.

A PodDisruptionBudget constrains voluntary eviction; it does not protect against node loss, OOM, rank failure, application crash, or every rollout configuration. terminationGracePeriodSeconds is only an upper bound. The proxy, Endpoint Picker, engine, and application must agree on readiness removal, connection draining, maximum request duration, and forced termination.

Run failure drills under offered load:

kill one worker rank and confirm the whole sharded replica leaves routing;
kill a gateway and Endpoint Picker instance, including the FailOpen or FailClose path;
interrupt a stream before and after the first token;
exhaust KV capacity and verify bounded preemption or admission rejection;
make model storage slow or corrupt and verify startup quarantine;
remove a GPU, inject an Xid where the test platform permits, and verify replacement;
lose one RDMA or NVLink path and observe latency, errors, and fallback;
roll back while the new version has long streams in flight.

Security and tenancy¶

Treat model servers as internal data-plane components. Engine API keys generally cover selected routes and are not a complete tenant, policy, or network security layer.

Terminate TLS and enforce identity, authorization, quota, schema, body, token, media, tool, and sampling limits at the gateway.
Expose an explicit route allowlist. Administrative, profiling, adapter-loading, cache-reset, tokenization, debug, and metrics endpoints stay on a protected network.
Apply default-deny NetworkPolicies between gateway, Endpoint Picker, model servers, registry, object storage, and telemetry.
Use read-only immutable images, non-root identities where supported, minimal Linux capabilities, seccomp, scoped ServiceAccounts, and dedicated cache volumes.
Verify downloaded hashes and signed provenance. Do not allow request-controlled model, tokenizer, adapter, media, or filesystem locations.
Partition or salt prefix and adapter caches by security domain. Cross-tenant timing and reuse can leak prompt membership even when tokens are not returned.
Encrypt external traffic and assess rank, collective, KV-transfer, and storage links separately. Engine inter-rank traffic may be unencrypted by default.
Bound outbound fetches to prevent SSRF through media, model, or adapter URLs.
Redact prompts, outputs, credentials, tool arguments, and retrieved documents from default logs and traces. Define retention and deletion for any sampled content.
Bound raw body bytes, declared fanout, output reservation, media count and bytes, and URL policy before expensive preprocessing. After bounded rendering, tokenization, media inspection, and grammar validation, enforce exact input-token, pixel or frame, grammar-state, KV-reservation, and compute budgets before GPU admission.

System dependencies and artifacts¶

The deployment is reproducible only if these inputs are versioned together:

Dependency or artifact	Contract to record	Failure if omitted
Model weights	Repository, immutable revision, file hashes, license, architecture	Silent weight drift or incomplete shards
Tokenizer and processor	Revision, special tokens, normalization, media limits	Different token counts, prompts, or tensor shapes
Chat template and parsers	Template hash, tool format, reasoning and output parser versions	Correct model behind incompatible API behavior
Quantization	Method, calibration data and revision, scale layout, kernel and hardware matrix	Load failure, quality loss, or dequantized fallback
Adapter set	Base-model identity, adapter hashes, rank and target modules, tenant owner	Wrong adapter, unsafe sharing, or cache collision
Engine and kernels	Release, image digest, build flags, attention backend, graph settings	Performance and numerical behavior drift
GPU platform	SKU, memory, MIG mode, driver, firmware, CUDA and collective libraries	Unsupported kernels, collectives, or capacity
Topology	Replica count, TP/PP/EP/DP degrees, placement, NUMA and fabric	Unexpected collectives and failure domains
Runtime policy	Context, batch, scheduler, cache, preemption, sampling and feature flags	Unexplained SLO and output differences
Serving contract	Routes, schema, auth, errors, streaming, cancellation, usage accounting	False OpenAI compatibility and unsafe retries
Fleet policy	Admission, quotas, priorities, routing, rollout, drain and autoscaling	Local optimization with system-wide overload
Validation corpus	Golden prompts, boundary and failure cases, load trace, acceptance gates	An upgrade with no falsifiable success condition

The registry entry should identify the entire deployable bundle, not only the weight directory. Store benchmark and conformance results with the bundle and the exact hardware topology that produced them.

Selection workflow¶

Specify the contract. List models, modalities, adapters, tools, structured outputs, streaming, maximum context, quality bounds, regions, tenancy, and data rules.
Characterize the trace. Measure arrival bursts, input/output distributions, prefix reuse, cancellation, sampling, and feature mix. Avoid averages without distributions.
Set SLOs and rejection policy. Define TTFT, ITL, end-to-end, availability, goodput, deadlines, priority, quotas, and overload responses.
Freeze candidate artifacts. Pin model, tokenizer, template, processor, quantization, engine image, and hardware.
Prove correctness. Compare tokens, logits or accepted tolerance, deterministic generations, tool calls, constraints, Unicode, stop rules, and malformed requests.
Establish one-replica curves. Sweep prompt/output buckets, concurrency, batch-token limits, cache state, and features. Record latency, goodput, HBM, power, and failures.
Choose the smallest parallelism. Prefer independent replicas when one copy fits and meets latency. Add TP, PP, or EP only for fit, latency, or model-architecture evidence.
Evaluate optimizations independently. Quantization, prefix caching, chunking, graphs, speculation, and disaggregation each need quality, memory, latency, and compatibility gates.
Design admission and routing. Decide queue ownership, locality, fairness, deadlines, shedding, failure mode, and cancellation before scaling replicas.
Size steady state and transitions. Include cold start, one failure domain, canary overlap, drains, and supply lag.
Drill failures and rollbacks. Test rank, replica, gateway, router, storage, fabric, metrics, and client disconnect paths under load.
Revalidate every coupled change. Driver, CUDA, engine, kernel, model, template, quantization, topology, and routing upgrades can invalidate earlier results.

Don't-miss checklist¶

API and correctness¶

The public API contract names supported routes and fields instead of claiming complete OpenAI equivalence.
Model, tokenizer, processor, template, parser, quantization, adapter, and engine are immutable and linked.
Golden tests cover empty, one-token, maximum, Unicode, stop, deterministic, stochastic, tool, grammar, streaming, and malformed cases as applicable.
Usage accounting states whether cached, reasoning, speculative, rejected, and cancelled tokens are counted.
The quality gate compares the production engine and precision path with an accepted reference.

Performance and capacity¶

Load tests use the production path, open-loop arrivals, trace replay, warm and cold caches, and realistic length buckets.
TTFT, ITL, TPOT, end-to-end latency, throughput, and goodput have explicit definitions.
HBM includes weights, KV, workspaces, graph pools, adapter state, fragmentation, and transient peaks.
Parallelism and placement match physical GPU, NUMA, NVLink, PCIe, NIC, and failure topology.
Optimization combinations have a versioned compatibility matrix and negative tests.
Capacity retains room for one failure domain, canary overlap, and cold-start lag.

Production control¶

Admission rejects impossible, expired, unsupported, or over-quota work before GPU queueing.
Queue ownership, late binding, priority, fairness unit, locality, and shedding are declared.
Autoscaling uses queue or token demand, has missing-metric behavior, and drains slowly enough for long streams.
Readiness proves model identity and generation during rollout; high-frequency liveness remains cheap.
Streaming preserves no-buffer behavior and propagates cancellation and deadlines.
Retry policy distinguishes before-first-token from after-first-token failures.
Rollout, drain, rollback, Pod disruption, and forced-termination deadlines agree.
Metrics cardinality and prompt/output retention are bounded.
Failure drills cover rank, replica, gateway, router, storage, fabric, metrics, and overload paths.

Security and operations¶

Tenant identity and cache identity cannot be supplied or forged independently.
Administrative and metrics routes are not exposed through the public wildcard.
Model and adapter paths cannot trigger arbitrary outbound fetches.
Network policy, encryption, runtime identity, secret delivery, and artifact provenance are tested.
Alerts connect SLO symptoms to queue, KV, GPU, rank, fabric, supply, and rollout state.
A named owner can disable admission, drain a version, roll back, and preserve evidence during an incident.

Failure modes¶

Symptom	Likely cause	Discriminating check	Corrective direction
TTFT rises while ITL stays flat	Prefill or queue saturation	Split gateway queue, engine queue, and prefill time by length	Admit less long work, chunk prefills, add prefill capacity, or isolate classes
ITL rises while TTFT stays flat	Decode contention or interconnect pressure	Decode batch, per-token gap, HBM bandwidth, collective timeline	Reduce active sequences, fix topology, or add decode capacity
High throughput but low goodput	Batches violate tail SLO	Recompute goodput from per-request thresholds	Tune for the SLO, not aggregate tokens/s
GPUs appear idle while queues grow	Memory-bound decode, routing skew, blocked collectives, or metric mismatch	Per-replica queue/KV, kernel timeline, rank and link health	Rebalance late, correct signal, or repair the stalled replica
One replica is hot	Early binding or excessive cache affinity	Compare central assignments with local queue and prefix hits	Cap locality bias and route from fresh load state
OOM after a traffic-shape change	KV, graph, workspace, adapter, or fragmentation growth	Reconcile the full HBM ledger at failure time	Lower limits, reserve headroom, or separate shape pools
Prefix hit rate is high but latency worsens	Hashing, transfer, lookup, imbalance, or unusable short matches	Count reusable tokens and lookup/transfer time, not request hits	Raise reuse threshold, localize state, or disable the losing path
Speculation lowers throughput	Low acceptance or draft overhead at concurrency	Accepted tokens per verification and target calls per output token	Reduce depth, change method, or disable for that class
Disaggregation regresses TTFT or ITL	KV transfer and extra queue exceed phase benefit	Phase queue, transfer bytes/time, overlap, and retry	Repack or route only requests above measured break-even
Readiness passes but generation fails	Probe checks only process or engine loop	Run model-identity and synthetic generation gate	Quarantine until all ranks and artifact checks pass
Streams truncate during rollout	Readiness, drain, and termination deadlines disagree	Correlate disconnects with endpoint removal and SIGTERM	Stop assignment first and extend bounded drain
Retry duplicates output	Gateway retried after bytes were sent	Trace first-token timestamp and backend attempts	Disable post-commit retry; expose terminal stream error
Scale-up arrives too late	Cold start exceeds queue budget	Stage/load/warm durations versus oldest queue age	Keep warm floor, pre-stage, forecast, or use smaller artifacts
Scale-down causes immediate overload	Long streams or stale metrics ignored	Running sequence age, missing metrics, HPA recommendations	Stabilize and remove one drained replica at a time
Quality changes after an engine upgrade	Template, parser, kernel, precision, sampling, or model code drift	Diff the deployable bundle and golden outputs	Roll back the bundle, then isolate one change
Tenant data affects another tenant	Shared prefix/adapter cache identity	Cross-tenant timing canaries and cache-key audit	Salt, partition, or physically isolate state
Malformed requests return unstable errors	Backend-specific validation semantics	Versioned negative API conformance suite	Normalize at gateway and retain backend diagnostics
Metrics bill or memory explodes	Request, tenant, prompt, or raw error labels	Cardinality by metric and label	Move high-cardinality fields to sampled traces
Endpoint Picker loss changes policy	Unverified fail-open or fail-close implementation	Kill the picker under load and observe assignment	Align manifest, gateway behavior, alerts, and capacity

Open questions and validation¶

Answer these with measured evidence before approving a design:

Which exact workload trace, quality corpus, hardware, and immutable bundle produced each capacity number?
Where is the authoritative queue, and when does a request become committed to a replica or phase endpoint?
Which component owns deadlines, cancellation, priority translation, quotas, and overload rejection?
What is the security-domain key for prefix, KV, adapter, tokenizer, and media caches?
Which parallel ranks share one failure unit, and how is partial rank loss removed from routing?
At what measured prompt length, cache state, and load does disaggregation beat aggregated serving after KV transfer?
Which quantization and speculative paths preserve accepted quality on every deployed GPU architecture?
How do grammar, tool, multimodal, adapter, speculative, and graph features interact at the pinned engine release?
What happens to an admitted stream during client disconnect, gateway loss, replica loss, scale-down, and rollback?
Does missing telemetry freeze scale-down, create false zero demand, or trigger a policy fallback?
How much warm and rollback capacity remains during one node, zone, fabric, or storage failure?
Can a malicious request allocate unbounded tokenizer CPU, KV, grammar state, adapter state, output reservation, or outbound fetches?
Are live SLO metrics measured at the caller-visible gateway and reconciled with engine timings?
Which failure drills have been run under realistic offered load, and where are their traces and recovery times?
What forces a requalification: model, tokenizer, template, adapter, engine, kernel, CUDA, driver, firmware, topology, or policy change?

References¶

Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models. https://www.usenix.org/conference/osdi22/presentation/yu
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention. https://arxiv.org/abs/2309.06180
Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs. https://arxiv.org/abs/2312.07104
Agrawal et al., Sarathi: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. https://arxiv.org/abs/2308.16369
Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. https://arxiv.org/abs/2401.09670
Patel et al., Splitwise: Efficient Generative LLM Inference Using Phase Splitting. https://arxiv.org/abs/2311.18677
Qin et al., Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. https://arxiv.org/abs/2407.00079
Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. https://arxiv.org/abs/2205.14135
Leviathan et al., Fast Inference from Transformers via Speculative Decoding. https://arxiv.org/abs/2211.17192
vLLM, release v0.25.1 and tagged source. https://github.com/vllm-project/vllm/releases/tag/v0.25.1
vLLM, OpenAI-Compatible Server. https://docs.vllm.ai/en/stable/serving/openai_compatible_server/
vLLM, Production Metrics. https://docs.vllm.ai/en/stable/design/metrics/
vLLM, Parallelism and Scaling. https://docs.vllm.ai/en/stable/serving/parallelism_scaling/
vLLM, Security. https://docs.vllm.ai/en/stable/usage/security/
SGLang, release v0.5.15.post1 and tagged source. https://github.com/sgl-project/sglang/releases/tag/v0.5.15.post1
SGLang, Server Arguments. https://docs.sglang.io/advanced_features/server_arguments.html
SGLang, Sending Requests. https://docs.sglang.io/basic_usage/send_request.html
NVIDIA TensorRT-LLM, release v1.2.1. https://github.com/NVIDIA/TensorRT-LLM/releases/tag/v1.2.1
NVIDIA TensorRT-LLM, tagged Quick Start Guide. https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.1/docs/source/quick-start-guide.md
Hugging Face, Text Generation Inference repository and maintenance notice. https://github.com/huggingface/text-generation-inference
ggml-org, llama.cpp tag b10069. https://github.com/ggml-org/llama.cpp/releases/tag/b10069
Hugging Face, unsloth/SmolLM2-135M-Instruct-GGUF at immutable revision 9e6855bc4be717fca1ef21360a1db4b29d5c559a. https://huggingface.co/unsloth/SmolLM2-135M-Instruct-GGUF/tree/9e6855bc4be717fca1ef21360a1db4b29d5c559a
Qwen, Qwen3-0.6B at immutable revision c1899de289a04d12100db370d81485cdf75e47ca. https://huggingface.co/Qwen/Qwen3-0.6B/tree/c1899de289a04d12100db370d81485cdf75e47ca
NVIDIA, Dynamo Overall Architecture. https://docs.nvidia.com/dynamo/design-docs/overall-architecture
NVIDIA, NIXL. https://github.com/ai-dynamo/nixl
Kubernetes Gateway API Inference Extension, InferencePool. https://gateway-api-inference-extension.sigs.k8s.io/api-types/inferencepool/
Kubernetes Gateway API Inference Extension, v1 API Reference. https://gateway-api-inference-extension.sigs.k8s.io/reference/spec/
Kubernetes Gateway API Inference Extension, release v1.5.0 getting-started maturity notice. https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/v1.5.0/site-src/_includes/intro.md
Kubernetes Gateway API Inference Extension, release v1.5.0 Flow Control. https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/v1.5.0/site-src/guides/flow-control.md
Kubernetes Gateway API Inference Extension, release v1.5.0 Priority and Capacity. https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/v1.5.0/site-src/concepts/priority-and-capacity.md
Kubernetes, Horizontal Pod Autoscaling. https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/
Kubernetes, Pod Lifecycle. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
Kubernetes, Disruptions. https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
KEDA, Prometheus scaler. https://keda.sh/docs/latest/scalers/prometheus/
SGLang, release v0.5.15.post1 benchmark TPOT calculation. https://github.com/sgl-project/sglang/blob/v0.5.15.post1/python/sglang/benchmark/serving.py#L1097-L1103
vLLM, Benchmark Serving CLI. https://docs.vllm.ai/en/stable/cli/bench/serve/
Fielding et al., HTTP Semantics, retrying requests. https://www.rfc-editor.org/rfc/rfc9110.html#name-retrying-requests
Kubernetes, Multi-tenancy. https://kubernetes.io/docs/concepts/security/multi-tenancy/