Markdown

Runbook: serving replica or GPU node lost¶

Scope: a GPU node or a serving replica disappears from a live inference fleet. Restore serving capacity without cascading the shortfall into the survivors, and decide what happens to the requests that were in flight when it went.

Run this when an inference replica leaves the fleet unexpectedly: node NotReady, pod evicted or OOM-killed, a rank lost inside a multi-node replica, or a GPU fault under a healthy-looking pod. Severity depends on how much of the pool went with it. The order is: stop routing to it, protect the survivors, recover capacity, then decide about the in-flight requests. Do not start with "Kubernetes will reschedule it".

Commands are reference templates on real APIs. Metric names moved in June 2025, when vllm:gpu_cache_usage_perc was deprecated in favour of vllm:kv_cache_usage_perc. It is both a release and an engine boundary: that change touched V1 only, so at the transition V1 exported both names while V0 exported only the old one, and releases before it export only the old one under either engine. Counters are also documented without the _total suffix the Prometheus client appends at exposition. Scrape /metrics on the engine you actually run rather than trusting a matrix.

This is the availability counterpart to the inference SLO-breach runbook, which hunts a latency cause in a fleet that is still whole. Here the fleet is not whole. Hardware triage of the failed GPU belongs to the GPU fault runbook; the group-level failure semantics of a sharded replica are in multi-node inference replicas; the retry contract the procedure depends on is stated in the inference system map.

The thing that makes this different from a stateless service outage: the replica's KV cache lived in the GPU's memory and is gone. Recovery is not "restart the pod", it is "reload tens or hundreds of gigabytes of weights, rebuild the process group where applicable, re-register, and re-warm", while the surviving replicas carry traffic they were not sized for.

Trigger¶

A node goes NotReady, or a serving pod is CrashLoopBackOff, Evicted, or OOMKilled.
One rank of a multi-node replica exits; the whole group is unavailable even though most pods are Running.
The router's healthy endpoint count drops, or per-replica request rate goes to zero on one endpoint.
An Xid or ECC alert fires on a node that is currently serving (reliability and RAS).
Often paired, within a minute, with rising queue depth and TTFT on every surviving replica. That second symptom is the incident; the first is only its cause.

Pre-checks¶

Confirm it is capacity loss, not a slow replica. A replica that is up and degraded is the SLO-breach path, not this one. Check that the endpoint is actually gone from the router's ready set, not merely slow.
Establish the size of the loss. How many replicas, what fraction of the pool, and is it one failure domain or several? One replica of eight is a capacity event; three of eight in one rack is an outage.
Check for a common cause before treating it as one node. Several replicas leaving inside a minute is a rollout, a node-pool operation, an image pull failure, a full node filesystem, or a fabric event, not four coincidences.
Read the warm floor. How much loaded, registered spare capacity exists right now (GPU pool segmentation sizes it). This number decides whether the next twenty minutes are a non-event or an incident.

Flow¶

stateDiagram-v2
    [*] --> Detect
    Detect --> Derouted: endpoint removed from ready set
    Detect --> Deroute: still receiving traffic
    Deroute --> Derouted: force endpoint out
    Derouted --> Protect: survivors under load
    Protect --> Warm: warm capacity available
    Protect --> Provision: no warm capacity
    Warm --> Inflight
    Provision --> Inflight: accept the lead time
    Inflight --> Verify: retry policy applied
    Verify --> Cause: capacity restored
    Cause --> [*]: node triaged or replaced

Procedure¶

NS=serving
EP=inference-llm            # service / LWS / deployment name

Get it out of the routing set first. Everything else is worse while requests are still being sent to a dead replica. Confirm the endpoint is gone rather than assuming the controller noticed:

# `-o wide` prints addresses, NOT readiness. An endpoint can be listed and already
# derouted. The conditions are what the router acts on, so print them.
kubectl -n $NS get endpointslice -l kubernetes.io/service-name=$EP \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\t"}{.conditions.ready}{"\t"}{.conditions.serving}{"\t"}{.conditions.terminating}{"\n"}{end}'
# If the Service sets publishNotReadyAddresses, not-ready endpoints are routable anyway.
kubectl -n $NS get svc $EP -o jsonpath='{.spec.publishNotReadyAddresses}{"\n"}'
kubectl -n $NS get pods -l app=$EP -o wide
kubectl get nodes -o wide | grep -v ' Ready'

If a dead endpoint is still listed, the readiness signal is the defect and waiting out a probe period tuned for a different failure is the wrong response. Force it out:

# Check FIRST whether the routing label is also the controller's selector.
kubectl -n $NS get deploy,replicaset,leaderworkerset -o \
  custom-columns=KIND:.kind,NAME:.metadata.name,SEL:.spec.selector.matchLabels

If the Service selects on a label the controller also selects on, do not remove it. Stripping app= from a live pod makes the controller stop counting it, so it creates a replacement while the old pod keeps its GPUs. Route on a label the workload controller does not select, for example serving=true, so it can be removed safely:

kubectl -n $NS label pod <pod> serving-          # router-only label, safe to remove
# Last resort, and only once you accept losing its in-flight requests and any
# post-mortem state: forcing deletion does not wait for graceful shutdown.
kubectl -n $NS delete pod <pod> --grace-period=0 --force

Fix the probe afterwards. A dead replica that stays in the ready set is a defect that will recur, and if the fleet has no router-only label, adding one is the durable fix.

For a multi-node replica, the unit is the group. Losing one rank means the whole group must leave the ready set, and a group whose leader went is not repairable in place (multi-node inference replicas).

Protect the survivors before recovering capacity. The pool just lost a fraction of its service rate and is receiving the same arrival rate. Left alone, queue depth grows on every remaining replica and the incident spreads from one replica to all of them. Tighten admission now, at the gateway, not after TTFT has already gone (QoS and admission control):
```
# Read the pressure on what is left before deciding how hard to shed.
for p in $(kubectl -n $NS get pods -l app=$EP -o name); do
  kubectl -n $NS exec $p -- \
    curl -s localhost:8000/metrics | grep -E \
    'vllm:num_requests_waiting|vllm:num_requests_running|vllm:kv_cache_usage_perc|vllm:num_preemptions_total'
done
```
Rising num_requests_waiting with kv_cache_usage_perc near its ceiling and a climbing preemption count is the survivors going into thrash. The demand-surge runbook has the full protect-prioritise-bound-degrade ladder; a capacity loss and a demand spike are the same arithmetic seen from opposite sides.
Decide the in-flight requests now, before recovering capacity. Their KV cache died with the replica, and steps 4 and 5 can run for minutes. The boundary is not negotiable: a request may be retried on another replica only before any bytes have been committed to the caller. After the first streamed token, the honest action is a terminal stream error, not a silent replay that produces a second, different continuation of the same answer. The inference system map states the full contract; do not re-derive it per service.

Where the request triggered an external effect, separate the two retries completely. Re-running inference is cheap and safe; re-running the action is neither. Every proposed effect needs its own operation id and its own ledger so a repeated inference cannot produce a duplicate effect (the action-execution boundary).

Promote warm capacity, do not create it. Warm means already loaded, healthy and registered. Shift traffic onto those replicas; do not scale a deployment, because scaling creates new pods that must pull an image and load weights, which is exactly the lead time the warm floor exists to avoid:
```
# Promote: give the warm pool weight at the router, or release its readiness gate.
kubectl -n $NS annotate --overwrite service/$EP router.example.com/pool-weight-warm="100"
# If promotion is done by label, select the EXACT warm replicas for this model and
# revision. A bare `-l pool=warm` matches every warm pod in the namespace, including
# other models, and needs --overwrite to touch a label that already has a value.
kubectl -n $NS label --overwrite \
  -l pool=warm,app=$EP,model-revision=$REV serving=true
```
Scaling comes next, and its job is to rebuild the warm floor rather than to serve this incident. If there is no warm capacity, you are now paying the full provisioning plus weight-load time and there is no lever that shortens it. Say so explicitly in the incident channel, with a number, so nobody waits for a recovery that is fifteen minutes out believing it is two.
Reschedule onto capacity that can actually host the shape. A replacement needs a complete, qualified, topology-appropriate allocation, not an equal number of free GPUs:
```
kubectl -n $NS describe pod <pending-pod> | sed -n '/Events/,$p'
# Allocatable is the node's static ceiling, not what is free. Subtract what is already
# requested, or read the scheduler's own view.
kubectl describe node <node> | sed -n '/Allocated resources/,/^Events/p'
```
Insufficient nvidia.com/gpu with free GPUs visible in the fleet means fragmentation: the GPUs exist, an admissible group does not (scheduler GPU job pending).
Only now, triage the hardware. The node is out of the serving path, so this is no longer on the critical path:
```
NODE=<the node that failed>        # set it explicitly; an inherited value targets elsewhere
ssh "$NODE" 'nvidia-smi --query-gpu=index,name,ecc.errors.uncorrected.volatile.total,\
clocks_event_reasons.active --format=csv'
ssh "$NODE" 'dmesg -T | grep -i xid | tail'
```
Route by fault class: Xid and ECC to the GPU fault runbook, thermal to the thermal emergency runbook, driver or module to the driver module load failure runbook. Do not return the node to the pool on a clean nvidia-smi alone; it must pass the qualification gate (GPU health gating).
Restore traffic gradually, weighting by measured rate. A replacement replica is ready before it is fast: its prefix cache is empty, so its early requests are all misses and its effective service rate climbs over a warm-up window. Give it an equal share the instant it goes ready and you overload the one replica that cannot yet carry it, while the warm replicas sit under-used. Weight the router by each replica's measured rate and ramp over minutes, watching TTFT on the new endpoints specifically rather than on the fleet average, which hides this entirely.

The model below was executed; the figures in the prose are its output.

# ramp_back.py -- validated: why step 7 restores traffic gradually rather than at once.
# A replacement replica is READY before it is FAST: its prefix cache is empty, so every
# early request is a cache miss and its effective service rate climbs over a warm-up
# window. Routing it an equal share the instant it goes ready is the last mistake of this
# incident, and it is self-inflicted. Three results:
#   (1) equal-share-on-ready overloads the cold replica while the warm ones sit under-used,
#       so the fleet loses goodput it already had;
#   (2) ramping in proportion to each replica's MEASURED rate keeps every replica inside
#       its deadline and finishes the warm-up no later;
#   (3) the effect scales with how cold the replica starts and vanishes when it does not,
#       so this is a claim about cache warm-up and not about routing in general;
#   (4) and routing is not a substitute for admission. If arrivals already require the
#       replacement's warm capacity, no share can conjure it, and shedding has to stay on
#       until the cache is warm. The model was written expecting rate-weighting to hold in
#       every case and it does not, which is the useful part.
# numpy only.
import numpy as np

WARM_RATE = 20.0        # requests/s a warm replica sustains at the target latency
COLD_FRAC = 0.25        # a cold replica starts at this fraction of warm throughput
WARMUP_S = 120          # seconds for the prefix cache to reach steady state
DEADLINE_S = 30.0
WARM_REPLICAS = 3


def rate_at(t, warm_s=WARMUP_S):
    """Effective service rate of the replacement replica, climbing linearly to warm."""
    if warm_s <= 0:
        return WARM_RATE
    return WARM_RATE * (COLD_FRAC + (1 - COLD_FRAC) * min(t, warm_s) / warm_s)


def simulate(policy, seconds=600, warm_s=WARMUP_S, cold_frac=COLD_FRAC, arrival=None):
    """Default arrivals are what the SURVIVING warm replicas can already serve, which is the
    situation after a loss the fleet absorbed: the replacement is headroom, not a
    dependency. Pass `arrival` to model a fleet that is relying on it."""
    global COLD_FRAC
    saved, COLD_FRAC = COLD_FRAC, cold_frac
    if arrival is None:
        arrival = WARM_RATE * WARM_REPLICAS
    queues = [0.0] * (WARM_REPLICAS + 1)          # last entry is the replacement
    late = served = 0
    peak_wait_new = 0.0
    for t in range(seconds):
        new_rate = rate_at(t, warm_s)
        rates = [WARM_RATE] * WARM_REPLICAS + [new_rate]
        if policy == "equal_on_ready":
            share = [1.0 / (WARM_REPLICAS + 1)] * (WARM_REPLICAS + 1)
        elif policy == "ramp_on_measured_rate":
            total = sum(rates)
            share = [r / total for r in rates]     # weight by what each can actually do
        else:
            raise ValueError(policy)
        for i, (s, r) in enumerate(zip(share, rates)):
            queues[i] = max(0.0, queues[i] + arrival * s - r)
            wait = queues[i] / r
            if i == len(rates) - 1:
                peak_wait_new = max(peak_wait_new, wait)
            done = min(queues[i] + arrival * s, r)
            served += done
            if wait > DEADLINE_S:
                late += done
    COLD_FRAC = saved
    return {"served": served, "late": late, "goodput": served - late,
            "peak_wait_new": peak_wait_new, "final_depth_new": queues[-1]}


eq = simulate("equal_on_ready")
rm = simulate("ramp_on_measured_rate")

# --- (1) Equal share on ready overloads the replica that cannot yet carry it.
assert eq["peak_wait_new"] > DEADLINE_S and rm["peak_wait_new"] <= DEADLINE_S
print(f"1  peak queue wait on the new replica: equal-share {eq['peak_wait_new']:.0f}s "
      f"(deadline {DEADLINE_S:.0f}s), rate-weighted {rm['peak_wait_new']:.1f}s")

# --- (2) And it costs fleet goodput that the warm replicas could have served.
assert rm["goodput"] > eq["goodput"]
print(f"2  goodput over the window: equal-share {eq['goodput']:.0f}, rate-weighted "
      f"{rm['goodput']:.0f} (+{rm['goodput'] / eq['goodput'] - 1:.0%}); the warm replicas "
      "were never the constraint")

# --- (3) The damage is confined to the warm-up window, which is why an hourly dashboard
# average hides it entirely and the only place it shows is per-replica during the ramp.
assert eq["late"] > 0 and rm["late"] == 0
print(f"3  requests completed past the deadline: equal-share {eq['late']:.0f}, "
      f"rate-weighted {rm['late']:.0f}; backlog remaining on the new replica "
      f"{eq['final_depth_new']:.0f} vs {rm['final_depth_new']:.0f}")

# --- (4) Adversarial: the claim is about COLDNESS, not about routing. A replica that
# starts warm makes the two policies identical, so a fleet with no prefix cache to warm
# gains nothing from ramping and should not pay its complexity.
warm_start = {p: simulate(p, cold_frac=1.0, warm_s=0) for p in
              ("equal_on_ready", "ramp_on_measured_rate")}
assert warm_start["equal_on_ready"]["goodput"] == warm_start["ramp_on_measured_rate"]["goodput"]
assert warm_start["equal_on_ready"]["late"] == 0
print(f"4  a replica that starts warm: both policies give "
      f"{warm_start['equal_on_ready']['goodput']:.0f} goodput and 0 late requests")

# --- (5) Boundary: severity scales with how cold the start is. Sweeping it shows the
# equal-share penalty appearing gradually rather than at one magic threshold.
print("5  equal-share peak wait by starting fraction: " + ", ".join(
    f"{f:.2f}->{simulate('equal_on_ready', cold_frac=f)['peak_wait_new']:.0f}s"
    for f in (1.0, 0.75, 0.5, 0.25)))

# --- (6) Property, over the grid: when the replacement is headroom rather than a
# dependency, rate-weighted routing leaves nothing late at any coldness or warm-up length.
for f in (0.1, 0.25, 0.5, 0.9):
    for w in (30, 120, 600):
        r = simulate("ramp_on_measured_rate", cold_frac=f, warm_s=w)
        assert r["late"] == 0 and r["peak_wait_new"] <= DEADLINE_S, (f, w, r)
print("6  over 12 (coldness, warm-up) shapes with the replacement as headroom: "
      "rate-weighted routing leaves 0 late requests")

# --- (7) Adversarial, and it refuted the guess this model started from: routing is not a
# substitute for admission. Raise arrivals to what the fleet needs the replacement to carry
# once warm, and BOTH policies go late, because the capacity does not exist yet. Keep the
# demand-surge shedding on until the cache is warm; the router cannot fix this one.
full = {p: simulate(p, arrival=WARM_RATE * (WARM_REPLICAS + 1), cold_frac=0.1, warm_s=600)
        for p in ("equal_on_ready", "ramp_on_measured_rate")}
assert all(r["late"] > 0 for r in full.values())
print("7  arrivals needing the replacement's WARM capacity: "
      + ", ".join(f"{p} late={r['late']:.0f}" for p, r in full.items())
      + "  (routing only chooses WHERE the lateness lands: equal-share concentrates it on "
        "the cold replica, rate-weighting spreads it across all four)")

print("all assertions passed")

Executed output:

1  peak queue wait on the new replica: equal-share 31s (deadline 30s), rate-weighted 0.0s
2  goodput over the window: equal-share 35682, rate-weighted 36000 (+1%); the warm replicas were never the constraint
3  requests completed past the deadline: equal-share 313, rate-weighted 0; backlog remaining on the new replica 0 vs 0
4  a replica that starts warm: both policies give 36000 goodput and 0 late requests
5  equal-share peak wait by starting fraction: 1.00->0s, 0.75->0s, 0.50->11s, 0.25->31s
6  over 12 (coldness, warm-up) shapes with the replacement as headroom: rate-weighted routing leaves 0 late requests
7  arrivals needing the replacement's WARM capacity: equal_on_ready late=6585, ramp_on_measured_rate late=34267  (routing only chooses WHERE the lateness lands: equal-share concentrates it on the cold replica, rate-weighting spreads it across all four)
all assertions passed

Cases 1 to 3 are the cost of routing on replica count instead of replica throughput: the cold replica's queue wait crosses the deadline while the fleet as a whole was never short of capacity, and the damage is confined to the warm-up window, which is why an hourly average shows nothing. Cases 4 and 5 keep the claim honest. It is entirely a statement about cache warm-up, it disappears when the replica starts warm, and its severity scales smoothly with how cold the start is rather than appearing at a threshold.

Case 7 is the limit, and it refuted the assumption this model was written on. Rate-weighted routing is not a substitute for admission control. Once arrivals require the capacity the replacement will have when warm, both policies deliver late requests, because that capacity does not exist yet. Routing only decides where the lateness lands: equal-share concentrates it on the cold replica, rate-weighting spreads it across all four. Keep the shedding from step 2 in place until the cache is warm, not until the replica is ready (the demand-surge runbook owns that ladder).

Verification¶

Ready replica count back to target. up == 1 means the scrape succeeded, not that the replica is ready or routable, so check readiness at the workload and the EndpointSlice, and use the metric only as a liveness cross-check:

kubectl -n $NS get deploy/$EP -o jsonpath='{.status.readyReplicas}/{.status.replicas}{"\n"}'
kubectl -n $NS get endpointslice -l kubernetes.io/service-name=$EP \
  -o jsonpath='{range .items[*].endpoints[*]}{.conditions.ready}{"\n"}{end}' | sort | uniq -c

Queue drained on the survivors: vllm:num_requests_waiting back to baseline and the preemption counter's rate back to zero.
TTFT and TPOT back under SLO on the new replicas specifically, segmented by pod, not fleet-averaged. A fleet average hides one cold replica serving badly (SLOs for inference serving):
```
histogram_quantile(0.99,
  sum by (le, pod) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
```
No duplicated external effects. If step 5 involved actions, the reconciliation pass shows no duplicate and no orphan.
Warm floor restored. Capacity is not recovered while the spare that absorbed this failure is still spent; the next failure has nothing behind it.
The failed node is drained, labelled and out of the schedulable pool until it passes qualification.

Rollback¶

There is nothing to roll back in the failure itself, but two of the actions above are reversible and must be:

# 1. Undo the emergency admission tightening once capacity is back and stable.
kubectl -n $NS annotate --overwrite deployment/$EP gateway.example.com/shed-tier-
# 2. If a replacement landed on a pool it does not belong in, stop it landing there again
# BEFORE deleting the pod. Deleting alone just lets the controller recreate it in place.
kubectl cordon <wrong-node>          # or fix the nodeSelector, affinity or taint first
kubectl -n $NS delete pod <pod-on-wrong-pool> --wait=false
kubectl uncordon <wrong-node>        # once the constraint that misplaced it is corrected

If the loss was correlated with a deploy rather than with hardware, treat it as a bad rollout and revert the revision the same way it shipped, rather than continuing to replace replicas that will fail the same way (the SLO-breach runbook has the revert path).

Leave the shed policy in place until the warm floor is restored, not merely until the SLO recovers. Removing it while the fleet has no spare means the next event starts from zero.

inference SLO breach: the fleet is whole and slow, rather than smaller.
inference demand surge: the same arithmetic when demand rises instead of capacity falling.
GPU fault, drain, reset, RMA: the hardware disposition of the node that failed.
scheduler GPU job pending: the replacement will not schedule.
checkpoint recovery and resume: the training-side analogue of losing a node mid-work.
operational runbooks: the runbook index.

References¶

vLLM production metrics (queue depth, running requests, KV-cache usage, preemptions): https://docs.vllm.ai/en/latest/usage/metrics.html
Kubernetes EndpointSlices (how a ready endpoint leaves the routing set): https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
Kubernetes pod lifecycle and probes (readiness versus liveness versus startup): https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
Kubernetes node conditions and the node controller's eviction behaviour: https://kubernetes.io/docs/concepts/architecture/nodes/