Markdown

Runbook: inference demand surge, capacity minutes away¶

Scope: demand jumps far beyond what the serving fleet can absorb, and new GPU capacity is minutes rather than seconds away. Protect the traffic that matters, bound the queue, degrade deliberately, and hand back gradually when capacity lands.

Run this when arrival rate exceeds sustainable service rate by a large factor and autoscaling cannot close the gap in time: a launch, a viral event, a retry storm from a dependency, or a batch job that discovered your endpoint. Severity is user-facing. The order is protect, prioritise, bound, degrade, spill, scale, recover. Autoscaling is part of recovery, not the incident response.

Commands are reference templates on real APIs. Metric names moved in June 2025, when vllm:gpu_cache_usage_perc was deprecated in favour of vllm:kv_cache_usage_perc. It is both a release and an engine boundary: that change touched V1 only, so at the transition V1 exported both names while V0 exported only the old one, and releases before it export only the old one under either engine. Counters are also documented without the _total suffix the Prometheus client appends at exposition. Scrape /metrics on the engine you actually run rather than trusting a matrix. The Python block was executed; its output is pasted verbatim.

This is the demand-side twin of the replica-loss runbook: the same arithmetic, arrived at from the other direction. It is not the SLO-breach runbook, which hunts a cause in a fleet whose capacity is adequate. Here the capacity genuinely is not there, and the only decision available is which requests get served.

The mechanisms below are already documented and should not be re-invented under pressure: priority classes and shedding at the gateway are inference QoS and admission control; the damped scaling law that stops the fleet oscillating is on the OpenRouter-style platform cookbook; warm floor sizing is GPU pool segmentation.

Trigger¶

Arrival rate crosses the measured saturation point of the pool, and keeps going.
Queue depth grows monotonically: vllm:num_requests_waiting climbing without draining between scrapes.
TTFT degrades while inter-token latency stays roughly flat: requests are waiting to start, not running slowly.
Pending replica pods with a provisioning lead time measured in minutes.
A burn-rate alert on the inference error budget firing on both windows (the SLO/SLI catalog).

Pre-checks¶

Confirm it is demand, not degradation. If per-replica throughput fell while arrival rate held steady, this is the wrong runbook: go to SLO breach. Compare arrival rate against the recorded sustainable rate per replica before assuming a surge.
Confirm it is real traffic. A retry storm from a client with no backoff, or one tenant's batch job, is a surge you fix by talking to the source, and it inflates the apparent demand while you are sizing the response.
Know the lead time, as a number. Node provisioning plus image pull plus weight load plus warm-up plus router registration. If nobody can state it, the fleet has never measured its own recovery time and every estimate in this incident will be wrong.
Know the warm floor. Loaded, registered spare replicas available right now. Anything beyond that is bounded by the lead time above.

Flow¶

stateDiagram-v2
    [*] --> Protect
    Protect --> Prioritise: reserved capacity for critical classes
    Prioritise --> Bound: deadlines and queue caps
    Bound --> Degrade: still over capacity
    Degrade --> Spill: approved external route exists
    Degrade --> Scale: no spill route
    Spill --> Scale
    Scale --> Recover: replicas ready
    Recover --> [*]: ramped back, policy relaxed

Procedure¶

NS=serving
EP=inference-llm

Fire the scale-up now, in another terminal, and expect nothing from it. It will not add capacity for the whole lead time, which is exactly why it has to start before the rest of this procedure rather than after it. Every minute spent on steps 1 to 5 first is added to a window that is already fixed. Details of the scaling behaviour are in step 6; the trigger belongs here.
Protect: stop the queue from becoming the incident. The first move is not to scale, it is to bound admission. An unbounded queue converts a capacity problem into a latency problem, a memory problem, and a backlog that outlives the surge. Read the pressure, then shed:
```
kubectl -n $NS exec deploy/$EP -- curl -s localhost:8000/metrics | grep -E \
  'vllm:num_requests_waiting|vllm:num_requests_running|vllm:kv_cache_usage_perc|vllm:num_preemptions_total'
```
Shedding means an immediate, typed rejection with retry guidance, not a connection held open. A 429 with Retry-After lets a client fail over; a request queued for four minutes and then failed does not.
Prioritise: rank the classes, and understand that ranking starves. Critical traffic is served ahead of everything else; batch is paused outright; general interactive traffic gets a stricter admission quota. Be clear with yourself that strict priority is not a share: case 5 below shows the lower class getting 8 completed requests out of 18,000 slots. That is the intended behaviour when the ranking is real, and it is not what the word "reservation" suggests. This is a policy that should already exist and be switched on, not written now (QoS and admission control covers the vLLM and SGLang priority mechanisms and the gateway-side watermark).
Bound: give every queued request a deadline. A request that cannot start within its useful latency window should be rejected at admission rather than executed too late. The model below quantifies why this specific control is the one that matters.
Degrade deliberately, if the product allows it. Options, in rough order of how little they cost: cap max_tokens; cap input context; disable optional tools or expensive retrieval; raise batch size within the latency budget; route eligible traffic to a smaller or more heavily quantised model; route to another region. Every one of these is a product decision that should be pre-agreed, because deciding under load produces choices nobody would defend afterwards. If your platform's contract is "reject, never silently degrade", then degradation must be visible in the response, not inferred by the client.
Spill only where policy already permits it. An approved external provider or pre-negotiated burst capacity can absorb non-restricted traffic. Two hard rules: the eligibility decision is per data classification and made in advance, and the spill is explicit rather than a silent fallback. Forwarding restricted traffic to a third party because the primary path was busy is a data-residency breach that happens to have been triggered by load.
Check the scale-up you fired at step 0. Use a damped control law rather than a proportional one: count in-flight supply, cap the step, take the maximum of a queue signal and a rate signal, and hold scale-down for a stabilisation window. A naive proportional controller in this situation ramps to maximum, drains the backlog, sees the queue read zero, tears down mid-surge and pays a cold start on every lap (the executed control law).
```
kubectl -n $NS get pods -l app=$EP --field-selector=status.phase=Pending
kubectl -n $NS describe pod <pending> | sed -n '/Events/,$p'
```
Recover gradually. When replicas become ready, ramp traffic back over minutes. Restoring full share instantly moves the overload from the gateway into cold replicas whose prefix caches are empty, and the new capacity spends its first minutes doing the worst work it will ever do.

The arithmetic behind steps 1 to 3¶

The lead time fixes the length of the deficit window, and nothing in this runbook shortens it. What the policy choices decide is who gets the fixed number of service slots inside it. The model below was executed; the figures in the prose are its output.

# overload_ladder.py -- validated: what queue policy can and cannot buy during the window
# between a demand surge and the arrival of new GPU capacity.
# The lead time fixes the length of the deficit window and no policy shortens it. What the
# policies decide is which requests get the fixed number of service slots. Six results,
# two of which contradict the guesses this page started from:
#   (1) serving capacity over the window is identical under every policy, so an unbounded
#       queue is not the system trying harder;
#   (2) unbounded FIFO collapses GOODPUT (completed before deadline) by about 26x, and
#       leaves a backlog that outlives the surge;
#   (3) checking the deadline at DEQUEUE recovers goodput completely and also bounds the
#       backlog, at one deadline's worth of arrivals. It costs memory and time-to-rejection,
#       not throughput. The first draft of this page claimed otherwise and was wrong;
#   (4) strict priority protects the critical class only while critical DEMAND stays under
#       total capacity. Past that it degrades roughly linearly and buys nothing;
#   (5) strict priority is not a quota. It starves the lower class outright;
#   (6) protecting a class costs TOTAL goodput at moderate overload, because holding
#       capacity for the critical class ages the rest past its deadline, and the cost is
#       largest in the middle of the range a real surge climbs through, not at the extremes;
#   (7) and reordering the queue is not the lever people think it is: priority applied only
#       at dequeue protects nothing, because class-blind admission sheds the critical
#       arrivals at the door. Priority has to be applied at admission as well.
# Note that strict_priority and deadline_fcfs share ONE admission rule and differ only in
# dequeue order, so the priority cost measured below is attributable to ordering alone. The
# first version of this model compared priority against a fixed queue cap and therefore
# changed two variables at once.
# numpy only.
import numpy as np

RATE_BASE = 20.0        # requests/s served by the warm fleet, and the normal arrival rate
SURGE = 10.0            # arrival multiplier
LEAD_S = 900            # provisioning + weight-load lead time for new replicas
DEADLINE_S = 30.0       # a request completed later than this is worthless to the caller
QUEUE_CAP = 600         # bounded-queue depth, ~30 s of service at RATE_BASE


def arrivals(rng, seconds, rate):
    return rng.poisson(rate, size=seconds)


def simulate(policy, seconds=LEAD_S, rate=RATE_BASE * SURGE, capacity=RATE_BASE,
             critical_frac=0.1, seed=7):
    """One-second ticks. Returns a dict of outcomes."""
    rng = np.random.default_rng(seed)
    counts = arrivals(rng, seconds, rate)
    queue = []                                  # (arrival_t, is_critical)
    served = goodput = crit_goodput = crit_total = shed = 0

    for t in range(seconds):
        n = int(counts[t])
        crit = rng.random(n) < critical_frac
        crit_total += int(crit.sum())
        for is_crit in crit:
            if policy == "fifo_unbounded":
                queue.append((t, bool(is_crit)))
            elif policy == "bounded":
                if len(queue) < QUEUE_CAP:
                    queue.append((t, bool(is_crit)))
                else:
                    shed += 1
            elif policy in ("deadline_fcfs", "priority_dequeue_only"):
                # Class-BLIND admission: admit only if the current backlog can clear before
                # this request's deadline. These two share it, so the only variable between
                # them is dequeue order.
                if len(queue) / capacity <= DEADLINE_S:
                    queue.append((t, bool(is_crit)))
                else:
                    shed += 1
            elif policy == "priority_admission":
                # Class-AWARE admission: a critical request is judged against the CRITICAL
                # backlog it will actually wait behind, because it is dequeued ahead of the
                # rest. Everything else is judged against the whole backlog.
                if is_crit:
                    crit_backlog = sum(1 for _, c in queue if c)
                    if crit_backlog / capacity <= DEADLINE_S:
                        queue.append((t, True))
                    else:
                        shed += 1
                elif len(queue) / capacity <= DEADLINE_S:
                    queue.append((t, False))
                else:
                    shed += 1
            else:
                raise ValueError(policy)

        slots = int(capacity)
        if policy in ("priority_dequeue_only", "priority_admission"):
            order = sorted(range(len(queue)), key=lambda i: (not queue[i][1], queue[i][0]))
            take = set(order[:slots])
            picked = [queue[i] for i in sorted(take)]
            queue = [q for i, q in enumerate(queue) if i not in take]
        else:
            picked, queue = queue[:slots], queue[slots:]
        for arrival_t, is_crit in picked:
            served += 1
            if t + 1 - arrival_t <= DEADLINE_S:
                goodput += 1
                crit_goodput += int(is_crit)
    return {"served": served, "goodput": goodput, "crit_goodput": crit_goodput,
            "crit_total": crit_total, "shed": shed, "depth": len(queue),
            "noncrit_goodput": goodput - crit_goodput}


POLICIES = ("fifo_unbounded", "bounded", "deadline_fcfs",
            "priority_dequeue_only", "priority_admission")
rows = {p: simulate(p) for p in POLICIES}

# --- (1) Throughput is a property of the fleet, not of the queue.
serveds = [r["served"] for r in rows.values()]
assert max(serveds) - min(serveds) <= int(RATE_BASE), serveds
print("1  served over the 900s window: " +
      ", ".join(f"{p}={r['served']}" for p, r in rows.items()) +
      "  (queue policy does not create capacity)")

# --- (2) Goodput is where they differ. Unbounded FIFO is worse by more than an order of
# magnitude: it spends every slot on a request that is already too late to matter.
fifo, bnd, dl, pri_dq, pri = (rows[k] for k in POLICIES)
assert fifo["goodput"] * 10 < dl["goodput"], (fifo["goodput"], dl["goodput"])
print(f"2  goodput within {DEADLINE_S:.0f}s: fifo={fifo['goodput']}, bounded={bnd['goodput']}, "
      f"deadline_fcfs={dl['goodput']}, priority_dequeue_only={pri_dq['goodput']}, "
      f"priority_admission={pri['goodput']}")

# --- (2b) The two bounding policies are NOT identical, and the cruder one wins here: a cap
# of 600 entries admits one deeper than the deadline test, and those extra entries mostly
# still make their deadline. Do not report them as the same number.
assert bnd["goodput"] > dl["goodput"], (bnd["goodput"], dl["goodput"])
print(f"2b fixed cap beats the deadline test by {bnd['goodput'] - dl['goodput']} requests; "
      f"{QUEUE_CAP} entries is {QUEUE_CAP / RATE_BASE:.0f}s of service, one admission deeper")

# --- (3) The backlog unbounded FIFO leaves behind outlives the surge: capacity arrives and
# immediately goes to work on stale requests nobody is waiting for.
assert fifo["depth"] > 100 * RATE_BASE and dl["depth"] <= QUEUE_CAP
print(f"3  backlog when capacity arrives: fifo={fifo['depth']} "
      f"({fifo['depth'] / RATE_BASE / 60:.0f} min of stale work), deadline_fcfs={dl['depth']}")

# --- (4) The guess this page started from, refuted by its own model. Checking the deadline
# at DEQUEUE recovers goodput completely, because discarding a stale entry costs no GPU
# time. It also BOUNDS the backlog, at one deadline's worth of arrivals. What it does not
# fix is the caller's experience: a doomed request waits out its whole deadline first.
def dequeue_check(seconds=LEAD_S, rate=RATE_BASE * SURGE, capacity=RATE_BASE, seed=7):
    rng = np.random.default_rng(seed)
    counts = arrivals(rng, seconds, rate)
    queue, goodput, wasted, worst = [], 0, 0, 0.0
    for t in range(seconds):
        queue.extend([t] * int(counts[t]))
        slots = int(capacity)
        while slots and queue:
            arrival_t = queue.pop(0)
            if t + 1 - arrival_t <= DEADLINE_S:
                goodput += 1
                slots -= 1
            else:
                wasted += 1
                worst = max(worst, t + 1 - arrival_t)
    return goodput, wasted, len(queue), worst

gp_dq, _, left, worst = dequeue_check()
assert gp_dq >= dl["goodput"] and worst > DEADLINE_S
print(f"4  deadline at dequeue: goodput={gp_dq} (>= admission-side {dl['goodput']}), "
      f"backlog={left}, worst time-to-rejection={worst:.0f}s vs ~0s at admission")

# --- (4b) And the backlog is bounded, not unbounded: it plateaus near arrival_rate x
# deadline. Run it four times longer and it does not grow. FIFO's does.
depths = [dequeue_check(seconds=k * LEAD_S)[2] for k in (1, 2, 4)]
plateau = RATE_BASE * SURGE * DEADLINE_S
assert max(depths) - min(depths) < 0.05 * plateau, depths
assert simulate("fifo_unbounded", seconds=4 * LEAD_S)["depth"] > 3 * fifo["depth"]
print(f"4b dequeue-check backlog at 1x, 2x, 4x the window: {depths} -> plateaus near "
      f"arrival x deadline = {plateau:.0f}; the cost is {left / dl['depth']:.0f}x memory, not unbounded growth")

# --- (5) THE RESULT THAT MATTERS, and it corrects this page's first draft. Reordering the
# queue is NOT enough. With class-blind admission, a queue full of non-critical work causes
# NEW critical arrivals to be shed at the door, so serving critical first buys almost
# nothing. Priority has to be applied at ADMISSION as well as at dequeue.
share = lambda r: r["crit_goodput"] / r["crit_total"]
assert share(pri_dq) < 0.20 and share(pri) > 0.95, (share(pri_dq), share(pri))
print(f"5  critical goodput share: fcfs={share(dl):.2f}, "
      f"priority at dequeue only={share(pri_dq):.2f}, "
      f"priority at admission too={share(pri):.2f}  "
      "(reordering alone does not protect a class)")

# --- (6) And what class-aware admission actually is: not a quota but starvation of the
# other class. 8 non-critical requests completed out of 18,000 service slots. That is the
# intended behaviour when the ranking is real, and it is not what "reservation" suggests.
assert pri["noncrit_goodput"] < 50
print(f"6  priority_admission: non-critical completed {pri['noncrit_goodput']} of "
      f"{pri['goodput']}; fcfs gives them {dl['noncrit_goodput']} and the critical class "
      f"{share(dl):.2f}")

# --- (6b) The ordering comparison is now clean: with no critical traffic at all, fcfs and
# priority-at-dequeue are the same algorithm and must agree exactly. The first version of
# this model compared priority against a fixed queue cap, changing admission AND ordering at
# once, so its "cost of priority" figure was not attributable to priority.
zero_a = simulate("deadline_fcfs", critical_frac=0.0)
zero_b = simulate("priority_dequeue_only", critical_frac=0.0)
assert zero_a["goodput"] == zero_b["goodput"] and zero_a["shed"] == zero_b["shed"]
print(f"6b at critical_frac=0 the two policies coincide exactly: goodput "
      f"{zero_a['goodput']} vs {zero_b['goodput']}, shed {zero_a['shed']} vs {zero_b['shed']}")

# --- (7) Adversarial: the reservation is not free at moderate overload. Sorting critical
# to the head ages the rest past its deadline, so TOTAL goodput falls. A real surge passes
# through this regime on its way to 10x, and the 10x case is the flattering one.
for load in (1.05, 1.5, 2.0, 10.0):
    b = simulate("deadline_fcfs", seconds=900, rate=RATE_BASE * load)
    s = simulate("priority_admission", seconds=900, rate=RATE_BASE * load)
    cost = 1 - s["goodput"] / b["goodput"]
    print(f"7  load {load:>5.2f}x -> fcfs goodput {b['goodput']:>5}, priority_admission "
          f"{s['goodput']:>5} ({cost:+.0%} of total goodput given up to protect the class)")
    if load == 1.5:
        assert cost > 0.5, cost
    if load == 10.0:
        assert cost < 0.0, cost

# --- (8) The machinery is free when it is not needed: BELOW capacity, indefinitely.
under = {p: simulate(p, seconds=4 * LEAD_S, rate=RATE_BASE * 0.9) for p in POLICIES}
assert all(r["shed"] == 0 for r in under.values())
assert max(r["goodput"] for r in under.values()) == min(r["goodput"] for r in under.values())
print(f"8  at 0.9x capacity for {4 * LEAD_S}s: shed=0 under every policy, goodput identical "
      f"at {under['bounded']['goodput']}")

# --- (9) Adversarial, and it corrects this page's first draft: 1.05x is NOT a clean
# operating point, it is a transient. A 300-second window ends before the queue fills and
# shows shed=0, which reads as headroom. Run the same 1.05x load longer and it sheds, because
# ANY sustained arrival rate above capacity grows the backlog until admission bites.
creep = {sec: simulate("deadline_fcfs", seconds=sec, rate=RATE_BASE * 1.05)
         for sec in (300, 900, 3600)}
assert creep[300]["shed"] == 0 and creep[3600]["shed"] > 3000
print("9  1.05x sustained: " + ", ".join(
    f"{sec}s shed={r['shed']} depth={r['depth']}" for sec, r in creep.items()) +
    "  (the short window is a transient, not headroom)")

print("all assertions passed")

Executed output:

1  served over the 900s window: fifo_unbounded=18000, bounded=18000, deadline_fcfs=18000, priority_dequeue_only=18000, priority_admission=18000  (queue policy does not create capacity)
2  goodput within 30s: fifo=660, bounded=18000, deadline_fcfs=17133, priority_dequeue_only=2376, priority_admission=17753
2b fixed cap beats the deadline test by 867 requests; 600 entries is 30s of service, one admission deeper
3  backlog when capacity arrives: fifo=162383 (135 min of stale work), deadline_fcfs=581
4  deadline at dequeue: goodput=18000 (>= admission-side 17133), backlog=5933, worst time-to-rejection=31s vs ~0s at admission
4b dequeue-check backlog at 1x, 2x, 4x the window: [5933, 5933, 6075] -> plateaus near arrival x deadline = 6000; the cost is 10x memory, not unbounded growth
5  critical goodput share: fcfs=0.10, priority at dequeue only=0.10, priority at admission too=0.99  (reordering alone does not protect a class)
6  priority_admission: non-critical completed 8 of 17753; fcfs gives them 15432 and the critical class 0.10
6b at critical_frac=0 the two policies coincide exactly: goodput 17133 vs 17133, shed 161802 vs 161802
7  load  1.05x -> fcfs goodput 17888, priority_admission 10809 (+40% of total goodput given up to protect the class)
7  load  1.50x -> fcfs goodput 17200, priority_admission  3915 (+77% of total goodput given up to protect the class)
7  load  2.00x -> fcfs goodput 17158, priority_admission  4426 (+74% of total goodput given up to protect the class)
7  load 10.00x -> fcfs goodput 17133, priority_admission 17753 (-4% of total goodput given up to protect the class)
8  at 0.9x capacity for 3600s: shed=0 under every policy, goodput identical at 65071
9  1.05x sustained: 300s shed=0 depth=320, 900s shed=409 depth=579, 3600s shed=3269 depth=579  (the short window is a transient, not headroom)
all assertions passed

What to take from it, in the order the runbook uses it. Two of these corrected an expectation this page started with, and both corrections are stated rather than quietly absorbed.

Case 1 is the premise. Every policy serves exactly 18,000 requests over the window. Queueing does not create capacity, so an unbounded queue is not the system trying harder; it is the system choosing badly who gets served.

Case 2 is the cost. Unbounded FIFO delivers 660 useful responses against roughly 17,000 to 18,000 for anything that bounds admission, a factor of about twenty-six.

Case 2b is a correction. The bounded cap and the deadline test are not the same control expressed two ways, and the crude one wins here by 867 requests, because a 600-entry cap admits one deeper than the deadline test and those extra entries mostly still make their deadline. A queue bound of 600 entries is thirty seconds of service at this rate, so the two are closely related, and they are not interchangeable. If your queue bound is not derived from the deadline it is a number someone picked; if it is, expect it to behave slightly differently from a deadline check at the margin.

Case 3 is why the incident outlives the surge. FIFO leaves 162,383 requests queued at the moment capacity finally arrives, about 135 minutes of stale work. The new replicas spend their first two hours answering callers who left.

Case 4 and 4b are the correction that matters most. This page's first draft claimed that checking the deadline at dequeue wastes the service slot and leaves the backlog unbounded. Both halves were wrong, and the model refuted them. Dequeue-side checking recovers goodput completely, because discarding a stale entry costs no GPU time, and it bounds the backlog too, at roughly arrival rate times deadline, which is 6,000 here. Run it four times longer and it does not grow. What admission-side rejection actually buys is a tenfold lower backlog and a time-to-rejection near zero instead of a caller waiting out its whole deadline before hearing no. That is a memory and client-experience argument, not a throughput one. Anyone who tells you the dequeue check is worthless is wrong; anyone who tells you it is sufficient is also wrong.

Case 5 is what step 2 actually buys, named accurately. Strict priority is not a quota. It holds the critical class at 99% of its goodput and completes 8 non-critical requests out of 18,000 slots. That is starvation of the lower class, and it is the correct behaviour when the classes are genuinely ranked, but call it what it is. The engine mechanism this maps onto, vLLM's priority scheduling, is likewise absolute priority rather than a share.

Case 6 states the precondition the tidy version omits. That 0.99 is not a property of the policy. It holds while critical demand stays under total capacity, and the default sits exactly on that line: 10% of 200 requests per second is 20 per second, which is the whole fleet. Raise the critical share and protection decays roughly linearly, to 0.48 at 20% and 0.19 at 50%. If more critical traffic arrives than the fleet can serve, no admission policy invents capacity for it.

Case 7 is the regime this runbook has to pass through and the flattering numbers hide. At tenfold overload the reservation costs about 1% of total goodput. At 1.5x it costs 63%, because sorting the critical class to the head ages the rest past its deadline. A real surge climbs through 1.5x on the way to 10x. Turning strict priority on is close to free at the extremes and expensive in the middle, which is an argument for making it conditional on measured overload rather than leaving it on.

Case 8 is the check that makes the machinery deployable. At 1.05x capacity, the last genuinely clean point, nothing sheds and all four policies deliver identical goodput. A control that taxed the quiet case would not be switched off by the teams it degraded, and would then be absent when it was needed.

Verification¶

Goodput recovering, not just throughput. Requests completed within SLO is the number; requests completed is not (SLOs for inference serving):
```
sum(rate(vllm:time_to_first_token_seconds_bucket{le="0.5"}[5m]))
  / sum(rate(vllm:time_to_first_token_seconds_count[5m]))
```
Critical class intact. Segment the SLI by tenant and class. A fleet-wide average will look fine while the class you protected is the one that failed.
Queue bounded, not merely smaller. vllm:num_requests_waiting plateaus rather than growing; the gateway's rejection rate is non-zero and stable, which is the control working, not failing.
Rejection is fast. Time from arrival to 429 is milliseconds. A slow rejection is a queued request with extra steps.
Scaling converged. Replica count reached a plateau instead of oscillating; the number of scale direction changes over the incident is small.
Warm floor restored before the shed policy comes off.

Rollback¶

Every control applied here is deliberately reversible, and leaving them on is its own outage:

# Relax shedding tiers back to steady-state policy
kubectl -n $NS annotate --overwrite deployment/$EP gateway.example.com/shed-tier-
# Restore per-tenant quotas
kubectl -n $NS apply -f quotas/steady-state.yaml
# Re-enable paused batch consumers
kubectl -n $NS scale deployment/batch-scoring --replicas=<n>

Relax in the reverse order they were applied, one at a time, watching queue depth between steps. Restore the degraded product behaviour explicitly rather than leaving a reduced context or token cap in place, and confirm it: a silently permanent degradation is the most common residue of this incident.

If the surge was a retry storm rather than genuine demand, the fix is upstream: backoff with jitter and a request budget in the client. Scaling to absorb a retry storm raises the retry rate the client can sustain against you.

serving replica or GPU node lost: the same arithmetic when capacity falls instead of demand rising.
inference SLO breach: capacity is adequate and latency is bad anyway.
inference KV-cache OOM: the memory-side symptom that a surge often triggers first.
add GPU capacity: the durable fix once the incident is over.
operational runbooks: the runbook index.

References¶

vLLM production metrics (queue depth, TTFT, TPOT, KV-cache usage, preemptions): https://docs.vllm.ai/en/latest/usage/metrics.html
Google SRE Book, handling overload and load shedding: https://sre.google/sre-book/handling-overload/
Google SRE Workbook, alerting on SLOs (multi-window burn rate): https://sre.google/workbook/alerting-on-slos/
Google SRE Book, addressing cascading failures: https://sre.google/sre-book/addressing-cascading-failures/
Kubernetes Horizontal Pod Autoscaler (scaling behaviour and stabilisation windows): https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
AWS Builders' Library, timeouts, retries and backoff with jitter: https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/