Skip to content
Markdown

GPU pool segmentation: should training and inference share hardware

Scope: deciding whether latency-critical inference, batch and evaluation work, and distributed training run on the same physical GPUs, and how to express that decision in a Kubernetes estate. Covers the interference paths that make the question non-obvious, the three-pool default, the direction preemption may flow, why reclaiming a training job is not a recovery mechanism for serving, and how to size the warm floor. The partitioning mechanisms themselves are MIG, MPS and fractional sharing; the scheduler choice is Slurm vs Kubernetes; fleet-level demand modelling is GPU capacity planning.

The Python block was executed; its output is pasted verbatim. Manifest fragments are reference templates, not applied to a cluster during authoring.

What it is

Two separate decisions get collapsed into one and should not be:

  1. Do these workloads share a control plane? Usually yes. One Kubernetes cluster, one GPU Operator, one telemetry stack, one image pipeline is less to operate and less to get wrong.
  2. Do these workloads share physical GPUs? Usually no, and the burden of proof sits with sharing.

Pool segmentation is the second decision made explicitly: a set of named capacity classes, each with its own nodes, its own utilisation target, its own admission rules, and its own definition of what an incident is.

Why use it

Because the two workloads want opposite things from the same hardware.

Inference wants warm weights, predictable tail latency, fast failover and stable capacity. An inference pool deliberately holds unused headroom, and that headroom is the product: it is what absorbs traffic variance and node loss without breaching an SLO.

Training wants saturation. It runs every GPU flat out for hours, allocates in large gangs, generates heavy collective and checkpoint traffic, and holds a topology domain for as long as the job lasts.

Put them on the same nodes and they contend on more paths than the GPU:

Contended resource How it shows up in serving
GPU memory KV-cache headroom shrinks, preemption thrash, KV-cache OOM
SM time decode slows, inter-token latency rises
NVLink and PCIe tensor-parallel collectives inside a replica slow down
Network fabric checkpoint writes and gradient all-reduce crowd the same links
Host CPU, page cache, NUMA tokenisation and the sampling loop stall (NUMA and CPU pinning)
Local and shared storage checkpoint I/O bursts collide with model loading
Scheduling capacity a training gang consumes the free GPUs a failed replica needed

The last row is the one that turns an interference problem into an availability problem, and it is the reason that preempting training on demand does not work as a recovery plan. The model below quantifies that.

When to use it (and when not)

Segment when: the inference workload has a tail-latency SLO with a budget attached; a failure domain must be recoverable within a stated time; or the training and serving stacks move at different speeds, which they almost always do.

Share, deliberately and with reservations, when: the estate is small enough that a dedicated serving pool would sit mostly idle; the background work is genuinely checkpointable and preemptible; and you are willing to measure the interference rather than assume it away.

Do not share when: the inference workload is the business-critical path and the failure budget is tight; or the background workload can hold a topology domain for hours, because then "preemptible" is only true at checkpoint boundaries you do not control.

Architecture

flowchart TB
  subgraph CP["One control plane"]
    OP["GPU Operator, telemetry, image pipeline, policy"]
  end
  CP --> A
  CP --> B
  CP --> C
  subgraph A["Pool 1: production inference"]
    A1["Homogeneous, qualified nodes<br/>warm replicas + failure headroom<br/>utilisation target below the SLO knee"]
  end
  subgraph B["Pool 2: batch and evaluation"]
    B1["Higher utilisation, weak latency SLO<br/>absorbs overflow and offline scoring"]
  end
  subgraph C["Pool 3: training and fine-tuning"]
    C1["Gang scheduling, topology domains<br/>checkpoint storage, faster stack cadence"]
  end
  B -. "preemptible, checkpointed" .-> A
  C -. "opportunistic only, never the recovery path" .-> B

The dotted edges are the only sanctioned direction of reclaim: critical serving may take capacity back from checkpointable work. Nothing flows the other way by default.

How to use it: three pools and the controls that hold them apart

Pool 1, production inference. Homogeneous, qualified nodes (compatibility profile). Warm replicas for every critical model. Explicit failure headroom. Utilisation targeted below the knee of the measured latency curve, not at some fleet-wide percentage.

Pool 2, batch and general inference. Evaluation, asynchronous scoring, internal endpoints, overflow. Runs hot because queueing costs little here.

Pool 3, training and fine-tuning. Gang admission, topology-aware placement, checkpoint storage, and a stack cadence that may legitimately run ahead of the serving profile.

The controls that keep them apart, cheapest first:

Control What it actually buys
Separate node pools the strongest scheduling and failure boundary short of separate clusters
Taints and tolerations stops accidental cross-pool placement, including from other teams' manifests
Node affinity on profile labels selects a qualified GPU, fabric and driver combination, not "a GPU"
ResourceQuota bounds a namespace's consumption so one team cannot take the pool
PriorityClass decides who wins during contention; see the direction rule below
Kueue or Volcano queues admission, fair share, borrowing and reclaim across cohorts
Reserved warm replicas the only thing that makes recovery fast; quotas do not load weights
Checkpoint requirement what makes background work safely interruptible in fact, not on paper
Separate storage paths keeps checkpoint bursts off the model-loading path

Kubernetes priority and preemption is the mechanism for the contention rule, and its default is worth knowing: a pending pod of higher priority can cause lower-priority pods to be evicted, unless the pod sets preemptionPolicy: Never, which makes it non-preempting while still letting it queue ahead.1 Give background training a non-preempting class and give serving a class that can reclaim.

MIG and time-slicing are partitioning tools, not isolation boundaries between these pools. MIG gives hardware-backed compute and memory partitions on supported GPUs and is a reasonable way to pack several small inference models onto one large device. What it does not give you is a node-level failure boundary: instances share the physical GPU, the host, the PCIe path, the power and thermal envelope, the driver and the node, so a node, driver or board fault takes all of them together. The per-instance fault containment that MIG does provide is real and is a different scope. Time-slicing gives you utilisation and, on NVIDIA's own account, "no memory or fault-isolation between replicas" and no guarantee of a proportional share of compute; reading that as no useful performance isolation under a tail-latency SLO is this page's inference, and it is why time-slicing does not belong under one. The comparison table on MPS has the per-mechanism isolation columns; do not re-derive it per cluster.

How to develop with it: size the warm floor, and stop calling preemption a recovery path

The tempting argument is that a dedicated inference pool is waste, because training could hold that capacity and yield it back on demand. The quantity that settles it is capacity-seconds of deficit: serving capacity the fleet owed and did not have. The model below was executed; the figures in the prose are its output.

# warm_floor.py -- validated: why reclaiming GPUs from a preemptible training job is not a
# recovery mechanism for latency-critical serving, and how big the warm floor has to be.
# The quantity that matters is capacity-seconds of deficit: serving capacity the fleet owed
# and did not have. Five results:
#   (1) preemption's cost is dominated by weight load, not by eviction;
#   (2) driving eviction to zero still leaves the whole load time, so the lever does not
#       exist on that axis, which is why the answer is pre-loaded capacity;
#   (3) the rule is a threshold on load time, derived here rather than asserted, and below
#       it preemption does meet the error budget;
#   (4) the three claims on spare capacity do NOT all combine the same way. A rollout is
#       voluntary, so it maxes against the others. A demand surge and a node failure are
#       independent events that can coincide, so surviving both means adding them;
#   (5) which term dominates flips with the shape of the fleet, so a fixed percentage is
#       wrong on at least one of any two fleets.
# numpy only.
import numpy as np


def unavailable(replica_gpus, t_evict, t_load, t_detect=15.0):
    """GPU-seconds during which one replica's capacity is absent from the fleet, when the
    replacement comes from reclaimed capacity. Detection, eviction and load are serial.
    This measures LOST CAPACITY, not an SLO deficit: whether the loss is felt depends on
    the demand at the time, which is why the warm-floor model below takes arrival rate
    separately. It is the right quantity for comparing two recovery MECHANISMS, because
    the demand is identical under both."""
    return replica_gpus * (t_detect + t_evict + t_load)


def unavailable_warm(replica_gpus, t_detect=15.0, t_switch=2.0):
    """Same loss, absorbed by a replica that is already loaded and registered."""
    return replica_gpus * (t_detect + t_switch)


GPUS, T_EVICT, T_LOAD, T_DETECT = 8, 45.0, 420.0, 15.0

d_preempt, d_warm = unavailable(GPUS, T_EVICT, T_LOAD), unavailable_warm(GPUS)
assert (d_preempt, d_warm) == (3840.0, 136.0)
print(f"1  one 8-GPU replica lost: preempt-and-load leaves {d_preempt:.0f} GPU-seconds "
      f"of capacity absent, warm standby {d_warm:.0f} -> {d_preempt / d_warm:.0f}x")

# --- (2) Where the deficit sits. Eviction is a tenth of it; weight load is seven eighths.
shares = {"detect": GPUS * T_DETECT, "evict": GPUS * T_EVICT, "load": GPUS * T_LOAD}
frac = {k: v / d_preempt for k, v in shares.items()}
assert frac["load"] > 8 * frac["evict"]
assert abs(sum(frac.values()) - 1.0) < 1e-12
print("2  deficit composition: " + ", ".join(f"{k} {v:.1%}" for k, v in frac.items()))

# --- (3) Adversarial: make preemption instant. Engineering a perfect checkpoint-and-yield
# removes only the eviction share. The mechanism cannot close the gap.
d_instant = unavailable(GPUS, 0.0, T_LOAD)
assert d_instant / d_warm > 25
print(f"3  eviction driven to 0 s: {d_instant:.0f} GPU-seconds absent, still "
      f"{d_instant / d_warm:.0f}x the warm path")

# --- (4) The rule is a threshold, not a prohibition. Solve for the load time at which
# preemption meets a budget, and check both sides of it.
def max_load_for_budget(budget, replica_gpus, t_evict, t_detect=T_DETECT):
    return budget / replica_gpus - t_detect - t_evict

BUDGET = 600.0            # GPU-seconds of absent capacity the SLO can absorb at peak
t_ok = max_load_for_budget(BUDGET, GPUS, T_EVICT)
assert np.isclose(t_ok, 15.0)
assert unavailable(GPUS, T_EVICT, 12.0) < BUDGET < unavailable(GPUS, T_EVICT, 16.0)
print(f"4  budget {BUDGET:.0f} GPU-s -> preemption suffices only up to a {t_ok:.0f}s load. "
      f"12 s load costs {unavailable(GPUS, T_EVICT, 12.0):.0f} (passes); "
      f"16 s costs {unavailable(GPUS, T_EVICT, 16.0):.0f} (fails); "
      f"the real 420 s load costs {d_preempt:.0f}, {d_preempt / BUDGET:.1f}x the budget")

# --- (5) The three claims, and how they actually combine. Getting this wrong in either
# direction is expensive: summing all three over-buys, maxing all three under-buys.
def claims(rate_base, rate_peak, rate_loaded, rate_per_replica, lead_s, ramp_s,
           replica_gpus, domain_replicas):
    """burst: extra replicas the arrival ramp demands before newly provisioned capacity is
    ready, with the rate climbing linearly from base to peak over ramp_s.
    `rate_loaded` is the sustainable service rate ALREADY deployed and warm, which is not
    the same as the base arrival rate: a fleet operated below the saturation knee has spare
    service rate by construction, and only demand above it needs new capacity. Conflating
    the two, as the first version of this model did, overstates the burst term."""
    if ramp_s <= 0 or rate_per_replica <= 0:
        raise ValueError("ramp_s and rate_per_replica must be positive")
    if rate_loaded < rate_base:
        raise ValueError("loaded capacity below base arrival rate is already saturated")
    reached = rate_base + (rate_peak - rate_base) * min(lead_s, ramp_s) / ramp_s
    burst = int(np.ceil(max(0.0, reached - rate_loaded) / rate_per_replica))
    return {"burst": burst * replica_gpus,
            "failure": domain_replicas * replica_gpus,
            "rollout": replica_gpus}


def floor(c):
    """A rollout is a choice: defer it during an outage or a surge, so it maxes against
    them. A surge and a node failure are independent and can coincide, so surviving both
    means holding both. Returns (survive_both, survive_worst_single)."""
    both = c["burst"] + c["failure"]
    return max(both, c["rollout"]), max(c["burst"], c["failure"], c["rollout"])


f = claims(20.0, 200.0, 20.0, 20.0, 900.0, 900.0, GPUS, 2)
both, worst = floor(f)
assert (f["burst"], f["failure"], f["rollout"]) == (72, 16, 8)
assert (both, worst) == (88, 72)
print(f"5  10x surge, 2-replica domain: burst {f['burst']}, failure {f['failure']}, "
      f"rollout {f['rollout']} GPUs -> {both} to survive a surge AND a domain loss, "
      f"{worst} for the worst single event")

# --- (6) Adversarial: the max-of-three rule that reads well is wrong by the failure term
# whenever a surge and an outage can coincide, which is exactly when it matters.
assert worst < both and both - worst == f["failure"]
print(f"6  max-of-three under-buys by {both - worst} GPUs, precisely the failure domain: "
      "the rule is defensible for the rollout term only, because that one is voluntary")

# --- (7) Which term dominates flips with the fleet. A fixed percentage is calibrated on
# one of these and wrong on the other.
g = claims(20.0, 60.0, 20.0, 20.0, 900.0, 900.0, GPUS, 6)
assert (g["burst"], g["failure"]) == (16, 48)
print(f"7  gentler surge, rack-sized domain: burst {g['burst']}, failure {g['failure']} "
      f"-> {floor(g)[0]} GPUs; the dominant term is not a constant")

# --- (8) Boundary: a surge that ramps far slower than provisioning needs almost no warm
# capacity for the burst, because the new replicas arrive before the demand does.
h = claims(20.0, 200.0, 20.0, 20.0, 900.0, 9000.0, GPUS, 2)
assert h["burst"] == 8 and floor(h)[0] == 24
print(f"8  same 10x surge stretched 10x longer: burst falls {f['burst']} -> {h['burst']} "
      "GPUs; how fast demand arrives matters as much as how large it gets")

# --- (9) The correction the first version of this model needed: a fleet operated below the
# saturation knee already has warm service rate in hand, and only demand above THAT needs
# new capacity. Using the base arrival rate as the reference overstates the burst term, here
# by half, on the same surge.
loaded = claims(20.0, 60.0, 40.0, 20.0, 900.0, 900.0, GPUS, 6)
assert loaded["burst"] == 8 and g["burst"] == 16
print(f"9  gentler surge again, but with 40/s already loaded rather than 20/s: burst falls "
      f"{g['burst']} -> {loaded['burst']} GPUs, because half the rise lands on capacity "
      "the fleet already has warm")

# --- (10) Degenerate input fails loudly rather than dividing by zero.
for bad in ({"ramp_s": 0.0}, {"rate_per_replica": 0.0}, {"rate_loaded": 5.0}):
    try:
        claims(20.0, 200.0, bad.get("rate_loaded", 20.0),
               bad.get("rate_per_replica", 20.0), 900.0,
               bad.get("ramp_s", 900.0), GPUS, 2)
        raise AssertionError(f"degenerate input accepted: {bad}")
    except ValueError:
        pass
print("10 zero ramp time, zero per-replica rate, and loaded capacity below the base "
      "arrival rate all raise rather than producing a number")

print("all assertions passed")

Executed output:

1  one 8-GPU replica lost: preempt-and-load leaves 3840 GPU-seconds of capacity absent, warm standby 136 -> 28x
2  deficit composition: detect 3.1%, evict 9.4%, load 87.5%
3  eviction driven to 0 s: 3480 GPU-seconds absent, still 26x the warm path
4  budget 600 GPU-s -> preemption suffices only up to a 15s load. 12 s load costs 576 (passes); 16 s costs 608 (fails); the real 420 s load costs 3840, 6.4x the budget
5  10x surge, 2-replica domain: burst 72, failure 16, rollout 8 GPUs -> 88 to survive a surge AND a domain loss, 72 for the worst single event
6  max-of-three under-buys by 16 GPUs, precisely the failure domain: the rule is defensible for the rollout term only, because that one is voluntary
7  gentler surge, rack-sized domain: burst 16, failure 48 -> 64 GPUs; the dominant term is not a constant
8  same 10x surge stretched 10x longer: burst falls 72 -> 8 GPUs; how fast demand arrives matters as much as how large it gets
9  gentler surge again, but with 40/s already loaded rather than 20/s: burst falls 16 -> 8 GPUs, because half the rise lands on capacity the fleet already has warm
10 zero ramp time, zero per-replica rate, and loaded capacity below the base arrival rate all raise rather than producing a number
all assertions passed

Read case 2 and case 3 together. Eviction is 9.4% of the absent capacity and weight load is 87.5%; engineering a perfect instant yield removes the 9.4% and leaves you twenty-six times worse than a replica that was already loaded. There is no lever on the preemption axis that closes this. The argument for reserved warm capacity is an argument about load time, not about scheduling.

Note what the first three cases measure and what they do not. GPU-seconds of absent capacity is not an SLO deficit: whether a loss is felt depends on the demand at the time. It is the right quantity here precisely because the demand is identical under both recovery mechanisms, so it isolates the mechanism. The warm-floor model below is where arrival rate enters.

Case 4 keeps it honest. This is a threshold, not a prohibition. At the stated budget the crossover is a fifteen-second load; a small quantised model that loads in twelve seconds passes, and for that model preemptible background capacity is a perfectly good recovery path. Derive your own crossover from your own load time and budget rather than importing the conclusion.

Cases 5 and 6 are where the tidy version of the warm-floor rule is wrong, and it is worth being precise because the tidy version is the one that gets repeated. The three claims on spare capacity do not all combine the same way:

  • A rollout is voluntary. You can defer it during an outage or a surge, so it maxes against the other terms rather than adding to them.
  • A demand surge and a node failure are not voluntary, and they are independent. Nothing stops them coinciding. A fleet holding only the larger of the two survives whichever arrives first and not both.

Case 6 measures the gap: the neat max-of-three rule under-buys by exactly the failure domain, 16 GPUs on this fleet. Use the max for the rollout term only. This is a deliberate divergence from GPU capacity planning, whose fleet model folds burst and failure headroom into one additive percentage; that page sizes a training fleet against duty cycle, where the two are not separable, and this one sizes a serving fleet against an arrival rate, where they are. The additive treatment of burst and failure is the common ground.

Cases 7 to 9 show which term dominates is not a constant, which is why a fixed "keep twenty percent warm" rule is wrong on at least one of any two fleets. Case 8: the same tenfold surge stretched over ten times longer needs a ninth of the burst capacity, because the new replicas arrive before the demand does. Case 9 is a correction to this page's first version, which used the base arrival rate as the reference for the burst term. That was wrong: a fleet operated below the saturation knee already holds warm service rate by construction, and only demand above that needs new capacity. Taking loaded capacity as its own input halves the burst term on the same surge.

How to run it in production

Set the utilisation target from a measured saturation curve, not a number. Sweep concurrency and token volume against queue time, TTFT, inter-token latency, KV pressure and rejection rate; operate below the knee (SLOs for inference serving). A trading-critical or otherwise tightly-bounded workload will sit at a much lower utilisation than a batch pool, and that unused capacity is buying predictable latency and failure tolerance. The commercial figure is cost per successful request at the SLO, not GPU busy time. A GPU can be busy with poor batching, recomputation or blocked collectives.

Fix the preemption direction in policy, not in the incident. Critical inference may reclaim from checkpointable batch and training. Normal training does not preempt production serving: it would remove warm capacity instantly and take minutes to restore it, which is worse than the capacity it recovers. Exceptions exist, an urgent security-model retrain outranking a development endpoint being the usual one, and they belong in an explicit priority class rather than in an operator's judgement at two in the morning.

Make "preemptible" mean something. A workload allowed onto reclaimable capacity needs checkpointing, a restart policy, and a known bound on lost computation. Where the scheduler can see checkpoint age, prefer victims that just checkpointed.

Keep development off production nodes. Development legitimately changes engine versions, CUDA libraries, quantisation and memory settings; a runaway request there can exhaust GPU memory or destabilise a runtime. Share the automation and the promoted artefacts, not the mutable runtime capacity. Staging should reproduce the production compatibility profile at smaller scale, or it cannot catch driver and topology problems before production does.

Do not lean on a PodDisruptionBudget for this. A PDB bounds voluntary disruptions. It is not a defence against node failure, and it will not stop a training job from having taken the capacity you needed.

How to maintain it

  • Re-measure interference rather than assuming it. If you do share, run the serving benchmark with and without the background load and keep the delta as a standing number.
  • Alarm on the warm floor, not on utilisation. Loaded, healthy, router-registered spare replicas is the quantity; idle GPUs is not the same thing.
  • Recompute the floor when the model or the lead time changes. A larger model moves both the load time and the group size (multi-node inference replicas).
  • Review pool boundaries when the estate grows. What was one cluster with taints becomes separate clusters when blast radius or security requirements justify it, and that is a promotion, not a rewrite.
  • Track reclaim events. Frequent reclaim from the batch pool means the inference pool is undersized; zero reclaim ever means the sharing you built is not being used.

Failure modes

  • A training gang holds the GPUs a failed replica needed. Recovery waits for eviction plus a full weight load. The measured deficit is in case 1 above.
  • Preemption of training used as the recovery plan. It is a capacity plan, not a recovery plan, and the difference is the load time.
  • A fixed warm percentage. Right for the fleet it was calibrated on, wrong for the next one; cases 5 to 8 show the dominant term changing.
  • Burst and failure headroom folded into one max. They are independent events that can coincide; case 6 measures what that under-buys.
  • MIG treated as a failure boundary. Instances share a GPU, a host, a driver and a node. A node fault takes all of them.
  • Time-slicing under a latency SLO. It raises utilisation and provides no performance isolation, so tail latency becomes a function of the neighbour.
  • Shared checkpoint and model-load storage. A checkpoint burst stretches the load time of the replica you are trying to bring back.
  • Quota mistaken for reservation. A quota stops a team taking more; it does not hold anything for you.
  • One pool of generic nvidia.com/gpu. Sharing then happens by accident, and nobody decided it.

References

  • Kubernetes pod priority and preemption (preemptionPolicy, eviction of lower-priority pods): https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/
  • Kubernetes taints and tolerations: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
  • Kubernetes resource quotas: https://kubernetes.io/docs/concepts/policy/resource-quotas/
  • Kubernetes PodDisruptionBudget (voluntary disruptions only): https://kubernetes.io/docs/tasks/run-application/configure-pdb/
  • NVIDIA GPU Operator, time-slicing GPUs in Kubernetes: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
  • NVIDIA Multi-Instance GPU user guide: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
  • Kueue cluster queues, cohorts, borrowing and reclaim: https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/
  • Volcano queues and reclaim: https://volcano.sh/docs/concepts/queue/

Related: GPU capacity planning · Multi-node inference replicas · MIG partitioning · MPS · Fractional GPU sharing · Slurm vs Kubernetes · GPU consumption models · Node resource isolation · Glossary

  1. Kubernetes pod priority and preemption: a pending higher-priority pod can trigger eviction of lower-priority pods, and preemptionPolicy: Never places a pod in the scheduling queue ahead of lower-priority pods without preempting any of them. https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/ ↩