Skip to content
Markdown

Multi-node inference replicas as one scheduling unit

Scope: serving a model whose weights or KV budget exceed one node, where the deployment unit is a group of pods that must be scheduled, made ready, failed and upgraded together. Covers gang admission with LeaderWorkerSet plus Kueue or Volcano, topology-constrained placement, group-level readiness, leader-pod failure semantics, homogeneous replicas on a heterogeneous fleet, and the rollout arithmetic that decides whether an upgrade can start at all. The parallelism choice itself is inference parallelism strategies; the replica lifecycle and routing contract are in the inference system map; the training-side gang recipe is gang-scheduled training.

Manifests here are reference templates on real APIs, pinned to the versions named in each block and not applied to a cluster during authoring. The Python block was executed; its output is pasted verbatim.

What it is

A single-node replica is a pod. Kubernetes already knows how to schedule it, probe it, replace it and roll it. A replica that spans four nodes is none of those things: it is a set of pods that only means something as a set. Tensor-parallel ranks form one NCCL communication group at startup and hold shards of one weight matrix; a pipeline stage holds layers nobody else has. Three of four ranks running is not seventy-five percent of a replica, it is zero percent of a replica holding seventy-five percent of the GPUs.

Every platform primitive therefore has to be lifted from the pod to the group:

Primitive Pod-level answer Group-level answer
Scheduling admit each pod when a node fits admit the whole group or none of it
Placement spread or pack pods keep ranks inside one high-bandwidth domain
Readiness container passes /health every rank joined the process group and the model answers
Failure restart the pod the group is unavailable; remove it from routing, recreate it whole
Rollout replace pods one at a time replace a whole group at a time; never mix versions inside one
Capacity count GPUs count groups, then multiply

Why use it

Because the alternative fails in ways that are expensive and quiet. A Deployment of four pods each requesting eight GPUs will happily place two pods, leave two Pending, and hold sixteen GPUs that serve nothing. Two such deployments arriving together can each hold half of what they need and neither can progress, which is the classic partial-allocation deadlock (orchestration decision guide derives the packing argument with an executed model).

A pod-granular rolling update is worse than slow, it is incorrect. Replacing one rank of a live tensor-parallel replica puts two model versions in one collective. The executed model below counts those states rather than describing them.

topologySpreadConstraints does not fix any of this. The Kubernetes documentation is explicit that whenUnsatisfiable: DoNotSchedule "tells the scheduler not to schedule it" (emphasis on the singular is this page's), one pod, and lists among its known limitations that "there's no guarantee that the constraints remain satisfied when Pods are removed."7 It is a per-pod placement preference evaluated one pod at a time. It has no workload object, no quota, and no all-or-nothing semantics.

When to use it (and when not)

Reach for a group-scheduled replica when:

  • The model does not fit one node at the context length and concurrency you have to serve, after quantisation has been considered (quantization for inference).
  • A replica spans nodes and its ranks exchange activations every forward pass, so placement across a slow boundary is a throughput loss, not a preference.
  • You already run gang scheduling for training and want the serving estate to obey the same admission rules (Volcano, Kueue).

Do not reach for it when:

  • The replica fits one node. An eight-GPU replica on an eight-GPU server should be one pod requesting eight GPUs. Cross-pod gang machinery buys nothing and adds a control loop that can fail.
  • The workload is embarrassingly parallel. Independent replicas of a small model need load balancing, not co-scheduling.
  • You are reaching for it to fix a capacity problem. Gang admission makes a shortage visible as a pending group. It does not create a topology that is not there.

Architecture

flowchart TB
  subgraph CTRL["Workload controller"]
    LWS["LeaderWorkerSet<br/>replicas = groups, size = pods per group"]
  end
  subgraph ADM["Admission"]
    Q["Kueue Workload<br/>or Volcano PodGroup"]
    T["Topology constraint<br/>required or preferred domain"]
  end
  subgraph GROUP["One replica group"]
    L["Leader pod<br/>8 GPUs, rank 0"]
    W1["Worker pod<br/>8 GPUs"]
    W2["Worker pod<br/>8 GPUs"]
    W3["Worker pod<br/>8 GPUs"]
  end
  LWS --> Q
  Q --> T
  T -->|"all 32 GPUs in one domain, or stay pending"| GROUP
  L -.->|"NCCL process group"| W1
  W1 -.-> W2
  W2 -.-> W3
  GROUP --> R["Group readiness gate<br/>all ranks joined + inference probe"]
  R -->|"ready"| ROUTE["Router endpoint set"]
  R -->|"any rank lost"| EVICT["Remove endpoint, recreate whole group"]

The controller owns the shape, the admission layer owns whether it may start, and the readiness gate owns whether the router may see it. Keeping those three separate is what lets you change the engine without changing the scheduler.

How to use it: the workload shape

LeaderWorkerSet exists for exactly this deployment pattern. Its README states the purpose plainly: it "aims to address common deployment patterns of AI/ML inference workloads, especially multi-host inference workloads where the LLM will be sharded and run across multiple devices on multiple nodes."1

Field placement is the thing to get right. replicas is the number of groups and sits on spec; size and restartPolicy are properties of the group and sit under spec.leaderWorkerTemplate.

# lws-replica.yaml -- reference template, LWS v0.10.0, not applied during authoring.
# spec.replicas = how many complete replicas; leaderWorkerTemplate.size = pods per replica.
apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: llm-tp32
  labels:
    kueue.x-k8s.io/queue-name: serving-queue     # Kueue admits the whole LWS group
spec:
  replicas: 6                                    # six independent 4-node replicas
  rolloutStrategy:
    type: RollingUpdate
    rollingUpdateConfiguration:
      # The DEFAULTS are maxSurge 0 and maxUnavailable 1, which takes a group out of
      # service before replacing it. The capacity model below assumes the opposite, so
      # set it explicitly rather than inheriting a rollout that shrinks the fleet.
      maxSurge: 1
      maxUnavailable: 0
  leaderWorkerTemplate:
    size: 4                                      # leader + 3 workers = one replica
    restartPolicy: RecreateGroupOnPodRestart     # this is the v0.10.0 default; see note
    leaderTemplate:
      metadata:
        annotations:
          # BOTH templates need these. Leader and workers are separate PodSets, so an
          # annotation on the leader alone constrains a one-pod set and leaves the three
          # workers free to land anywhere.
          kueue.x-k8s.io/podset-required-topology: "cloud.provider.com/topology-block"
          kueue.x-k8s.io/podset-group-name: "llm-tp32"
      spec:
        containers:
          - name: engine
            image: vllm/vllm-openai:v0.24.0      # pin; never :latest
            resources:
              limits:
                nvidia.com/gpu: 8                # one pod per node, all 8 GPUs
    workerTemplate:
      metadata:
        annotations:
          kueue.x-k8s.io/podset-required-topology: "cloud.provider.com/topology-block"
          kueue.x-k8s.io/podset-group-name: "llm-tp32"
      spec:
        containers:
          - name: engine
            image: vllm/vllm-openai:v0.24.0
            resources:
              limits:
                nvidia.com/gpu: 8

Kueue's LeaderWorkerSet guidance is explicit about the duplication: add podset-required-topology to both the leader and worker templates, and give both the same podset-group-name, because that is what makes the leader and the workers land in one topology domain.3 An annotation on the leader alone is the single most common way to end up with a group that was admitted, is running, and is permanently slow.

Three further details worth pinning down before you copy this:

  • restartPolicy takes one of four values, Default, RecreateGroupOnPodRestart, RecreateGroupAfterStart and None, and defaults to RecreateGroupOnPodRestart. The value named Default is deprecated; the upstream type comment directs you to None for that old behaviour.1 If a manifest you inherited sets Default, it is stale.
  • LWS's own gang-scheduling feature is marked Alpha in the README, with the API subject to change.1 That is why the admission decision below is delegated to Kueue or Volcano rather than taken inside LWS.
  • The project also ships a second, separate API since v0.9.0, DisaggregatedSet in its own disaggregatedset.x-k8s.io/v1 group, for prefill/decode splits co-designed with llm-d. If you are heading toward disaggregated inference, check it before building the split by hand.

How to develop with it: all-or-nothing admission

Two mature options. Pick one per cluster; running both against the same nodes gives you two schedulers arguing over the same GPUs.

Kueue

Kueue's LeaderWorkerSet integration is enabled by default and is built on its plain-pod-group support.3 The group is admitted as one Workload carrying the aggregate request, held before admission, and released only when the whole thing fits its quota and flavour.

For bare pods rather than an LWS, the two markers are not both annotations, which is a common error:

# Plain pod group. NOTE the split: name is a LABEL, count is an ANNOTATION.
metadata:
  labels:
    kueue.x-k8s.io/pod-group-name: "llm-tp32-0"
  annotations:
    kueue.x-k8s.io/pod-group-total-count: "4"

Topology-aware scheduling is beta and on by default since Kueue v0.14; there is no gate to switch on, only one to switch it off.5 Constrain a group with either annotation on the pod template:

kueue.x-k8s.io/podset-required-topology: "cloud.provider.com/topology-block"   # hard
kueue.x-k8s.io/podset-preferred-topology: "cloud.provider.com/topology-rack"   # falls back up

The Topology object that names those levels is cluster-scoped and, in current releases, kueue.x-k8s.io/v1beta2; kubernetes.io/hostname may appear only at the lowest level.5 A page or manifest still saying v1beta1 for Topology or ResourceFlavor is behind.

The limitation to carry with you: Kueue's pod-group support is deliberately minimal. Its own documentation states that "Kueue does not re-create failed Pods", and that on preemption it "sends delete requests for all of the Pods in the group. It is the responsibility of the user or controller that created the original Pods to create replacement Pods."4 Kueue gives you gang admission. Group recreation is the workload controller's job, which is another reason to run LWS underneath rather than bare pods.

Volcano

Volcano models the gang directly as a PodGroup with a minimum membership, and refuses to allocate below it. Its documentation states that if "the cluster resource cannot meet the demand ... no pod or task in the PodGroup will be scheduled."6

Under LeaderWorkerSet, do not write the PodGroup by hand. LWS integrates with Volcano as a scheduler provider, and the integration is disabled by default. Enabling it takes three steps: set gangSchedulingManagement.schedulerProvider: volcano in the controller's configuration, uncomment the Volcano RBAC component in the manager's kustomization, and restart the controller. LWS then creates and owns one PodGroup per replica group, named <lwsName>-<groupIndex>-<templateRevisionHash>, and its pod webhook stamps the matching group annotation on every pod.2 A static PodGroup you author yourself carries the wrong name, so the pods never reference it and the gang gate does nothing. The pod templates still need schedulerName: volcano.

The hand-written form below is for bare pods or a controller with no provider integration, not for LWS.

# podgroup-replica.yaml -- reference template, Volcano v1.15.1, not applied during authoring.
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: llm-tp32-0
spec:
  minMember: 4                # four pods, or none of them
  minResources:
    nvidia.com/gpu: "32"      # and the aggregate must be available too
  queue: serving
  priorityClassName: serving-critical
  networkTopology:
    mode: hard                # tasks must land in one HyperNode; `soft` is best-effort
    highestTierAllowed: 2

Pods join by annotation and must name the scheduler:

# On the POD template. metadata and spec are siblings, not nested.
metadata:
  annotations:
    scheduling.k8s.io/group-name: llm-tp32-0
spec:
  schedulerName: volcano

Use scheduling.k8s.io/group-name, not the similarly named scheduling.volcano.sh/group-name. Both constants exist in the Volcano API package, but the scheduler's job-info lookup and the PodGroup controller read only the former; the volcano-prefixed key is consulted on specific paths such as vGPU device handling. A pod annotated only with the second one is not a member of the PodGroup, and nothing tells you so.

Two drift traps, both read from the API types at tag v1.15.1 rather than from the documentation pages, which do not carry them: networkTopology.mode is declared with +kubebuilder:default=hard, and a newer highestTierName field is mutually exclusive with highestTierAllowed ("HighestTierName and HighestTierAllowed cannot be set simultaneously"), enforced in the PodGroup validating webhook, so setting both is a rejection rather than a merge. And at the Job level Volcano spells the gang size minAvailable, not minMember; the two names describe the same idea at different objects.

How to run it in production

Group readiness, not pod readiness

A pod whose container started is not a rank that joined the collective. Gate the endpoint on an application-level signal that is only true once every rank is in and the model has produced a token. In practice that means a leader-side probe that fails until the group is whole, and a router that consumes the group's endpoint rather than individual pods.

Resist making that probe expensive. A readiness probe that runs a real generation on a short period is itself a load source, and a failed probe on a healthy-but-busy replica pulls capacity out of a fleet that is already hot. The inference system map covers the Staged -> Loading -> Warming -> Ready -> Draining lifecycle and the liveness/readiness/startup split in detail; do not re-derive it per service.

Leader failure is group failure

If the leader participates in model initialisation and the NCCL process group, losing it does not degrade the replica, it ends it. The correct response is: stop routing to the group first, then recreate the group whole. Electing a new Kubernetes leader does not repair an already-broken process group, and restarting one pod into a group whose collective has torn down gives you a pod that will never join.

In-flight requests on that replica lose their GPU-resident KV. Whether they may be retried elsewhere is a request-policy question with a hard boundary: only before any bytes have been committed to the caller. The inference system map states the retry contract; the incident procedure is the replica-loss runbook.

Availability for a multi-node model comes from running several complete groups across separate failure domains. It does not come from trying to make one group unbreakable.

Homogeneous replica, heterogeneous fleet

Within one synchronous group, mix nothing. Different GPU models mean different HBM, different kernels and a collective that runs at the speed of its slowest participant. Across independent groups, heterogeneity is fine and even useful: the router can weight traffic by each group's measured service rate.

Express that as qualified shapes rather than a raw GPU count. A Kueue ResourceFlavor pinned to nvidia.com/gpu.product, or a Volcano queue over a labelled node pool, turns "eight H100s" into a profile the model deployment can require and the scheduler can refuse. A model that has been benchmarked at TP=8 on H100 and TP=8 on A100 has two profiles with two different sustainable rates, not one profile with a footnote. If no qualified profile is free, the group should stay pending rather than land on hardware nobody has measured.

Rolling a group without a second cluster

The surge unit is a whole group. You need enough spare capacity for one complete replacement replica, not for a duplicate fleet, and not for one pod. The model below was executed; the numbers in the prose are its output.

# gang_rollout.py -- validated: the capacity arithmetic of upgrading multi-node inference
# replicas, and why the surge unit is a replica GROUP and never a pod.
# The shape is the one in the manifest above: each replica is 4 pods of 8 GPUs, so the
# group size is 4 and the group holds 32 GPUs. Those are different numbers and conflating
# them is how surge gets mis-sized. Four results:
#   (1) a pod-granular rolling update replaces pods inside a group that is still serving,
#       so that group holds two model versions at once. Both strategies are inspected by
#       the SAME function, so this is a measured difference, not an assumption;
#   (2) sizing surge in pods rather than groups understates the spare GPUs needed by the
#       group size, which is the number that decides whether the rollout can start at all;
#   (3) short of a whole group, the rollout must stall with the fleet intact rather than
#       strand GPUs in a group that can never reach readiness;
#   (4) required capacity is serving + max(largest failure domain, one rollout group),
#       because a rollout during an outage is a choice. A demand surge is not, which is
#       why the burst term is handled separately in GPU pool segmentation.
# numpy only.
import numpy as np

OLD, NEW, EMPTY = 0, 1, -1


def mixed_serving_groups(version, serving):
    """Inspection, shared by both strategies: how many groups that are currently taking
    traffic hold more than one model version? This is what makes a broken collective."""
    bad = 0
    for g in sorted(serving):
        live = {v for v in version[g] if v != EMPTY}
        bad += int(len(live) > 1)
    return bad


def rollout(groups, pods_per_group, gpus_per_pod, spare_gpus, unit="group"):
    """Step a one-at-a-time rollout. Returns (peak_gpus, mixed_windows, stalled)."""
    group_gpus = pods_per_group * gpus_per_pod
    steady = groups * group_gpus
    budget = steady + spare_gpus
    version = np.full((groups * 2, pods_per_group), EMPTY, dtype=int)
    version[:groups, :] = OLD                 # the running fleet
    serving = set(range(groups))
    peak, mixed, stalled = steady, 0, False

    if unit == "group":
        for g in range(groups):
            spare = groups + g                # a replacement group, built alongside
            if steady + group_gpus > budget:
                stalled = True
                break
            version[spare, :] = NEW           # build it whole, off the serving set
            peak = max(peak, steady + group_gpus)
            mixed += mixed_serving_groups(version, serving)
            serving.add(spare)                # atomic cutover, then drain the old group
            mixed += mixed_serving_groups(version, serving)
            serving.discard(g)
            version[g, :] = EMPTY
    else:                                     # unit == "pod": maxSurge counted in pods
        for g in range(groups):
            for p in range(pods_per_group):
                if steady + gpus_per_pod > budget:
                    stalled = True
                    break
                peak = max(peak, steady + gpus_per_pod)
                version[g, p] = NEW           # replaced in place, group still serving
                mixed += mixed_serving_groups(version, serving)
            if stalled:
                break
    return peak, mixed, stalled


GROUPS, PODS, GPUS_PER_POD = 6, 4, 8          # 6 replicas, each 4 nodes x 8 GPUs
GROUP_GPUS = PODS * GPUS_PER_POD              # 32
STEADY = GROUPS * GROUP_GPUS                  # 192

# --- (0) The two sizes that get conflated. The LWS group size is 4; the group holds 32.
assert (PODS, GROUP_GPUS) == (4, 32)
print(f"0  group size {PODS} pods, group footprint {GROUP_GPUS} GPUs, fleet {STEADY} GPUs")

# --- (1) Group-atomic surge with one spare group: completes, and no serving group ever
# holds two versions.
peak, mixed, stalled = rollout(GROUPS, PODS, GPUS_PER_POD, spare_gpus=GROUP_GPUS)
assert (mixed, stalled) == (0, False), (mixed, stalled)
assert peak == STEADY + GROUP_GPUS == 224
print(f"1  group-atomic surge, {GROUP_GPUS} spare GPUs -> peak {peak}, "
      f"mixed-version serving groups {mixed}, stalled {stalled}")

# --- (2) Adversarial: pod-granular surge. It fits in one pod of spare, looks cheaper, and
# is wrong. The same inspection function now finds mixed-version serving groups.
peak_p, mixed_p, stalled_p = rollout(GROUPS, PODS, GPUS_PER_POD,
                                    spare_gpus=GPUS_PER_POD, unit="pod")
assert stalled_p is False and peak_p == STEADY + GPUS_PER_POD
assert mixed_p == GROUPS * (PODS - 1) == 18, mixed_p
print(f"2  pod-granular surge, {GPUS_PER_POD} spare GPUs -> peak {peak_p}, "
      f"mixed-version windows {mixed_p} (a mixed TP group is a broken collective)")

# --- (3) The understatement. The pod view asks for one pod of spare; the correct unit asks
# for a whole group. The ratio is the group SIZE in pods, not the GPU count.
assert (peak - STEADY) // (peak_p - STEADY) == PODS
print(f"3  spare demanded: pod view {peak_p - STEADY} GPUs, group view {peak - STEADY} GPUs "
      f"-> understated by {PODS}x, exactly the group size in pods")

# --- (4) Boundary: short of a whole group, the group-atomic rollout stalls with the fleet
# intact and nothing half-built. The surge requirement is a step function, not a gradient:
# every amount of spare from 0 up to one pod short of a group behaves identically.
for spare in range(0, GROUP_GPUS, GPUS_PER_POD):
    peak_s, m_s, st_s = rollout(GROUPS, PODS, GPUS_PER_POD, spare_gpus=spare)
    assert (st_s, m_s, peak_s) == (True, 0, STEADY), (spare, st_s, m_s, peak_s)
print(f"4  every spare from 0 to {GROUP_GPUS - GPUS_PER_POD} GPUs -> stalled, 0 mixed, peak "
      f"still {STEADY}: the requirement is a step at one whole group, not a gradient")

# --- (5) Property over a grid of real serving shapes: group-atomic never mixes, and
# pod-granular always does once a group is more than one pod. Sweeping the shape is what
# makes this a result rather than a restatement of one assignment.
for g in range(1, 7):
    for pods in (1, 2, 4, 8, 16):
        for gpp in (1, 2, 4, 8):
            _, m_grp, st = rollout(g, pods, gpp, spare_gpus=pods * gpp)
            assert m_grp == 0 and st is False, (g, pods, gpp, m_grp, st)
            _, m_pod, _ = rollout(g, pods, gpp, spare_gpus=gpp, unit="pod")
            assert m_pod == g * (pods - 1), (g, pods, gpp, m_pod)
print("5  over 120 (fleet, pods-per-group, gpus-per-pod) shapes: group-atomic mixed count "
      "is always 0; pod-granular is always groups*(pods-1)")

# --- (6) Capacity. Failure headroom and rollout headroom both claim spare capacity, but a
# rollout during an outage is a choice, so these two take the max. A demand surge is not a
# choice and does not belong in this max; GPU pool segmentation handles that term.
def required(groups, group_gpus, failure_domain_groups):
    serving = groups * group_gpus
    c_failure = failure_domain_groups * group_gpus
    return serving + max(c_failure, group_gpus), serving + c_failure + group_gpus

floor, additive = required(GROUPS, GROUP_GPUS, 1)
assert (floor, additive) == (224, 256)
print(f"6  N+1 domain: floor {floor} GPUs (defer the rollout during an outage) vs "
      f"{additive} (roll anyway); the choice costs {additive - floor} idle GPUs")

# --- (7) Adversarial: the max flips once a failure domain exceeds one group. With a rack
# of 2 replicas as the domain, failure dominates and the rollout headroom is already paid.
floor2, _ = required(GROUPS, GROUP_GPUS, 2)
assert floor2 == STEADY + 2 * GROUP_GPUS == 256
print(f"7  rack domain = 2 groups: floor {floor2} GPUs; rollout headroom already covered "
      "by failure headroom, so the rollout costs nothing extra")

print("all assertions passed")

Executed output:

0  group size 4 pods, group footprint 32 GPUs, fleet 192 GPUs
1  group-atomic surge, 32 spare GPUs -> peak 224, mixed-version serving groups 0, stalled False
2  pod-granular surge, 8 spare GPUs -> peak 200, mixed-version windows 18 (a mixed TP group is a broken collective)
3  spare demanded: pod view 8 GPUs, group view 32 GPUs -> understated by 4x, exactly the group size in pods
4  every spare from 0 to 24 GPUs -> stalled, 0 mixed, peak still 192: the requirement is a step at one whole group, not a gradient
5  over 120 (fleet, pods-per-group, gpus-per-pod) shapes: group-atomic mixed count is always 0; pod-granular is always groups*(pods-1)
6  N+1 domain: floor 224 GPUs (defer the rollout during an outage) vs 256 (roll anyway); the choice costs 32 idle GPUs
7  rack domain = 2 groups: floor 256 GPUs; rollout headroom already covered by failure headroom, so the rollout costs nothing extra
all assertions passed

Four things to take from it.

Case 0 separates the two numbers that get conflated. The LWS group size is 4, the number of pods; the group footprint is 32 GPUs. Surge is sized in groups, and the understatement factor in case 3 is the size in pods, not the GPU count.

Case 2 is the important one. The pod-granular rollout does not fail, it completes, and it produces 18 windows in which a live replica held two model versions in one collective. Both strategies are inspected by the same function, which asks only whether a group that is currently taking traffic holds more than one version; the difference is measured rather than assumed. Case 5 sweeps 120 fleet and group shapes and the two counts hold at 0 and groups * (pods - 1) throughout.

Case 4 is the behaviour you want when capacity is short: the rollout stalls with the fleet intact and nothing stranded, rather than committing GPUs to a group that can never reach readiness.

Case 6 and case 7 show the capacity rule is not a constant. A rollout is voluntary, so it maxes against the failure term; once a failure domain is larger than one replica, the rollout headroom is already paid for. A demand surge is not voluntary and does not belong in that max, which is why GPU pool segmentation handles the burst term separately and adds it.

If the estate cannot hold even one surge group, the options are explicit and none of them is silent: reduce redundancy for the window, buy burst capacity, take a maintenance window, or buy headroom. Deciding by accident, and discovering it when a rollout stalls at three in the morning, is the outcome to avoid.

How to maintain it

  • Keep one gang mechanism. Kueue or Volcano, not both, over the same nodes. They are not the same kind of component (Kueue gates admission and quota and leaves placement to the Kubernetes scheduler, while Volcano supplies a scheduler of its own), which is exactly why running both over one pool gives you two things deciding whether a workload may start, with different views of free capacity.
  • Re-derive the surge unit when the shape changes. Going from TP=8 to TP=16 doubles the spare capacity a rollout needs. That number belongs in the capacity model (GPU capacity planning), not in someone's memory.
  • Watch pending groups as a first-class signal. kueue_pending_workloads and Volcano's pending PodGroup gauge tell you that groups are waiting; neither carries a reason or topology label, so they cannot by themselves distinguish "no capacity" from "no capacity in an acceptable topology". For that, read the Workload conditions and the scheduler events on a pending group (SLOs: training platform wires the queue-wait alert).
  • Cap how long a group may stay unschedulable. A group waiting for a topology that cannot become available blocks its queue. Give it an admission timeout and let smaller work backfill around it, rather than letting one request hold a queue open indefinitely.
  • Re-qualify each shape after a platform change. A driver, NCCL or engine bump changes the sustainable rate of a profile. Re-measure before the router keeps using the old weighting (fabric performance regression).

Failure modes

  • Half a group scheduled, GPUs held, nothing served. No gang admission, or the workload was submitted as a plain Deployment. Two of these deadlock each other.
  • Group admitted, ranks across a slow boundary. Gang admission without a topology constraint. The replica works and is slow forever; it shows up as a latency regression with no serving-config cause.
  • Endpoint receives traffic before ranks joined. Pod readiness used as group readiness. First requests fail or hang until initialisation finishes.
  • Mixed model versions in one collective. Pod-granular rolling update on a group. Symptoms range from a hard shape mismatch to silently wrong output, which is the worse case.
  • Leader restarted in place, group never recovers. The process group tore down with it; the surviving pods are waiting for a rendezvous that will not happen.
  • Rollout stalls at the first group. Surge sized in pods rather than groups. The stall is the safe outcome; the capacity model was the defect.
  • A "spare" node cannot host the group. Free GPUs exist but are fragmented across domains. Aggregate free capacity is not schedulable capacity for a gang.
  • Preemption deletes the group and nothing rebuilds it. Bare pod groups under Kueue, which explicitly does not recreate pods. Run a controller that owns the group.

References

  • LeaderWorkerSet, README at v0.10.0 (multi-host inference, alpha gang scheduling): https://github.com/kubernetes-sigs/lws/blob/v0.10.0/README.md
  • LeaderWorkerSet API types at v0.10.0 (size and restartPolicy under leaderWorkerTemplate; rollout defaults): https://github.com/kubernetes-sigs/lws/blob/v0.10.0/api/leaderworkerset/v1/leaderworkerset_types.go
  • LeaderWorkerSet gang scheduling with Volcano at v0.10.0 (disabled by default; provider, RBAC, generated PodGroups): https://github.com/kubernetes-sigs/lws/blob/v0.10.0/docs/examples/sample/gang-scheduling/README.md
  • Kueue, running a group of plain Pods: https://kueue.sigs.k8s.io/docs/tasks/run/plain_pods/
  • Kueue, LeaderWorkerSet integration: https://kueue.sigs.k8s.io/docs/tasks/run/leaderworkerset/
  • Kueue, topology-aware scheduling: https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/
  • Volcano PodGroup (minMember, minResources): https://volcano.sh/docs/concepts/podgroup/
  • Volcano PodGroup and network-topology API types at v1.15.1 (mode default, highestTierName exclusivity): https://github.com/volcano-sh/volcano/blob/v1.15.1/staging/src/volcano.sh/apis/pkg/apis/scheduling/v1beta1/types.go
  • Volcano network-topology-aware scheduling (HyperNode, hard/soft): https://volcano.sh/docs/keyfeatures/networktopologyaware/
  • Kubernetes pod topology spread constraints: https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/
  • vLLM parallelism and scaling (tensor and pipeline parallel across nodes): https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html

Related: Inference system map · Inference parallelism · Gang-scheduled training · Volcano · Kueue quota · Topology-unaware scheduling · Node-level topology · Replica loss · GPU capacity planning · Glossary

  1. kubernetes-sigs/lws README and API types, read at v0.10.0. spec.rolloutStrategy.rollingUpdateConfiguration defaults to maxUnavailable: 1 and maxSurge: 0 (kubebuilder markers on the type, and the defaulting webhook fills the block when it is absent). the project targets "multi-host inference workloads where the LLM will be sharded and run across multiple devices on multiple nodes"; gang scheduling is marked "Alpha level, API may change in the future"; size and restartPolicy sit under spec.leaderWorkerTemplate, and restartPolicy defaults to RecreateGroupOnPodRestart, with the former Default value deprecated in favour of None. https://github.com/kubernetes-sigs/lws ↩↩↩

  2. kubernetes-sigs/lws at v0.10.0: gang scheduling "is disabled by default" and is enabled by setting gangSchedulingManagement.schedulerProvider, uncommenting the Volcano RBAC component and restarting the controller. pkg/schedulerprovider/interface.go declares SupportedSchedulerProviders = sets.New("volcano"), so Volcano is the only provider; volcano_provider.go creates the PodGroup, and GetPodGroupName formats it as <lwsName>-<groupIndex>-<revision>. https://github.com/kubernetes-sigs/lws/blob/v0.10.0/docs/examples/sample/gang-scheduling/README.md ↩

  3. Kueue's LeaderWorkerSet integration is "based on the Plain Pod Group integration" and "is enabled by default". https://kueue.sigs.k8s.io/docs/tasks/run/leaderworkerset/ ↩↩

  4. Kueue plain-pod groups: the group is marked with the kueue.x-k8s.io/pod-group-name label and the kueue.x-k8s.io/pod-group-total-count annotation; "Kueue does not re-create failed Pods", and on preemption "kueue sends delete requests for all of the Pods in the group. It is the responsibility of the user or controller that created the original Pods to create replacement Pods." https://kueue.sigs.k8s.io/docs/tasks/run/plain_pods/ ↩

  5. Kueue topology-aware scheduling is documented as beta and "enabled by default", with kueue.x-k8s.io/podset-required-topology and kueue.x-k8s.io/podset-preferred-topology selecting hard and soft domain constraints: https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/ . The Topology object itself is described on a separate page, which is where the claims that it is cluster-scoped, that it is kueue.x-k8s.io/v1beta2, and that "the kubernetes.io/hostname label can only be used at the lowest (last) level" come from: https://kueue.sigs.k8s.io/docs/concepts/topology/ ↩↩

  6. Volcano PodGroup: minMember is "the minimum number of pods or tasks running under the PodGroup. If the cluster resource cannot meet the demand ... no pod or task in the PodGroup will be scheduled"; minResources applies the same rule to aggregate resources. https://volcano.sh/docs/concepts/podgroup/ ↩

  7. Kubernetes pod topology spread constraints: DoNotSchedule "tells the scheduler not to schedule it", and the known limitations state "There's no guarantee that the constraints remain satisfied when Pods are removed." https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/ ↩