Skip to content
Markdown

Kubernetes GPU-node resource isolation

Scope: how a Kubernetes GPU node keeps system daemons, the kubelet, and neighboring pods from starving a GPU pod's CPU and memory (and vice versa), via Node Allocatable carve-outs (kubeReserved, systemReserved, reservedSystemCPUs), Guaranteed QoS plus the CPU/Memory Manager, node-pressure eviction thresholds, taints, cgroup v2 controls, and kubelet swap behavior.

What it is

Resource isolation on a GPU node is the set of kubelet- and scheduler-level mechanisms that partition a node's CPU, memory, and (partially) I/O between three tenants that must not interfere with each other: the OS and its daemons (sshd, udev, journald), Kubernetes' own agents (kubelet, container runtime), and the pods a user schedules. Without this partitioning, a single noisy pod, or the normal creep of system-daemon memory, can starve the kubelet itself or push a GPU training pod's CPU-bound data loader off its cores.

The primary mechanisms, in the order the kubelet applies them:

  • Node Allocatable: capacity minus reservations minus the hard eviction buffer, the number the scheduler actually bin-packs pods against.1
  • kubeReserved / systemReserved / reservedSystemCPUs: kubelet config fields that carve out CPU, memory, and PID headroom for Kubernetes daemons and the OS respectively.1
  • Guaranteed QoS + CPU Manager static + Memory Manager Static: the pod-level opt-in that turns a soft CFS-quota share into exclusive, pinned cores and protected memory, covered in depth in Topology-Aware GPU Scheduling in Kubernetes.45
  • Node-pressure eviction (evictionHard / evictionSoft): the kubelet's last line of defense, killing pods before the node itself runs out of memory or disk.3
  • Taints and tolerations: keep non-GPU workloads off expensive GPU nodes so they cannot compete for that node's CPU/memory at the scheduling level at all.8
  • cgroup v2 io.max / io.weight: kernel primitives for I/O isolation that Kubernetes does not yet expose as a pod-spec field, only as requests/limits for CPU and memory.910
  • memorySwap.swapBehavior (NoSwap / LimitedSwap): whether Burstable pods may spill to swap under memory pressure instead of being killed outright.1112

Why it matters

A GPU node's economics are dominated by the GPU, not the CPU or memory: an idle H100 because the kubelet OOM-killed the data-loader pod's neighbor, or throttled the kubelet itself out of CPU time and lost the node from the cluster, is a far more expensive failure than the equivalent event on a plain CPU node. Reservation and eviction settings exist precisely to keep the two "invisible" tenants (OS daemons, kubelet/container-runtime) alive under load so the visible tenant (the GPU pod) keeps running. Guaranteed QoS and the CPU/Memory Manager static policies close the other direction: they stop a different pod on the same node from stealing the cycles or memory bandwidth a latency-sensitive GPU pipeline depends on. This page is the isolation and reservation half of that story; Topology-Aware GPU Scheduling in Kubernetes is the alignment half (making sure the reserved/pinned resources are also on the same NUMA node as the GPU).

When it is needed (and when not)

Needed:

  • Any production GPU node, whether single-GPU inference or multi-GPU training: unreserved system daemons on a memory-dense GPU box are exactly as vulnerable to eviction storms as any other node, and the failure mode (losing the node, not just a pod) is worse when it is carrying an expensive accelerator.
  • Mixed clusters with GPU and non-GPU node pools, where taints are the only thing stopping a CPU-only batch job from landing on, and hogging CPU/memory on, a GPU node.
  • Guaranteed-QoS GPU pods that also want CPU Manager static pinning (per Topology-Aware GPU Scheduling in Kubernetes): the pod's requests/limits design and the node's kubeReserved/systemReserved/reservedSystemCPUs sizing have to agree, or the static policy either fails to start or leaves too small a shared pool for everything else.

Not needed / low value:

  • LimitedSwap on a GPU training node: it only helps Burstable pods, and a Guaranteed-QoS GPU pod (the common case) gets no swap allocation under the KEP's design, so enabling swap buys nothing for the pod that matters and adds tail-latency risk for the rest.12
  • cgroup v2 io.max/io.weight at the pod-spec level: do not design an architecture around per-pod Kubernetes-native I/O limits, that field does not exist yet; solve pod-level I/O contention at the container-runtime layer (containerd NRI plugin, CRI-O BlockIOClass) or with node-level nodefs/imagefs separation instead.910
  • enforceNodeAllocatable: [pods, kube-reserved, system-reserved] without first watching actual daemon usage: enforcing reservations as hard cgroup ceilings before you know real consumption can OOM-kill the kubelet or container runtime instead of protecting it.1

Architecture

flowchart TB
    CAP["Node Capacity<br/>(all CPU / memory / PIDs)"]
    KR["kubeReserved<br/>(kubelet, container runtime)"]
    SR["systemReserved<br/>(sshd, udev, journald, kernel)"]
    RSC["reservedSystemCPUs<br/>(explicit CPU list, takes<br/>precedence over kubeReserved/<br/>systemReserved cpu component)"]
    EV["evictionHard buffer<br/>(memory.available, nodefs, imagefs)"]
    ALLOC["Node Allocatable<br/>= Capacity - kubeReserved<br/>- systemReserved - evictionHard"]
    SCHED["Scheduler bin-packs<br/>against Allocatable"]
    TAINT["GPU-node taint<br/>nvidia.com/gpu=present:NoSchedule"]
    TOL["GPU pod toleration"]
    QOS["Guaranteed QoS<br/>(requests == limits, CPU+mem)"]
    CPUM["CPU Manager static<br/>exclusive core pinning"]
    MEMM["Memory Manager Static<br/>reservedMemory per NUMA node"]
    PODCG["Pod cgroup<br/>(cpuset + memory.min/max)"]
    GPUPOD["GPU pod runs isolated"]
    NPE["Node-pressure eviction<br/>kills BestEffort, then Burstable<br/>over-request, before Guaranteed"]

    CAP --> ALLOC
    KR --> ALLOC
    SR --> ALLOC
    RSC -.->|substitutes cpu portion| KR
    EV --> ALLOC
    ALLOC --> SCHED
    TAINT --> TOL
    TOL --> SCHED
    SCHED --> QOS
    QOS --> CPUM
    QOS --> MEMM
    CPUM --> PODCG
    MEMM --> PODCG
    PODCG --> GPUPOD
    EV -.->|breach triggers| NPE
    NPE -.->|protects| GPUPOD

Node Allocatable is the gate the scheduler sees; taints are the gate that keeps the wrong pods off the node entirely; Guaranteed QoS plus the CPU/Memory Manager is what turns a scheduled pod's requests into an actual pinned, protected cgroup; and eviction is the backstop that fires if all of that under-provisions and the node still runs out of resources.

How: implement, integrate, maintain

1. Size Node Allocatable: kubeReserved, systemReserved, evictionHard

Node Allocatable is computed by the kubelet as:

Allocatable = Capacity - kubeReserved - systemReserved - evictionHard

kubeReserved covers Kubernetes system daemons (kubelet, container runtime); systemReserved covers non-Kubernetes OS daemons; evictionHard is held back, not consumed, as the buffer the kubelet must still have free before it starts evicting pods.1

Worked example, a 128-vCPU / 1024Gi GPU node:

# /var/lib/kubelet/config.yaml (KubeletConfiguration)
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
kubeReserved:
  cpu: "2"
  memory: 8Gi
systemReserved:
  cpu: "1"
  memory: 4Gi
evictionHard:
  memory.available: "100Mi"      # kubelet default; shown explicitly here
  nodefs.available: "10%"        # kubelet default
  nodefs.inodesFree: "5%"        # kubelet default
  imagefs.available: "15%"       # kubelet default
enforceNodeAllocatable:
  - pods                          # start here; add kube-reserved/system-reserved after watching usage

By hand: Capacity is 128 CPU and 1024Gi = 1,048,576Mi memory.

  • CPU Allocatable = 128 - 2 (kubeReserved) - 1 (systemReserved) = 125 CPU.
  • Memory Allocatable = 1,048,576Mi - 8,192Mi (kubeReserved, 8Gi) - 4,096Mi (systemReserved, 4Gi) - 100Mi (evictionHard) = 1,036,188Mi ≈ 1,011.91Gi.

evictionHard.memory.available: 100Mi, nodefs.available: 10%, nodefs.inodesFree: 5%, and imagefs.available: 15% are the kubelet's built-in defaults; only memory and ephemeral-storage are supported eviction resources.13 Soft eviction (evictionSoft) has no built-in default; it is opt-in and must be paired with evictionSoftGracePeriod (per-signal grace period before eviction fires) and typically evictionMaxPodGracePeriod.3

enforceNodeAllocatable (default [pods]) controls whether the kubelet also creates and enforces the kube-reserved/system-reserved cgroup ceilings themselves; adding kube-reserved or system-reserved requires setting kubeReservedCgroup/systemReservedCgroup to an existing cgroup, and the kubelet will fail to start if that cgroup does not exist.1

2. Pin CPUs to system daemons with reservedSystemCPUs

Instead of (not in addition to) the cpu field inside kubeReserved/systemReserved, the kubelet accepts an explicit CPU list. It takes precedence: "the explicit CPU list specified by --reserved-cpus takes precedence over the CPU reservation specified by --kube-reserved and --system-reserved."2 This is the option to use whenever CPU Manager static is also enabled, since the static policy requires a non-zero CPU reservation (via one path or the other) or its shared pool would be emptied by exclusive allocations.2

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
reservedSystemCPUs: "0-3"   # 4 physical cores reserved for OS + Kubernetes daemons
cpuManagerPolicy: static

On the same 128-CPU node, reservedSystemCPUs: "0-3" reserves 4 cores, leaving 124 CPUs in the CPU Manager's exclusively-allocatable shared pool (128 - 4), separate from the memory-side Node Allocatable arithmetic above. Reserved CPUs are taken, in integer quantity, from the shared pool in ascending physical-core-ID order.2

3. Make the GPU pod Guaranteed QoS, and compose CPU Manager + Memory Manager

Guaranteed QoS, CPU Manager static, and Memory Manager Static are already specified in detail in Topology-Aware GPU Scheduling in Kubernetes; this page only adds the reservation angle: reservedMemory (required once memoryManagerPolicy: Static is set) must sum, across NUMA nodes, to kubeReserved + systemReserved + evictionHard[memory.available], or the kubelet fails to start.6 That is the same reservation arithmetic from step 1, now also fenced off per-NUMA-node for the Memory Manager. A pod only gets exclusive cores and protected memory once it is Guaranteed (every container's CPU and memory requests equal limits); Burstable and BestEffort pods draw from whatever the static/exclusive allocations left in the shared pool.45

apiVersion: v1
kind: Pod
metadata:
  name: gpu-trainer
spec:
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"
  containers:
    - name: train
      image: nvcr.io/nvidia/pytorch:25.04-py3
      resources:
        requests:
          cpu: "16"
          memory: 64Gi
          nvidia.com/gpu: "4"
        limits:                # requests == limits on both CPU and memory -> Guaranteed
          cpu: "16"
          memory: 64Gi
          nvidia.com/gpu: "4"

An optional, still-immature layer under this: the kubelet's cgroup v2 Memory QoS feature (FEATURE STATE: Kubernetes v1.22 [alpha], disabled by default) sets memory.high on Burstable containers to throttle before they hit memory.max, computed as memory.high = requests + memoryThrottlingFactor * (limits - requests) with memoryThrottlingFactor defaulting to 0.9; Guaranteed containers get no memory.high because requests already equal limits.7 Because it is still alpha years after introduction, do not depend on it for hard isolation guarantees; use Guaranteed QoS and the Memory Manager for that.

4. Keep other workloads off the GPU node with taints

Taints stop the scheduler from placing non-tolerating pods on a node at all (NoSchedule), or evict already-running ones (NoExecute, with an optional tolerationSeconds grace window), or merely discourage placement (PreferNoSchedule, best-effort only).8

kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule

tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

nvidia.com/gpu=present:NoSchedule is a widely used convention, not something the NVIDIA GPU Operator itself applies. Google Kubernetes Engine, for example, auto-applies exactly this taint (and an auto-toleration via the ExtendedResourceToleration admission controller) to GPU node pools, but only when the cluster also has at least one non-GPU node pool; it is not retroactively applied if a non-GPU pool is added later.13 On self-managed clusters, apply the taint yourself and give every GPU-requesting pod spec an explicit toleration; do not assume the toleration is automatic unless your platform documents that admission-controller behavior.

5. cgroup v2 I/O: what Kubernetes does and does not isolate

cgroup v2 exposes a unified io controller (io.max for hard IOPS/bandwidth caps, io.weight for proportional share), succeeding cgroup v1's blkio.9 Kubernetes uses cgroup v2 for CPU and memory requests/limits and for the alpha Memory QoS feature above, but it does not expose a pod-spec field for io.max or io.weight; a 2018 upstream issue tracking per-pod blkio/IO limiting was closed as stale without a fix landing, and no such field has shipped since.10 Do not architect around a Kubernetes-native I/O guarantee. The available levers are: separate nodefs/imagefs filesystems so image churn cannot starve workload I/O (both have dedicated eviction signals, see step 1), container-runtime-specific mechanisms (containerd NRI plugins, CRI-O BlockIOClass), or dedicated NVMe/local-SSD node pools per workload class.

6. NoSwap versus LimitedSwap

memorySwap.swapBehavior in the KubeletConfiguration chooses whether Kubernetes workloads may use swap at all; failSwapOn: false must also be set, or the kubelet refuses to start on a node with swap enabled.11 NoSwap is the default: "workloads running as Pods on this node do not and cannot use swap," though non-Kubernetes processes, including the kubelet itself, still can.11 LimitedSwap lets pods use swap, but only Burstable-QoS pods that are not high-priority, static, or mirrored; Guaranteed pods get none (their memory is already fully reserved) and BestEffort pods get none (they are the first evicted under pressure anyway).12 The per-container swap ceiling is proportional to its memory request's share of node memory: container swap limit = (container memory request / node memory) * available swap.12

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failSwapOn: false
memorySwap:
  swapBehavior: LimitedSwap   # or NoSwap (default)

Swap support progressed alpha (v1.22) to beta1 (v1.28) to beta2 (v1.30, which also removed the earlier UnlimitedSwap option in favor of just NoSwap/LimitedSwap) per the upstream KEP.12 Verify the feature's current stage and default against the exact minor version you run; it is one of the faster-moving kubelet features and this page does not assert it has reached GA. For a Guaranteed-QoS GPU pod specifically, swap changes nothing: it is not eligible either way, so NoSwap is the right default for GPU training/inference nodes and LimitedSwap only matters for co-located Burstable sidecars or batch jobs on the same node.

Maintain

  • Re-derive the Node Allocatable arithmetic (step 1) whenever node instance types change; kubeReserved/systemReserved sized for one SKU's daemon footprint does not transfer to a bigger or smaller node.
  • Watch enforceNodeAllocatable rollout carefully: adding kube-reserved/system-reserved enforcement after only pods was enforced can newly OOM-kill a daemon that was previously allowed to burst past its reservation.1
  • Audit taints after every managed-Kubernetes node-pool change (adding a first non-GPU pool, migrating GPU pools) since some platforms only apply the GPU taint at pool-creation time, not retroactively.13
  • Track the swap feature gate's stage on upgrade; a cluster that relied on the removed UnlimitedSwap behavior needs its kubelet config migrated to LimitedSwap or NoSwap explicitly.12

Failure modes

  • Under-reserved kubeReserved/systemReserved: the kubelet or container runtime gets starved under load, the node goes NotReady, and every GPU pod on it is rescheduled, an outage far more expensive than a plain-CPU-node equivalent.
  • reservedSystemCPUs sized smaller than actual daemon usage: system daemons spill into the CPU Manager's shared pool, jittering the "shared" CPUs that Burstable/BestEffort pods (and any un-pinned sidecars) rely on.
  • Guaranteed QoS misconfigured as Burstable (limits set but requests omitted, or requests != limits): the pod silently loses CPU pinning and Memory Manager protection; nothing rejects the pod, it just does not get the isolation the operator assumed.
  • Missing GPU-node toleration on a non-GPU-requesting pod, combined with a missing or removed taint: cheap CPU-only pods land on the GPU node and compete for the same CPU/memory reservation the GPU pod needs.
  • evictionHard set too tight (or left at conservative defaults) on a memory-dense GPU node: the kubelet evicts BestEffort/Burstable neighbors well before the node is actually in danger, wasting headroom; too loose, and the kubelet has no runway to react before real memory exhaustion.
  • Assuming pod-level I/O isolation exists: without a runtime-specific mechanism, one pod's checkpoint-write burst or image pull can starve another pod's I/O on the same nodefs/imagefs, since Kubernetes has no native io.max/io.weight pod field.10
  • Enabling LimitedSwap expecting it to help a Guaranteed GPU pod: it structurally cannot, only Burstable pods are eligible, so it adds swap-thrash risk elsewhere on the node for zero benefit to the workload that matters.12

References

Related: Topology-Aware GPU Scheduling in Kubernetes · NUMA Affinity and CPU Pinning for GPU Pipelines · GPU Containerization Performance · Glossary

  1. Kubernetes docs, "Reserve Compute Resources for System Daemons": Allocatable = Capacity - kubeReserved - systemReserved - evictionHard; evictionHard defaults memory.available: 100Mi, nodefs.available: 10%, nodefs.inodesFree: 5%, imagefs.available: 15%; only memory and ephemeral-storage are supported eviction resources; enforceNodeAllocatable defaults to [pods], valid values [pods]/[pods, system-reserved, kube-reserved]; kubeReservedCgroup/systemReservedCgroup must reference an existing cgroup or the kubelet fails to start. https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ ↩↩↩↩↩↩↩

  2. Kubernetes docs, "Control CPU Management Policies on the Node": CPU Manager static policy assigns exclusive CPUs only to Guaranteed-QoS containers with integer CPU requests; --reserved-cpus (kubelet config reservedSystemCPUs) takes precedence over --kube-reserved/--system-reserved for the CPU reservation; a non-zero reservation via one of these paths is required when static is enabled; reserved CPUs are taken from the shared pool in ascending physical-core-ID order. https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/ ↩↩↩↩

  3. Kubernetes docs, "Node-pressure Eviction": hard eviction thresholds trigger immediate eviction (no PodDisruptionBudget respect, 0s grace period); soft thresholds (evictionSoft) require evictionSoftGracePeriod; evictionPressureTransitionPeriod default 5m; memory.available is computed from cgroupfs, not free -m, and respects Node Allocatable. https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/ ↩↩↩

  4. Kubernetes docs, "Pod Quality of Service Classes": Guaranteed requires every container's CPU and memory requests to equal limits, both greater than zero; Burstable is any pod with at least one request/limit that does not qualify as Guaranteed; BestEffort has none at all; BestEffort pods are preferentially evicted under node pressure. https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/ ↩↩

  5. See 2 and Topology-Aware GPU Scheduling in Kubernetes for how CPU Manager static and Memory Manager Static compose with Guaranteed QoS and the Topology Manager. ↩↩

  6. Kubernetes docs, "Memory Manager": memoryManagerPolicy: Static requires reservedMemory per NUMA node, whose sum across all nodes must equal kubeReserved + systemReserved + evictionHard[memory.available]; only affects Guaranteed-QoS pods. https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/ ↩

  7. Kubernetes docs, "Pod Quality of Service Classes," Memory QoS with cgroup v2 section: FEATURE STATE: Kubernetes v1.22 [alpha] (disabled by default); Burstable containers get memory.high = requests + memoryThrottlingFactor * (limits - requests), memoryThrottlingFactor defaults to 0.9; Guaranteed containers get no memory.high since requests equal limits; requires cgroup v2, kernel 5.9+ recommended. https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/ ↩↩

  8. Kubernetes docs, "Taints and Tolerations": NoSchedule blocks new non-tolerating pods but does not evict running ones; NoExecute evicts non-tolerating pods immediately (or after tolerationSeconds); PreferNoSchedule is a soft, non-guaranteed preference; default toleration operator is Equal. https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ ↩↩

  9. Kubernetes docs, "About cgroup v2": cgroup v2 requires Linux kernel 5.8+ (containerd 1.4+ / CRI-O 1.20+, systemd cgroup driver recommended); the page notes the MemoryQoS feature relies on cgroup v2 memory primitives (see 7 for its actual v1.22 [alpha] stage, the authoritative source); the page does not describe a pod-spec-level io.max/io.weight field. https://kubernetes.io/docs/concepts/architecture/cgroups/ ↩↩↩

  10. kubernetes/kubernetes issue #70364, "Limiting blkio with cgroups by annotation": tracks the long-standing absence of a native, portable Kubernetes field for per-pod blkio/IO (cgroup v1 blkio / cgroup v2 io) weight or IOPS limits; opened 2018-10-29, closed 2019-03-28 by the stale-issue bot without a fix landing, and no such field has shipped in Kubernetes since. https://github.com/kubernetes/kubernetes/issues/70364 ↩↩↩↩

  11. Kubernetes docs, "Swap Memory Management": memorySwap.swapBehavior field; NoSwap (default): "workloads running as Pods on this node do not and cannot use swap," though non-Kubernetes processes and the kubelet itself still can; LimitedSwap: "Kubernetes workloads can utilize swap memory"; failSwapOn: false required for the kubelet to start on a swap-enabled node. https://kubernetes.io/docs/concepts/cluster-administration/swap-memory-management/ ↩↩↩

  12. Kubernetes Enhancement Proposal sig-node/2400: swap access under LimitedSwap is granted only to Burstable-QoS pods, excluding high-priority, static, and mirrored pods; Guaranteed pods are excluded because their memory is already fully reserved, BestEffort pods are excluded since they are first evicted under pressure; per-container swap limit is proportional to (container memory request / node memory) * available swap; feature progressed alpha (v1.22) to beta1 (v1.28) to beta2 (v1.30, which also removed the earlier UnlimitedSwap option). https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2400-node-swap/README.md ↩↩↩↩↩↩↩

  13. Google Cloud, "Run GPUs in GKE Standard node pools": GKE automatically applies the nvidia.com/gpu=present:NoSchedule taint (and a matching auto-toleration via the ExtendedResourceToleration admission controller) to GPU node pools, but only when the cluster already has at least one non-GPU node pool, and not retroactively if a non-GPU pool is added afterward. https://cloud.google.com/kubernetes-engine/docs/how-to/gpus ↩↩