Copy for agents Markdown

Kubernetes GPU-node resource isolation

Scope: how a Kubernetes GPU node keeps system daemons, the kubelet, and neighboring pods from starving a GPU pod's CPU and memory (and vice versa), via Node Allocatable carve-outs (kubeReserved, systemReserved, reservedSystemCPUs), Guaranteed QoS plus the CPU/Memory Manager, node-pressure eviction thresholds, taints, cgroup v2 controls, and kubelet swap behavior.

Independently verified here: a real kind cluster (Kubernetes v1.31.0, containerd) was bootstrapped with a KubeletConfiguration implementing this page's exact policy (cpuManagerPolicy: static, memoryManagerPolicy: Static, reservedMemory, reserved-cpus, kubeReserved, systemReserved, evictionHard), not a hand-simulated model of one. This surfaced a real, reproducible defect this page previously did not carry a caveat for: kubelet v1.31.0 rejects the strict-cpu-reservation CPU Manager policy option outright and crash-loops (unknown CPU Manager Policy option: "strict-cpu-reservation"), documented below with the exact log line. With that option removed, the resulting Allocatable, a Guaranteed pod's exclusive CPU pinning, its NUMA memory reservation, an oversized-pod scheduling rejection, and a cordon/drain/uncordon cycle were all exercised against the live node and its real cpu_manager_state/memory_manager_state files and cgroupfs cpuset.cpus, not asserted from documentation. See "Executed: rendering and validating the policy on a live node" below for the full transcript.

What it is

Resource isolation on a GPU node is the set of kubelet- and scheduler-level mechanisms that partition a node's CPU, memory, and (partially) I/O between three tenants that must not interfere with each other: the OS and its daemons (sshd, udev, journald), Kubernetes' own agents (kubelet, container runtime), and the pods a user schedules. Without this partitioning, a single noisy pod, or the normal creep of system-daemon memory, can starve the kubelet itself or push a GPU training pod's CPU-bound data loader off its cores.

The primary mechanisms, in the order the kubelet applies them:

• Node Allocatable: capacity minus reservations minus the hard eviction buffer, the number the scheduler actually bin-packs pods against.¹
• kubeReserved / systemReserved / reservedSystemCPUs: kubelet config fields that carve out CPU, memory, and PID headroom for Kubernetes daemons and the OS respectively.¹
• Guaranteed QoS + CPU Manager static + Memory Manager Static: the pod-level opt-in for exclusive CPUs and NUMA-aware memory placement, covered in depth in Topology-Aware GPU Scheduling in Kubernetes. Memory Manager placement does not create memory.min reclaim protection; that is a separate Memory QoS feature.⁴⁵⁸
• Node-pressure eviction (evictionHard / evictionSoft): the kubelet's last line of defense, killing pods before the node itself runs out of memory or disk.³
• Taints and tolerations: keep non-GPU workloads off expensive GPU nodes so they cannot compete for that node's CPU/memory at the scheduling level at all.⁹
• cgroup v2 io.max / io.weight: kernel primitives for I/O isolation that Kubernetes does not yet expose as a pod-spec field, only as requests/limits for CPU and memory.¹⁰¹¹
• memorySwap.swapBehavior (NoSwap / LimitedSwap): whether Burstable pods may spill to swap under memory pressure instead of being killed outright.¹²¹³

Why use it

A GPU node's economics are dominated by the GPU, not the CPU or memory: an idle H100 because the kubelet OOM-killed the data-loader pod's neighbor, or throttled the kubelet itself out of CPU time and lost the node from the cluster, is a far more expensive failure than the equivalent event on a plain CPU node. Reservation and eviction settings keep the two "invisible" tenants (OS daemons, kubelet/container-runtime) alive under load so the GPU pod keeps running. Guaranteed QoS and CPU Manager can give a workload exclusive CPUs; Memory Manager aligns its allocation to NUMA nodes. Reclaim protection and memory-bandwidth isolation require separate mechanisms and must not be inferred from placement. This page is the isolation and reservation half of that story; Topology-Aware GPU Scheduling in Kubernetes is the alignment half.

When to use it (and when not)

Needed:

• Any production GPU node, whether single-GPU inference or multi-GPU training: unreserved system daemons on a memory-dense GPU box are exactly as vulnerable to eviction storms as any other node, and the failure mode (losing the node, not just a pod) is worse when it is carrying an expensive accelerator.
• Mixed clusters with GPU and non-GPU node pools, where taints are the only thing stopping a CPU-only batch job from landing on, and hogging CPU/memory on, a GPU node.
• Guaranteed-QoS GPU pods that also want CPU Manager static pinning (per Topology-Aware GPU Scheduling in Kubernetes): the pod's requests/limits design and the node's kubeReserved/systemReserved/reservedSystemCPUs sizing have to agree, or the static policy either fails to start or leaves too small a shared pool for everything else.

Not needed / low value:

• LimitedSwap on a GPU training node: it only helps Burstable pods, and a Guaranteed-QoS GPU pod (the common case) gets no swap allocation under the KEP's design, so enabling swap buys nothing for the pod that matters and adds tail-latency risk for the rest.¹³
• cgroup v2 io.max/io.weight at the pod-spec level: do not design an architecture around per-pod Kubernetes-native I/O limits, that field does not exist yet; solve pod-level I/O contention at the container-runtime layer (containerd NRI plugin, CRI-O BlockIOClass) or with node-level nodefs/imagefs separation instead.¹⁰¹¹
• enforceNodeAllocatable: [pods, kube-reserved, system-reserved] without first watching actual daemon usage: enforcing reservations as hard cgroup ceilings before you know real consumption can OOM-kill the kubelet or container runtime instead of protecting it.¹

Architecture

flowchart TB
    CAP["Node Capacity<br/>(all CPU / memory / PIDs)"]
    KR["kubeReserved<br/>(kubelet, container runtime)"]
    SR["systemReserved<br/>(sshd, udev, journald, kernel)"]
    RSC["reservedSystemCPUs<br/>(explicit CPU list, takes<br/>precedence over kubeReserved/<br/>systemReserved cpu component)"]
    EV["evictionHard buffer<br/>(memory.available, nodefs, imagefs)"]
    ALLOC["Node Allocatable<br/>= Capacity - kubeReserved<br/>- systemReserved - evictionHard"]
    SCHED["Scheduler bin-packs<br/>against Allocatable"]
    TAINT["GPU-node taint<br/>nvidia.com/gpu=present:NoSchedule"]
    TOL["GPU pod toleration"]
    QOS["Guaranteed QoS<br/>(requests == limits, CPU+mem)"]
    CPUM["CPU Manager static<br/>exclusive core pinning"]
    MEMM["Memory Manager Static<br/>reservedMemory per NUMA node"]
    PODCG["Pod cgroup<br/>(cpuset + memory.max;<br/>memory.min only with tiered Memory QoS)"]
    GPUPOD["GPU pod runs isolated"]
    NPE["Node-pressure eviction<br/>kills BestEffort, then Burstable<br/>over-request, before Guaranteed"]

    CAP --> ALLOC
    KR --> ALLOC
    SR --> ALLOC
    RSC -.->|substitutes cpu portion| KR
    EV --> ALLOC
    ALLOC --> SCHED
    TAINT --> TOL
    TOL --> SCHED
    SCHED --> QOS
    QOS --> CPUM
    QOS --> MEMM
    CPUM --> PODCG
    MEMM --> PODCG
    PODCG --> GPUPOD
    EV -.->|breach triggers| NPE
    NPE -.->|protects| GPUPOD

Node Allocatable is the gate the scheduler sees; taints keep the wrong pods off the node; Guaranteed QoS plus CPU Manager turns integral CPU requests into exclusive CPU sets; Memory Manager supplies NUMA placement hints and reservations. Memory reclaim protection is optional Memory QoS behavior, not an automatic consequence of Memory Manager. Eviction is the backstop when reservations are insufficient.

How to use it

1. Size Node Allocatable: kubeReserved, systemReserved, evictionHard

Node Allocatable is computed by the kubelet as:

Allocatable = Capacity - kubeReserved - systemReserved - evictionHard

kubeReserved covers Kubernetes system daemons (kubelet, container runtime); systemReserved covers non-Kubernetes OS daemons; evictionHard is held back, not consumed, as the buffer the kubelet must still have free before it starts evicting pods.¹

Worked example, a 128-vCPU / 1024Gi GPU node:

# /var/lib/kubelet/config.yaml (KubeletConfiguration)
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
kubeReserved:
  cpu: "2"
  memory: 8Gi
systemReserved:
  cpu: "1"
  memory: 4Gi
evictionHard:
  memory.available: "100Mi"      # kubelet default; shown explicitly here
  nodefs.available: "10%"        # kubelet default
  nodefs.inodesFree: "5%"        # kubelet default
  imagefs.available: "15%"       # kubelet default
enforceNodeAllocatable:
  - pods                          # start here; add kube-reserved/system-reserved after watching usage

By hand: Capacity is 128 CPU and 1024Gi = 1,048,576Mi memory.

• CPU Allocatable = 128 - 2 (kubeReserved) - 1 (systemReserved) = 125 CPU.
• Memory Allocatable = 1,048,576Mi - 8,192Mi (kubeReserved, 8Gi) - 4,096Mi (systemReserved, 4Gi) - 100Mi (evictionHard) = 1,036,188Mi ≈ 1,011.91Gi.

evictionHard.memory.available: 100Mi, nodefs.available: 10%, nodefs.inodesFree: 5%, and imagefs.available: 15% are the kubelet's built-in defaults; only memory and ephemeral-storage are supported eviction resources.¹³ Soft eviction (evictionSoft) has no built-in default; it is opt-in and must be paired with evictionSoftGracePeriod (per-signal grace period before eviction fires) and typically evictionMaxPodGracePeriod.³

enforceNodeAllocatable (default [pods]) controls whether the kubelet also creates and enforces the kube-reserved/system-reserved cgroup ceilings themselves; adding kube-reserved or system-reserved requires setting kubeReservedCgroup/systemReservedCgroup to an existing cgroup, and the kubelet will fail to start if that cgroup does not exist.¹

2. Pin CPUs to system daemons with reservedSystemCPUs

Instead of (not in addition to) the cpu field inside kubeReserved/systemReserved, the kubelet accepts an explicit CPU list. It takes precedence: "the explicit CPU list specified by --reserved-cpus takes precedence over the CPU reservation specified by --kube-reserved and --system-reserved."² This is the option to use whenever CPU Manager static is also enabled, since the static policy requires a non-zero CPU reservation (via one path or the other) or its shared pool would be emptied by exclusive allocations.²

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# Example assumes SMT is disabled. With SMT, list every sibling thread of each reserved core.
reservedSystemCPUs: "0-3"   # four logical CPU IDs, verified with lscpu -e
cpuManagerPolicy: static
cpuManagerPolicyOptions:
  full-pcpus-only: "true"         # exclusive assignments contain complete physical cores
  # strict-cpu-reservation: "true" would also prevent workloads from executing on
  # reservedSystemCPUs, but confirm your kubelet build actually accepts it first -- see below.

On the same 128-logical-CPU node, reservedSystemCPUs: "0-3" removes four logical CPUs from exclusive allocation, leaving 124 logical CPUs available to CPU Manager. It does not create an "exclusively-allocatable shared pool": Guaranteed containers with integral requests receive exclusive assignments; Burstable, BestEffort, and fractional-CPU containers use the shared pool. On SMT hosts, derive the list from lscpu -e=CPU,CORE,SOCKET,NODE and reserve complete sibling sets. The ascending physical-core rule applies when kubelet derives an integer reservation; an explicit reservedSystemCPUs list uses the IDs supplied.²

strict-cpu-reservation is not safe to assume works on your kubelet build. Tested directly against a real kubelet v1.31.0 (with featureGates: {CPUManagerPolicyAlphaOptions: true} set, which gates the alpha CPU Manager policy options as a group), setting strict-cpu-reservation: "true" made the kubelet crash-loop on every start:

E ... container_manager_linux.go:328] "Failed to initialize cpu manager" \
    err="new static policy error: unknown CPU Manager Policy option: \"strict-cpu-reservation\""
E ... run.go:72] "command failed" err="failed to run Kubelet: new static policy error: \
    unknown CPU Manager Policy option: \"strict-cpu-reservation\""

The node never reaches Ready in this state; systemctl status kubelet shows a rapidly climbing restart counter. full-pcpus-only alone, with the same feature gate, starts cleanly (see the executed section below). Before shipping a node policy that includes strict-cpu-reservation, confirm your specific kubelet build actually recognizes it (check kubelet --help output or your distribution's release notes for the CPU Manager policy options it ships), and test a real systemctl start kubelet / node-Ready cycle in CI against that exact build, not just a YAML lint. Do not carry this option forward from documentation or an older cluster without that check.

3. Make the GPU pod Guaranteed QoS, and compose CPU Manager + Memory Manager

Guaranteed QoS, CPU Manager static, and Memory Manager Static are specified in detail in Topology-Aware GPU Scheduling in Kubernetes. The reservation constraint is exact: reservedMemory, summed across NUMA nodes, must equal kubeReserved + systemReserved + evictionHard[memory.available], or kubelet startup fails.⁶ Guaranteed containers with integral CPU requests can receive exclusive CPUs, and Memory Manager can align their memory to NUMA nodes. Neither Guaranteed QoS nor Memory Manager alone writes memory.min or reserves memory bandwidth.⁴⁵⁸

apiVersion: v1
kind: Pod
metadata:
  name: gpu-trainer
spec:
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"
  containers:
    - name: train
      image: nvcr.io/nvidia/pytorch:25.04-py3
      resources:
        requests:
          cpu: "16"
          memory: 64Gi
          nvidia.com/gpu: "4"
        limits:                # requests == limits on both CPU and memory -> Guaranteed
          cpu: "16"
          memory: 64Gi
          nvidia.com/gpu: "4"

Memory QoS remains an alpha, disabled-by-default cgroup v2 feature. The original policy can set memory.high for Burstable containers. Kubernetes 1.36 added memoryReservationPolicy: TieredReservation, which maps the calculated pod memory request to memory.min and container requests to memory.low; the default policy is None, which writes neither protection value.⁷⁸ Treat this as reclaim protection, not NUMA placement or memory-bandwidth isolation, and gate it by the exact Kubernetes minor version.

Executed: rendering and validating the policy on a live node

Everything in this subsection was run against a real kind (Kubernetes-in-Docker) v1.31.0 cluster on an 18-logical-CPU, ~94Gi host, not asserted from the formulas above. kind accepts KubeletConfiguration fields through kubeadmConfigPatches, which is how this policy actually reaches the running kubelet:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        reserved-cpus: "0-1"
  - |
    kind: KubeletConfiguration
    cgroupDriver: systemd
    cpuManagerPolicy: static
    cpuManagerPolicyOptions:
      full-pcpus-only: "true"
    cpuManagerReconcilePeriod: 10s
    memoryManagerPolicy: Static
    reservedMemory:
    - numaNode: 0
      limits:
        memory: 1792Mi
    kubeReserved:
      memory: 1Gi
    systemReserved:
      memory: 512Mi
    evictionHard:
      memory.available: "256Mi"
      nodefs.available: "10%"
      nodefs.inodesFree: "5%"
      imagefs.available: "15%"
    featureGates:
      CPUManagerPolicyAlphaOptions: true

kind create cluster --name gpu-node-isolation --config kind-config.yaml

Node Allocatable verification. kubectl describe node reported real Capacity/Allocatable, and both match the formula in step 1 exactly, computed from the node's real 18 CPUs / 98,873,756Ki capacity:

Capacity:                              Allocatable:
  cpu:      18                           cpu:      16
  memory:   98873756Ki                   memory:   97038748Ki

CPU: 18 - 2 (reserved-cpus 0-1) = 16, exact. Memory: 98873756Ki - 1048576Ki (kubeReserved 1Gi) - 524288Ki (systemReserved 512Mi) - 262144Ki (evictionHard 256Mi) = 97038748Ki, exact to the byte, confirming the Allocatable formula in step 1 is not just documentation prose but what a real kubelet actually computes.

CPU and Memory Manager state, and cgroup enforcement, for a real Guaranteed pod. A pod requesting cpu: "4", memory: "2Gi" (requests == limits, QoS Class: Guaranteed confirmed via kubectl get pod -o jsonpath='{.status.qosClass}') was scheduled, and the kubelet's own state files were read directly from the node:

$ cat /var/lib/kubelet/cpu_manager_state
{"policyName":"static","defaultCpuSet":"0-1,6-17",
 "entries":{"<pod-uid>":{"worker":"2-5"}}, ...}

$ cat /var/lib/kubelet/memory_manager_state
{"policyName":"Static","machineState":{"0":{
   "memoryMap":{"memory":{"total":101246726144,"systemReserved":1879048192,
     "allocatable":99367677952,"reserved":2199912448,"free":97167765504}}}},
 "entries":{"<pod-uid>":{"worker":[{"numaAffinity":[0],"type":"memory","size":2147483648}]}, ...}}

The pod got exactly 4 exclusive CPUs (2-5), leaving 0-1,6-17 as the reserved-plus-shared default set. systemReserved in the memory-manager's per-NUMA-cell accounting (1,879,048,192 bytes) is exactly 1792Mi, the reservedMemory value, confirming which config field lands in which accounting bucket. reserved (2,199,912,448 bytes) is exactly the sum of this pod's 2Gi allocation plus a 50Mi kindnet-cni allocation already on the node; free is allocatable - reserved, byte-exact. The real cgroupfs confirms the same number one layer down, at the kernel enforcement point, not just the kubelet's internal bookkeeping:

$ cat /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-pod<uid>.slice/cri-containerd-<id>.scope/cpuset.cpus
2-5

Pressure test: an oversized Guaranteed pod is rejected, not partially scheduled. With the 4-CPU pod still running (12 of 16 Allocatable CPUs free), a second Guaranteed pod requesting cpu: "13" was submitted and correctly refused by the scheduler, not by the kubelet after the fact:

Warning  FailedScheduling  default-scheduler  0/1 nodes are available:
  1 Insufficient cpu. preemption: 0/1 nodes are available:
  1 No preemption victims found for incoming pod.

Drain and rollback. kubectl cordon followed by kubectl drain --ignore-daemonsets --delete-emptydir-data evicted the Guaranteed pod along with CoreDNS and the local-path-provisioner; one pod (local-path-provisioner) exceeded the drain's grace timeout and produced a real, correctly-surfaced partial-failure error (context deadline exceeded) rather than a silent success, which is what a real drain script must handle (retry or extend --timeout, and check kubectl get node for SchedulingDisabled rather than trusting the drain command's own exit code alone). After the Guaranteed pod's eviction completed, cpu_manager_state on the node correctly reverted to {"defaultCpuSet":"0-17"} with no entries, confirming the CPU Manager releases exclusive cores back to the pool on pod termination rather than leaking them. kubectl uncordon restored scheduling.

How to develop with it

Render node policy and workload manifests from the same node-SKU data: logical CPU topology, NUMA memory reservations, daemon measurements, and GPU taints. CI should reject a pod that loses Guaranteed QoS or a kubelet configuration whose reservedMemory sum diverges from the memory reservation equation.

def parse_cpu_list(spec: str) -> set[int]:
    cpus = set()
    for part in spec.split(","):
        bounds = [int(x) for x in part.split("-")]
        if len(bounds) == 1:
            cpus.add(bounds[0])
        elif len(bounds) == 2 and bounds[0] <= bounds[1]:
            cpus.update(range(bounds[0], bounds[1] + 1))
        else:
            raise ValueError(f"invalid CPU list element: {part}")
    return cpus


def validate_reservations(cpu_spec, sibling_groups, reserved_memory_mib,
                          kube_mib, system_mib, eviction_mib):
    reserved = parse_cpu_list(cpu_spec)
    for siblings in sibling_groups:
        overlap = reserved.intersection(siblings)
        if overlap and overlap != set(siblings):
            raise ValueError(f"partial SMT core reservation: {sorted(siblings)}")
    expected_memory = kube_mib + system_mib + eviction_mib
    if sum(reserved_memory_mib) != expected_memory:
        raise ValueError("reservedMemory does not equal kube+system+eviction")
    return len(reserved), expected_memory


# SMT disabled: four logical IDs are four complete cores.
count, memory = validate_reservations(
    "0-3", [{0}, {1}, {2}, {3}], [512, 512], 256, 256, 512
)
assert (count, memory) == (4, 1024)

# SMT enabled: reserving only one thread from each core is rejected.
try:
    validate_reservations(
        "0-3", [{0, 64}, {1, 65}, {2, 66}, {3, 67}], [1024], 256, 256, 512
    )
except ValueError as exc:
    assert "partial SMT core" in str(exc)
else:
    raise AssertionError("accepted partial SMT-core reservation")

# Exact sibling sets pass; a mismatched memory sum does not.
assert validate_reservations(
    "0-3,64-67", [{0, 64}, {1, 65}, {2, 66}, {3, 67}],
    [512, 512], 256, 256, 512
)[0] == 8
try:
    validate_reservations("0", [{0}], [512], 256, 256, 512)
except ValueError as exc:
    assert "reservedMemory" in str(exc)
else:
    raise AssertionError("accepted mismatched reservedMemory")

print("kubelet reservation validation: all asserts passed")

Executed output:

kubelet reservation validation: all asserts passed

Keep other workloads off the GPU node with taints

Taints stop the scheduler from placing non-tolerating pods on a node at all (NoSchedule), or evict already-running ones (NoExecute, with an optional tolerationSeconds grace window), or merely discourage placement (PreferNoSchedule, best-effort only).⁹

kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule

tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

nvidia.com/gpu=present:NoSchedule is a widely used convention, not something the NVIDIA GPU Operator itself applies. Google Kubernetes Engine, for example, auto-applies exactly this taint (and an auto-toleration via the ExtendedResourceToleration admission controller) to GPU node pools, but only when the cluster also has at least one non-GPU node pool; it is not retroactively applied if a non-GPU pool is added later.¹⁴ On self-managed clusters, apply the taint yourself and give every GPU-requesting pod spec an explicit toleration; do not assume the toleration is automatic unless your platform documents that admission-controller behavior.

How to run it in production

cgroup v2 I/O: what Kubernetes does and does not isolate

cgroup v2 exposes a unified io controller (io.max for hard IOPS/bandwidth caps, io.weight for proportional share), succeeding cgroup v1's blkio.¹⁰ Kubernetes uses cgroup v2 for CPU and memory requests/limits and for the alpha Memory QoS feature above, but it does not expose a pod-spec field for io.max or io.weight; a 2018 upstream issue tracking per-pod blkio/IO limiting was closed as stale without a fix landing, and no such field has shipped since.¹¹ Do not architect around a Kubernetes-native I/O guarantee. The available levers are: separate nodefs/imagefs filesystems so image churn cannot starve workload I/O (both have dedicated eviction signals, see step 1), container-runtime-specific mechanisms (containerd NRI plugins, CRI-O BlockIOClass), or dedicated NVMe/local-SSD node pools per workload class.

NoSwap versus LimitedSwap

memorySwap.swapBehavior in the KubeletConfiguration chooses whether Kubernetes workloads may use swap at all; failSwapOn: false must also be set, or the kubelet refuses to start on a node with swap enabled.¹² NoSwap is the default: "workloads running as Pods on this node do not and cannot use swap," though non-Kubernetes processes, including the kubelet itself, still can.¹² LimitedSwap lets pods use swap, but only Burstable-QoS pods that are not high-priority, static, or mirrored; Guaranteed pods get none (their memory is already fully reserved) and BestEffort pods get none (they are the first evicted under pressure anyway).¹³ The per-container swap ceiling is proportional to its memory request's share of node memory: container swap limit = (container memory request / node memory) * available swap.¹³

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failSwapOn: false
memorySwap:
  swapBehavior: LimitedSwap   # or NoSwap (default)

Swap support progressed alpha (v1.22) to beta1 (v1.28) to beta2 (v1.30, which also removed the earlier UnlimitedSwap option in favor of just NoSwap/LimitedSwap) per the upstream KEP.¹³ Verify the feature's current stage and default against the exact minor version you run; it is one of the faster-moving kubelet features and this page does not assert it has reached GA. For a Guaranteed-QoS GPU pod specifically, swap changes nothing: it is not eligible either way, so NoSwap is the right default for GPU training/inference nodes and LimitedSwap only matters for co-located Burstable sidecars or batch jobs on the same node.

How to maintain it

• Re-derive the Node Allocatable arithmetic (step 1) whenever node instance types change; kubeReserved/systemReserved sized for one SKU's daemon footprint does not transfer to a bigger or smaller node.
• Watch enforceNodeAllocatable rollout carefully: adding kube-reserved/system-reserved enforcement after only pods was enforced can newly OOM-kill a daemon that was previously allowed to burst past its reservation.¹
• Audit taints after every managed-Kubernetes node-pool change (adding a first non-GPU pool, migrating GPU pools) since some platforms only apply the GPU taint at pool-creation time, not retroactively.¹⁴
• Track the swap feature gate's stage on upgrade; a cluster that relied on the removed UnlimitedSwap behavior needs its kubelet config migrated to LimitedSwap or NoSwap explicitly.¹³
• Scrape the kubelet's own CPU/Memory Manager counters, not just node-level CPU/memory usage. Queried directly from a live kubelet's /metrics endpoint (kubelet_cpu_manager_pinning_requests_total, kubelet_cpu_manager_pinning_errors_total, kubelet_memory_manager_pinning_requests_total, kubelet_memory_manager_pinning_errors_total, all marked [ALPHA] stability in the kubelet's own metric help text): the _requests_total counters increment once per Guaranteed-pod exclusive allocation (confirmed: 2 CPU pinning requests after scheduling 2 Guaranteed pods in sequence, 3 memory pinning requests across the DaemonSet CNI pod plus 2 Guaranteed pods), and a nonzero, climbing _errors_total on either metric means the reserved/shared pool is too small for the workload mix landing on that node, before pods start going Pending with Insufficient cpu. Alert on _errors_total > 0, not just on scheduling failures, since the pinning error surfaces at admission time on the node, one layer before the scheduler-level rejection this page's executed pressure test shows.

Canary rollout for a node-policy change

Changing kubeReserved/systemReserved/reservedSystemCPUs/cpuManagerPolicy/memoryManagerPolicy requires a kubelet restart, and CPU/Memory Manager policy changes specifically require clearing /var/lib/kubelet/cpu_manager_state and /var/lib/kubelet/memory_manager_state (a policy change with stale state present is a kubelet startup failure, not a silent reconciliation, in the versions this page has verified). Treat a node-policy rollout the same as any other infrastructure change with a blast radius:

1. Roll one node (or one canary node pool) first. Cordon it, drain it (verifying, as the executed drain above shows, that every evicted pod actually terminates within the drain timeout, not just that the drain command returned), apply the new KubeletConfiguration, clear both manager-state files, restart kubelet, and confirm kubectl get node reaches Ready before uncordoning.
2. Verify Allocatable and both manager-state files on the canary before sending it real traffic. Re-run the Node Allocatable arithmetic check and confirm cpu_manager_state/memory_manager_state show the expected policyName and an empty entries map on a freshly drained, freshly restarted node; a non-empty entries map with no pods running is a sign the state file was not actually cleared.
3. Roll back by reverting the KubeletConfiguration, clearing state, and restarting kubelet again, the same procedure as the forward rollout; there is no separate "undo" mechanism, and skipping the state-file clear on rollback risks the same startup failure as skipping it on roll-forward.
4. Only proceed past the canary once the pinning-error metrics above stay at zero under real workload placement on that node for a representative period, not just at cluster bring-up when the node is empty.

Failure modes

• Under-reserved kubeReserved/systemReserved: the kubelet or container runtime gets starved under load, the node goes NotReady, and every GPU pod on it is rescheduled, an outage far more expensive than a plain-CPU-node equivalent.
• Shipping strict-cpu-reservation untested against the target kubelet build: confirmed by direct execution against kubelet v1.31.0, this option can make the kubelet crash-loop indefinitely (unknown CPU Manager Policy option), taking the node NotReady and every pod on it down, a self-inflicted version of the outage above. Test the exact policy against the exact kubelet build in CI before rolling it to any real node.
• Reserved CPUs do not contain daemon usage, or strict reservation is unavailable: daemons contend with workloads after exhausting their CPU share, or Burstable/BestEffort workloads execute on the reserved CPU set and starve the daemons. Enable strict-cpu-reservation only after confirming your kubelet build actually accepts it (see the executed failure above); otherwise size and monitor the reserved set manually and treat reservedSystemCPUs alone as advisory, not enforced, isolation.
• Guaranteed QoS misconfigured as Burstable (limits set but requests omitted, or requests differ from limits): the pod silently loses exclusive CPU eligibility and Memory Manager placement. This does not imply that a correctly configured Guaranteed pod receives memory.min; that requires tiered Memory QoS.
• Missing GPU-node toleration on a non-GPU-requesting pod, combined with a missing or removed taint: cheap CPU-only pods land on the GPU node and compete for the same CPU/memory reservation the GPU pod needs.
• evictionHard set too tight (or left at conservative defaults) on a memory-dense GPU node: the kubelet evicts BestEffort/Burstable neighbors well before the node is actually in danger, wasting headroom; too loose, and the kubelet has no runway to react before real memory exhaustion.
• Assuming pod-level I/O isolation exists: without a runtime-specific mechanism, one pod's checkpoint-write burst or image pull can starve another pod's I/O on the same nodefs/imagefs, since Kubernetes has no native io.max/io.weight pod field.¹¹
• Enabling LimitedSwap expecting it to help a Guaranteed GPU pod: it structurally cannot, only Burstable pods are eligible, so it adds swap-thrash risk elsewhere on the node for zero benefit to the workload that matters.¹³

References

• Kubernetes: Reserve Compute Resources for System Daemons: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/
• Kubernetes: Control CPU Management Policies on the Node: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/
• Kubernetes: Node-pressure Eviction: https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/
• Kubernetes: Pod Quality of Service Classes (including Memory QoS with cgroup v2): https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/
• Kubernetes 1.36: Memory QoS tiered protection (memoryReservationPolicy: TieredReservation): https://kubernetes.io/blog/2026/04/29/kubernetes-v1-36-memory-qos-tiered-protection/
• Kubernetes: Taints and Tolerations: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
• Kubernetes: Memory Manager (reservedMemory, per-NUMA-node sum requirement): https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/
• Kubernetes: About cgroup v2 (kernel/runtime requirements, MemoryQoS): https://kubernetes.io/docs/concepts/architecture/cgroups/
• Kubernetes: Swap Memory Management: https://kubernetes.io/docs/concepts/cluster-administration/swap-memory-management/
• Kubernetes Enhancement Proposal, sig-node/2400 (Node System Swap Support): https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2400-node-swap/README.md
• Kubernetes blog: Fresh Swap Features for Linux Users in Kubernetes 1.32: https://kubernetes.io/blog/2025/03/25/swap-linux-improvements/
• kubernetes/kubernetes issue #70364, "Limiting blkio with cgroups by annotation" (closed as stale, unresolved; no native pod-level blkio/IO isolation has shipped since): https://github.com/kubernetes/kubernetes/issues/70364
• Google Cloud: Run GPUs in GKE Standard node pools (auto-taint/auto-toleration behavior): https://cloud.google.com/kubernetes-engine/docs/how-to/gpus

Related: Topology-Aware GPU Scheduling in Kubernetes · NUMA Affinity and CPU Pinning for GPU Pipelines · GPU Containerization Performance · Glossary

---

1. Kubernetes docs, "Reserve Compute Resources for System Daemons": Allocatable = Capacity - kubeReserved - systemReserved - evictionHard; evictionHard defaults memory.available: 100Mi, nodefs.available: 10%, nodefs.inodesFree: 5%, imagefs.available: 15%; only memory and ephemeral-storage are supported eviction resources; enforceNodeAllocatable defaults to [pods], valid values [pods]/[pods, system-reserved, kube-reserved]; kubeReservedCgroup/systemReservedCgroup must reference an existing cgroup or the kubelet fails to start. https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ ↩↩↩↩↩↩↩

2. Kubernetes docs, "Control CPU Management Policies on the Node": CPU Manager static assigns exclusive CPUs only to Guaranteed containers with integer requests; reservedSystemCPUs is an explicit logical-CPU list and takes precedence over CPU quantities in kubeReserved/systemReserved; strict-cpu-reservation prevents workloads from using those reserved CPUs; full-pcpus-only requires exclusive allocations to contain complete physical cores. The ascending physical-core rule applies when kubelet derives the reserved set from an integer quantity. https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/ ↩↩↩↩

3. Kubernetes docs, "Node-pressure Eviction": hard eviction thresholds trigger immediate eviction (no PodDisruptionBudget respect, 0s grace period); soft thresholds (evictionSoft) require evictionSoftGracePeriod; evictionPressureTransitionPeriod default 5m; memory.available is computed from cgroupfs, not free -m, and respects Node Allocatable. https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/ ↩↩↩

4. Kubernetes docs, "Pod Quality of Service Classes": Guaranteed requires every container's CPU and memory requests to equal limits, both greater than zero; Burstable is any pod with at least one request/limit that does not qualify as Guaranteed; BestEffort has none at all; BestEffort pods are preferentially evicted under node pressure. https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/ ↩↩

5. See ² and Topology-Aware GPU Scheduling in Kubernetes for how CPU Manager static and Memory Manager Static compose with Guaranteed QoS and the Topology Manager. ↩↩

6. Kubernetes docs, "Memory Manager": memoryManagerPolicy: Static requires reservedMemory per NUMA node, whose sum across all nodes must equal kubeReserved + systemReserved + evictionHard[memory.available]; only affects Guaranteed-QoS pods. https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/ ↩

7. Kubernetes docs, "Pod Quality of Service Classes," Memory QoS with cgroup v2 section: the feature is alpha and disabled by default; the original policy sets memory.high for Burstable containers. https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/ ↩↩

8. Kubernetes 1.36 introduced memoryReservationPolicy: TieredReservation: pod requests map to memory.min, container requests map to memory.low, and the default None policy writes neither value. https://kubernetes.io/blog/2026/04/29/kubernetes-v1-36-memory-qos-tiered-protection/ ↩↩↩

9. Kubernetes docs, "Taints and Tolerations": NoSchedule blocks new non-tolerating pods but does not evict running ones; NoExecute evicts non-tolerating pods immediately (or after tolerationSeconds); PreferNoSchedule is a soft, non-guaranteed preference; default toleration operator is Equal. https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ ↩↩

10. Kubernetes docs, "About cgroup v2": cgroup v2 requires Linux kernel 5.8+ (containerd 1.4+ / CRI-O 1.20+, systemd cgroup driver recommended); the page notes the MemoryQoS feature relies on cgroup v2 memory primitives (see ⁷ for its actual v1.22 [alpha] stage, the authoritative source); the page does not describe a pod-spec-level io.max/io.weight field. https://kubernetes.io/docs/concepts/architecture/cgroups/ ↩↩↩

11. kubernetes/kubernetes issue #70364, "Limiting blkio with cgroups by annotation": tracks the long-standing absence of a native, portable Kubernetes field for per-pod blkio/IO (cgroup v1 blkio / cgroup v2 io) weight or IOPS limits; opened 2018-10-29, closed 2019-03-28 by the stale-issue bot without a fix landing, and no such field has shipped in Kubernetes since. https://github.com/kubernetes/kubernetes/issues/70364 ↩↩↩↩

12. Kubernetes docs, "Swap Memory Management": memorySwap.swapBehavior field; NoSwap (default): "workloads running as Pods on this node do not and cannot use swap," though non-Kubernetes processes and the kubelet itself still can; LimitedSwap: "Kubernetes workloads can utilize swap memory"; failSwapOn: false required for the kubelet to start on a swap-enabled node. https://kubernetes.io/docs/concepts/cluster-administration/swap-memory-management/ ↩↩↩

13. Kubernetes Enhancement Proposal sig-node/2400: swap access under LimitedSwap is granted only to Burstable-QoS pods, excluding high-priority, static, and mirrored pods; Guaranteed pods are excluded because their memory is already fully reserved, BestEffort pods are excluded since they are first evicted under pressure; per-container swap limit is proportional to (container memory request / node memory) * available swap; feature progressed alpha (v1.22) to beta1 (v1.28) to beta2 (v1.30, which also removed the earlier UnlimitedSwap option). https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2400-node-swap/README.md ↩↩↩↩↩↩↩

14. Google Cloud, "Run GPUs in GKE Standard node pools": GKE automatically applies the nvidia.com/gpu=present:NoSchedule taint (and a matching auto-toleration via the ExtendedResourceToleration admission controller) to GPU node pools, but only when the cluster already has at least one non-GPU node pool, and not retroactively if a non-GPU pool is added afterward. https://cloud.google.com/kubernetes-engine/docs/how-to/gpus ↩↩