Markdown

Host OS and kernel tuning for GPU nodes¶

Scope: host kernel/OS knobs that keep GPUs fed, namely vm.swappiness/swapoff, transparent hugepages (training vs inference), the performance CPU governor and disabling deep C-states to kill GPU bubbles, isolcpus/nohz_full plus IRQ affinity (disable irqbalance), vm.dirty_ratio and O_DIRECT/io_uring for checkpoint writes, and jemalloc/tcmalloc allocator tuning. NUMA/CPU pinning itself lives in NUMA Affinity and CPU Pinning for GPU Pipelines; this page covers the rest of the host stack.

What it is¶

A set of OS- and kernel-level settings applied to a GPU node so the host never stalls the accelerator. The CPU prepares the next batch (load, tokenize, transform), dispatches kernels, and coordinates threads while the GPU works the current batch. If host-side work is slow or the scheduler places it poorly, the GPU goes idle, a bubble, waiting on the CPU, memory, or I/O. The knobs here remove the common sources of that idle time: memory swapping, page-table overhead, frequency scaling and deep idle states, interrupt jitter, page-cache write stalls during checkpointing, and allocator lock contention.

These are not GPU-driver settings (persistence mode, MPS, MIG live in GPU Software Stack and Node Administration and NVIDIA Container Toolkit and CDI). They are pure host Linux configuration applied via sysctl, kernel command line, cpupower, /proc/irq, and environment variables.

Why use it¶

System-level tuning is routinely skipped in favour of model-level work, but it can yield double-digit percentage gains on a well-instrumented node, and at cluster scale that translates into large compute savings.¹ The per-knob figures below are the book's reported numbers on its own test workload, not a guarantee for yours; treat each as a starting hypothesis and A/B test it on the actual model, batch shape, and hardware before committing it fleet-wide:

A single swapped-out page causes a multiple-orders-of-magnitude slowdown when the GPU next touches that data. This is a mechanical property of swap I/O latency versus DRAM, not workload-specific.¹
The book reports modest hugepage gains (roughly 3-5% throughput on its workload) from cutting page faults and TLB pressure; the size of the win depends on the working-set's TLB pressure, so measure it on your own model rather than assuming 3-5% carries over.¹
THP can add tail latency when an allocating task enters direct reclaim or memory compaction. Background kcompactd and khugepaged work is not a global stop-the-world pass; the Linux policy determines whether the allocating task stalls or only wakes background work. Measure compact_stall and allocation latency on the deployed kernel before selecting madvise or never.¹²
Deep C-state wakeups cost microseconds each; aggregated across a data-loader hot path they accumulate into GPU bubbles. Whether this is measurable depends on how tight the hot-path loop already is; profile before assuming it dominates.¹

The goal is steady-state: each GPU's utilization hovers near 100% and only dips at intentional synchronization barriers.¹

When to use it (and when not)¶

The table below is a starting point to validate per workload, not a fixed training-on/inference-off rule: a training job with a latency-sensitive online-eval side channel, or an inference server batching large requests, can each need the opposite column's setting. Confirm on the actual workload before standardizing a fleet-wide default.

Knob	Common starting point for training (throughput)	Common starting point for inference (latency)	Skip / caution
`vm.swappiness=0` / `swapoff -a`	Yes	Yes	Need enough RAM; otherwise the OOM killer reaps the job
Transparent hugepages	Often enabled (`always`/`madvise`)	Often disabled (`never`) or `madvise`	`never` removes compaction stalls but forces manual hugepage management; measure page-fault/TLB pressure on your workload rather than assuming the training default is a net win
`performance` governor + no deep C-states	Yes	Yes	Trades extra CPU power for responsiveness — fine when GPUs dominate the power budget
`isolcpus` / `nohz_full` + IRQ pinning	When data-loader jitter is observed	When tail latency matters	Real-time FIFO/RR priority usually unnecessary once threads are pinned; RT can starve other processes
Raised `vm.dirty_ratio`	Large checkpoints, sized to an absolute GB budget, not a flat percentage	—	High dirty ratios delay durability and can stall for minutes on TB-memory nodes; pair with `O_DIRECT` (not `io_uring` alone, see below) for latency-sensitive writes
`jemalloc`/`tcmalloc` tuning	Data-prep-heavy pipelines	Yes	Workload-dependent; measure before committing

On Grace-based superchips (GH200, GB200) the CPU and GPU are coherent over NVLink-C2C, but Linux still models CPU DRAM and GPU HBM as separate pools, so these host knobs still apply.¹ Per-node sysctls should be verified, not assumed: defaults differ by distribution (e.g. vm.swappiness defaults to 60³).

Architecture¶

flowchart LR
    SWAP["Swap policy"] --> GOAL["Bounded host latency"]
    THP["THP or explicit hugepages"] --> GOAL
    GOV["CPU frequency and idle policy"] --> GOAL
    ISOL["CPU and IRQ placement"] --> GOAL
    DIRTY["Write-back and direct I/O"] --> GOAL
    ALLOC["Host allocator"] --> GOAL
    GOAL --> STEADY["GPU remains fed"]

How to use it¶

These are reference templates. Validate every value on the target hardware and workload before rolling out. None of the numbers below have been hardware-tested here.

Virtual memory and swapping¶

vm.swappiness is the rough relative IO cost of swapping versus filesystem paging; lower means avoid swap. The kernel range is 0–200 with a default of 60.³ Set it low and disable swap devices outright for the job's lifetime:

# Persist: avoid swapping except under extreme memory pressure
sudo sysctl -w vm.swappiness=0
echo 'vm.swappiness = 0' | sudo tee /etc/sysctl.d/90-gpu-node.conf

# Disable all swap devices/files until next reboot
sudo swapoff -a

# Verify swap stays at zero
free -m
vmstat 1 5

Enforce no-swap and memory limits per workload with cgroups v2 (via Docker/Kubernetes) rather than relying on global settings.¹

Transparent hugepages (training vs inference)¶

THP enabled accepts always, madvise, or never; the active value is shown in brackets in the sysfs file.² Enable for throughput-bound training, set never (or madvise) for latency-bound inference to avoid khugepaged compaction stalls.¹²

# Inspect current policy
cat /sys/kernel/mm/transparent_hugepage/enabled    # e.g. [always] madvise never

# Runtime: disable for latency-sensitive inference
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

Make it boot-persistent on the kernel command line (append to GRUB_CMDLINE_LINUX, then regenerate grub):²

# /etc/default/grub
GRUB_CMDLINE_LINUX="... transparent_hugepage=never"

For very large preallocated pinned buffers, prefer explicit hugepages via vm.nr_hugepages/hugetlbfs for deterministic behaviour.¹ Configure the service or container's locked-memory limit before allocating large pinned regions, then check the CUDA return value: cudaHostAlloc and cudaHostRegister return an error such as cudaErrorMemoryAllocation when page locking fails; they do not document a transparent pageable fallback.⁸ Generic mlock(2) separately reports ENOMEM when RLIMIT_MEMLOCK is exceeded.⁹ Do not claim the limit caused a CUDA failure without checking the CUDA error and host logs, because CUDA does not define RLIMIT_MEMLOCK as the only possible failure source.

CPU governor and C-states¶

Pin the CPU at maximum frequency and prevent deep idle states so wakeups don't introduce latency. cpupower frequency-set -g performance sets the governor; under the Intel P-state driver's active mode only performance and powersave are available.⁵⁴

# Lock all cores to max frequency
sudo cpupower frequency-set -g performance

# Verify
cpupower frequency-info | grep -i "current policy\|governor"

Limit C-states at boot (C0 is active; deeper states sleep harder and wake slower). Many server BIOS/UEFI "high-performance" profiles set the governor to performance and disable deep C-states automatically.¹ On the kernel command line:

# /etc/default/grub  — cap idle depth; intel_idle.max_cstate=1 on Intel
GRUB_CMDLINE_LINUX="... processor.max_cstate=1 intel_idle.max_cstate=1 idle=poll"

idle=poll maximizes responsiveness at the cost of power and heat. Acceptable when GPUs dominate the node's power budget, but measure thermals.

CPU isolation and interrupt affinity¶

Reserve cores for data-pipeline threads and keep device interrupts on the local NUMA node. isolcpus removes CPUs from the general scheduler; nohz_full suppresses the timer tick on a core when only one thread is runnable there, and should be paired with rcu_nocbs.⁶ These are boot parameters:

# /etc/default/grub  — isolate cores 8-15 for the data pipeline
GRUB_CMDLINE_LINUX="... isolcpus=domain,managed_irq,8-15 nohz_full=8-15 rcu_nocbs=8-15"

When isolcpus is combined with managed_irq or nohz, include the domain flag to isolate from SMP balancing/scheduling.⁶

Disable irqbalance (or run it with bespoke rules) and pin each GPU/NIC interrupt to a core on its own NUMA node, so no remote node services the IRQ and evicts remote cache lines:¹

# Stop the rebalancer
sudo systemctl disable --now irqbalance

# Pin IRQ 142 to CPU0 (smp_affinity is a hex CPU mask)
echo 1 | sudo tee /proc/irq/142/smp_affinity

Prefer cgroup cpuset isolation in production (docker --cpuset-cpus, Kubernetes CPU management) to bind each workload to dedicated cores and memory.¹ Once threads are pinned, real-time FIFO/RR priorities are usually unnecessary and risk starving other processes.¹

Filesystem write-back and checkpoint I/O¶

Large checkpoint bursts fill the page cache and stall the training loop. vm.dirty_ratio is the percentage of system memory that may hold dirty pages before writers block; vm.dirty_background_ratio is the percentage at which the background flushers start.³ A higher dirty ratio lets the OS batch multi-GB checkpoints before flushing, but the ratio is a percentage, not an absolute cap: on a terabyte-memory node, vm.dirty_ratio=40 permits several hundred GB of dirty pages before writers block, which can turn an eventual flush into a multi-minute stall or add sustained memory pressure that competes with the training process. Size the value from an absolute budget appropriate to the node's RAM, not by copying a percentage from a smaller box:¹

# Batch more dirty data in RAM to smooth large checkpoint writes.
# Size for an absolute budget (here ~40 GB on a 1 TB node), not a blanket 40%.
sudo sysctl -w vm.dirty_ratio=4
sudo sysctl -w vm.dirty_background_ratio=1

For latency-sensitive workflows, bypass the page cache by opening checkpoint files with O_DIRECT. io_uring itself is only an asynchronous I/O submission interface: it does not bypass the page cache by default, and a buffered io_uring read/write still goes through the same cache as read(2)/write(2).¹⁰ To get async and cache-bypassing I/O, open the file O_DIRECT and submit through io_uring; using io_uring alone does not imply O_DIRECT. After a checkpoint write, posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED) is only a best-effort hint to drop clean cached pages; it does not reliably discard pages that are still dirty (unwritten), so call fsync/fdatasync first, or write with O_DIRECT so the pages never enter the cache to begin with.¹¹ Writing checkpoints from a separate thread, or using PyTorch distributed checkpointing, further hides the cost.¹

How to develop with it¶

Host CPU memory allocator¶

A swappable, fragmenting allocator injects unpredictable pauses into data preparation. Tune jemalloc to shard allocations into per-CPU arenas (narenas), purge off the hot path (background_thread), and delay returning freed pages to the OS (dirty_decay_ms/muzzy_decay_ms).¹⁷ dirty_decay_ms sets the time from a page becoming unused-dirty to being purged; background_thread shifts purging to dedicated threads and improves tail latency.⁷

export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2
export MALLOC_CONF="narenas:8,dirty_decay_ms:10000,muzzy_decay_ms:10000,background_thread:true"

tcmalloc benefits from larger per-thread caches so small allocations avoid global locks and syscalls:¹

export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4
export TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=$((512*1024*1024))
export TCMALLOC_RELEASE_RATE=16

Runnable configuration validator¶

This standard-library check validates CPU-list syntax, isolation, and write-back ordering before a profile is rendered into node configuration. It does not inspect a live kernel.

def parse_cpu_list(spec):
    cpus = set()
    for field in spec.split(","):
        bounds = [int(value) for value in field.split("-")]
        if len(bounds) == 1:
            start = end = bounds[0]
        elif len(bounds) == 2:
            start, end = bounds
        else:
            raise ValueError(f"invalid CPU field: {field}")
        if start < 0 or end < start:
            raise ValueError(f"invalid CPU range: {field}")
        new = set(range(start, end + 1))
        if cpus & new:
            raise ValueError(f"overlapping CPU range: {field}")
        cpus |= new
    return cpus


def validate_profile(workload, housekeeping, dirty_background, dirty_limit):
    workload_cpus = parse_cpu_list(workload)
    housekeeping_cpus = parse_cpu_list(housekeeping)
    if workload_cpus & housekeeping_cpus:
        raise ValueError("workload and housekeeping CPUs overlap")
    if not 0 <= dirty_background < dirty_limit <= 100:
        raise ValueError("dirty ratios must satisfy 0 <= background < limit <= 100")
    return workload_cpus, housekeeping_cpus


workload, housekeeping = validate_profile("8-15", "0-7", 1, 4)
assert workload == set(range(8, 16))
assert housekeeping == set(range(8))

for invalid in (
    ("0-3", "3-7", 1, 4),
    ("8-4", "0-3", 1, 4),
    ("8-15", "0-7", 4, 4),
):
    try:
        validate_profile(*invalid)
    except ValueError:
        pass
    else:
        raise AssertionError(f"invalid profile accepted: {invalid}")

print("Linux node profile validation: all asserts passed")

Executed output:

Linux node profile validation: all asserts passed

How to maintain it¶

Confirm sysctl values survive reboot (/etc/sysctl.d/) and that kernel-cmdline knobs (THP, isolcpus, nohz_full, C-states) appear in cat /proc/cmdline after boot.
Watch for regressions: free -m/vmstat (swap at zero), cpupower frequency-info (governor still performance), and GPU utilization (near 100% off barriers).
Re-validate per workload: THP, allocator, and dirty-ratio settings are workload-dependent; the training and inference profiles diverge.
In containers, mirror host policy through the security context, cgroups v2, and --ulimit memlock so pinning and no-swap actually take effect inside the pod. See GPU Containerization Performance and Topology-Aware GPU Scheduling in Kubernetes.

How to run it in production¶

Roll out one knob at a time to a canary node. Capture the old sysctl, kernel command line, IRQ masks, governor, THP policy, and allocator environment before applying a profile. Reboot when a boot parameter changes, verify effective state after reboot, then compare GPU idle gaps, allocation latency, reclaim/compaction counters, I/O latency, CPU power, and job throughput against the baseline.

Failure modes¶

Disabling swap without a memory limit converts reclaim pressure into an OOM kill.
THP direct allocation or compaction stalls increase tail latency on the deployed kernel.
isolcpus, cpusets, and IRQ masks overlap, so housekeeping work enters workload cores.
idle=poll raises CPU power and heat enough to reduce the node's thermal margin.
Dirty-page limits copied as percentages scale to unsafe byte counts on large-memory nodes.
A CUDA host allocation fails and the caller ignores its return code.
LD_PRELOAD points to an unavailable or ABI-incompatible allocator.

References¶

Chris Fregly, AI Systems Performance Engineering (O'Reilly), Chapter 3, "OS, Docker, and Kubernetes Tuning for GPU-Based Environments" — host OS/kernel tuning for GPU nodes (swappiness, THP training-vs-inference, governor/C-states, isolcpus/nohz_full, IRQ affinity, dirty_ratio, O_DIRECT/io_uring, jemalloc/tcmalloc).
Linux kernel — Transparent Hugepage Support (always/madvise/never, transparent_hugepage= boot param, sysfs control). https://docs.kernel.org/admin-guide/mm/transhuge.html
Linux kernel — /proc/sys/vm/ documentation (swappiness range 0–200 default 60, dirty_ratio, dirty_background_ratio). https://docs.kernel.org/admin-guide/sysctl/vm.html
Linux kernel — CPU Performance Scaling / CPUfreq (governors, P-state active mode). https://docs.kernel.org/admin-guide/pm/cpufreq.html
Red Hat — CPUfreq governors and cpupower frequency-set usage. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/power_management_guide/cpufreq_governors
Red Hat — isolcpus overview and flag interaction with managed_irq/nohz/domain. https://access.redhat.com/solutions/480473
jemalloc — TUNING.md (narenas, dirty_decay_ms/muzzy_decay_ms, background_thread). https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md
Linux man-pages — mlock(2): RLIMIT_MEMLOCK failure semantics. https://www.man7.org/linux/man-pages/man2/mlock.2.html
Linux man-pages — io_uring(7): buffered versus direct I/O is a property of the file descriptor, not of using the io_uring interface. https://man7.org/linux/man-pages/man7/io_uring.7.html
NVIDIA CUDA Runtime API, Host Memory Management (cudaHostAlloc, cudaHostRegister, and documented error returns): https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__MEMORY.html

Fregly, AI Systems Performance Engineering, Ch. 3. ↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩
Linux kernel, Transparent Hugepage Support. https://docs.kernel.org/admin-guide/mm/transhuge.html ↩↩↩↩
Linux kernel, /proc/sys/vm/ documentation. https://docs.kernel.org/admin-guide/sysctl/vm.html ↩↩↩
Linux kernel, CPU Performance Scaling. https://docs.kernel.org/admin-guide/pm/cpufreq.html ↩
Red Hat, CPUfreq governors. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/power_management_guide/cpufreq_governors ↩
Red Hat, isolcpus overview. https://access.redhat.com/solutions/480473 ↩↩
jemalloc TUNING.md. https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md ↩↩
NVIDIA CUDA Runtime API, Host Memory Management: cudaHostAlloc and cudaHostRegister return a cudaError_t, including documented allocation and invalid-value errors; callers must check the returned status. https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__MEMORY.html ↩
Linux man-pages, mlock(2): mlock()/mlockall() fail with ENOMEM when the RLIMIT_MEMLOCK limit is exceeded; there is no implicit pageable fallback. https://www.man7.org/linux/man-pages/man2/mlock.2.html ↩
io_uring(7) man page: io_uring is an asynchronous I/O submission/completion interface; whether an operation is buffered (page-cache-backed) or direct depends on how the target file descriptor was opened (O_DIRECT or not), not on using io_uring itself. https://man7.org/linux/man-pages/man7/io_uring.7.html ↩
Linux /proc/sys/vm/ and posix_fadvise(2) semantics: POSIX_FADV_DONTNEED is an advisory hint that the kernel may skip, and it only drops pages that are already clean; dirty pages must be written back (e.g. via fsync) before the hint reliably frees them. https://docs.kernel.org/admin-guide/sysctl/vm.html ↩