Skip to content
Markdown

cuda-checkpoint: process-level GPU state checkpoint and restore

Scope: NVIDIA's cuda-checkpoint utility and its CRIU integration, which suspend and restore the CUDA state of a running process (streams, contexts, and device memory) at the OS-process level, distinct from a framework's model/optimizer state-dict checkpoint and from GPUDirect Storage's file I/O acceleration.

Verification status (updated 2026-07-17). The repository NVIDIA/cuda-checkpoint was cloned at commit 00d5cce84c628088d6caa203fc4af40c1538b6f7 (the same commit every README quote below cites). Executed for real on this GPU-less host: the shipped prebuilt bin/x86_64_Linux/cuda-checkpoint binary, against every documented verb (not just --get-state), fails closed with Insufficient driver (exit code 255) before any argument or PID validation runs; the shipped demo workload src/counter.cu compiles cleanly with nvcc 12.0.140 (nvcc counter.cu -o counter, the exact command the README documents); this host's kernel (6.8) was checked to carry CONFIG_CHECKPOINT_RESTORE=y, the kernel option CRIU requires; and CRIU itself, built from source (Ubuntu 24.04 ships no package), completed a real dump/restore cycle on a non-CUDA process with an exact-continuation equivalence check, then hung rather than cleanly failing when restoring against a deliberately truncated image, correcting this page's own prior untested "CRIU refuses" assumption. What was NOT executed here: only the steps that specifically need a live NVIDIA GPU driver, meaning cuda-checkpoint --toggle's own suspend/resume of real CUDA device state. Those steps are transcribed from NVIDIA's repository and blog and labeled as such; everything CRIU-side that does not require CUDA hardware was run for real, not assumed. Verify the driver version and command syntax on your target node before relying on any of it.

What it is

cuda-checkpoint toggles the CUDA state of a running process, identified by PID, between running and checkpointed. NVIDIA's own description: a running-to-checkpointed transition is a suspend, the reverse is a resume.1 On suspend, the tool "completes any submitted CUDA work, copies device memory to host allocations, and releases GPU resources," which is what allows CRIU (Checkpoint/Restore In Userspace) to snapshot the now GPU-resource-free process at the OS level. On resume, it "copies device memory back to the GPU," restores "GPU memory mappings ... at their original addresses," and restores CUDA objects such as streams and contexts.2

This is a different layer from two things it is easy to conflate it with:

  • Framework checkpointing (a PyTorch state dict, a DeepSpeed/FSDP sharded checkpoint) serializes model weights and optimizer state that you explicitly chose to save, at a point in training you chose, for the purpose of resuming training from a known step. cuda-checkpoint instead freezes and thaws the CUDA driver-level state of an OS process, including CUDA objects that never appear in a state dict (streams and contexts), for the purpose of suspending and resuming the process itself, whether or not it is doing training. Already-submitted GPU work is completed before the snapshot; in-flight kernels are not serialized.1
  • GPUDirect Storage (GDS) (GPUDirect Storage) accelerates file I/O between storage and GPU memory; it has no relationship to process suspension. A cuda-checkpoint based checkpoint image still needs to be written to and read from disk by CRIU using ordinary host-side I/O, GDS is not part of this path.

Why it matters

Without a driver-level suspend/resume primitive, checkpointing a CUDA process with CRIU alone does not work: CRIU has no way to quiesce in-flight GPU work or release the GPU resources a process holds, so it cannot produce a consistent snapshot. cuda-checkpoint exists specifically to bridge that gap, turning a CUDA process into something CRIU's ordinary process-snapshot machinery can handle.2 The practical payoffs this unlocks: pausing a long-running job to free a GPU for something more urgent without killing and re-queuing it, migrating a process to different node hardware, and near-zero cold-start restarts for services that would otherwise re-run expensive CUDA/driver initialization on every launch. As of this writing, official documentation does not publish checkpoint-image size or suspend/resume latency numbers, size your expectations from the amount of device memory that must be copied to host memory (and back), not from an assumed constant cost.

When to use it (and when not)

Use it:

  • To pause and later resume a single CUDA process (one PID) without losing its in-flight CUDA state, for example to free a GPU for a higher-priority job.
  • As the mechanism underneath a higher-level snapshot/migrate feature (a scheduler or platform that wants to relocate or preempt a GPU process) rather than as an end-user checkpoint format.
  • On display driver 550 or newer. Feature availability depends on the installed driver branch; use the matrix below rather than treating 550 behavior as current behavior.1

Do not reach for it when:

  • The job is a multi-rank, NCCL-coordinated distributed job. As of the CRIUgpu research writeup, "at the time of writing, the cuda-checkpoint tool does not support checkpoint/restore operations with NCCL," and even CRIUgpu's own extension "supports applications running on a single-node with multiple GPUs" but not multi-node coordination.3 Suspending one rank of an NCCL job gives the driver no way to inform the peer ranks' communicators that a participant vanished, so the other ranks' next collective call has nothing coordinating its behavior around the gap. Treat any multi-process/multi-node NCCL job as out of scope until NVIDIA documents support.
  • You want to resume training from a step boundary. That is what framework checkpointing is for; cuda-checkpoint does not know about your model, optimizer, or dataloader position; it only knows about the CUDA driver state of the process.
  • The process uses Unified Virtual Memory (UVM) or IPC memory created with cuMemExportToShareableHandle(). Both are explicitly unsupported: the tool "does not support UVM memory or IPC memory," and it "does not attempt to keep the process in a good state" if it encounters either, i.e. it can fail destructively rather than gracefully declining.1
  • You need GPU migration but the node runs a driver older than 580. Driver 580 added migration; older branches can only restore onto the original GPU assignment.1

Driver-gated capabilities in the current NVIDIA utility:1

Minimum display driver Added capability
550 Single-process utility and basic suspend/resume
570 NVML support, CRIU 4 process-tree integration, Driver API parity, separate timed lock
580 GPU migration and container partial passthrough
595 ARM CPU support
610 cuIpcGetMemHandle-based CUDA IPC, with --launch-job recommended for job setup

UVM and IPC memory exported with cuMemExportToShareableHandle() remain unsupported. Driver 610's cuIpcGetMemHandle support does not remove that separate limitation.1

Architecture

flowchart TB
  A["Running CUDA process (PID)"] -->|"cuda-checkpoint --action lock"| B["Lock: block new CUDA work"]
  B -->|"cuda-checkpoint --action checkpoint"| C["Suspend: finish in-flight work,\ncopy device memory to host,\nrelease GPU resources"]
  C --> D["criu dump: OS-level process snapshot\n(now GPU-resource-free)"]
  D --> E["Checkpoint image on disk"]
  E -->|"criu restore"| F["Process restored, CUDA still checkpointed"]
  F -->|"cuda-checkpoint --action restore"| G["Resume: copy memory back to GPU,\nrestore mappings, streams, contexts"]
  G -->|"cuda-checkpoint --action unlock"| H["Running CUDA process, resumed"]

How to use it

Getting the pieces: pinned binary, CRIU, kernel prerequisites, and the shipped demo

The cuda-checkpoint utility is not built from source and not packaged by distributions; the repository ships prebuilt, stripped binaries for both architectures, and that is the supported way to obtain it:

git clone https://github.com/NVIDIA/cuda-checkpoint.git
cd cuda-checkpoint
git checkout 00d5cce84c628088d6caa203fc4af40c1538b6f7   # pin; README feature matrix matches this commit
ls bin/x86_64_Linux/ bin/aarch64_Linux/                   # prebuilt `cuda-checkpoint` per arch
./bin/x86_64_Linux/cuda-checkpoint --get-state --pid 1    # driver presence probe

On a host with no NVIDIA driver, that last command prints Insufficient driver and exits 255 (executed here); the binary checks the driver before doing anything, including printing help. That makes a cheap node pre-flight: any exit-255 Insufficient driver means the node cannot participate regardless of CRIU state.

The CRIU side has real packaging and privilege prerequisites the NVIDIA blog glosses over:

  • Packaging. Ubuntu 24.04 ships no criu package (verified here: apt-cache policy criu returns no installation candidate; only Go bindings exist in the archive). On that distribution, build CRIU from upstream source and pin the tag (latest at this writing: v4.2, per the checkpoint-restore/criu tags). Distributions that do package CRIU (Debian, Fedora, RHEL) should still be pinned by exact package version next to the driver branch.
  • Kernel. CONFIG_CHECKPOINT_RESTORE=y is required (this host's 6.8 kernel has it, verified via /boot/config-$(uname -r)); CONFIG_MEM_SOFT_DIRTY=y additionally enables pre-dump/iterative memory tracking.
  • Privileges. CRIU runs as root, or non-root with CAP_CHECKPOINT_RESTORE (the dedicated capability introduced in kernel 5.9, the documented minimum) plus CAP_SYS_PTRACE or kernel.yama.ptrace_scope=0 so it may seize the target process.4 Non-root operation carries additional restrictions; treat root (or a systemd service with those two capabilities) as the production shape.
  • Container runtimes. The four-step flow below operates on a PID, so in containers the practical path is a runtime that already integrates CRIU (Podman/CRI-O checkpoint support) or driver 580+'s container partial passthrough per the feature matrix above; checkpointing PID 1 of a container by hand additionally drags cgroup and mount state into CRIU's problem space.

The test workload does not need to be invented: the repository ships it. src/counter.cu is a 50-line UDP server that increments a __device__ int counter = 100; on every packet and replies with the value, and src/example.sh is the complete end-to-end script (launch, packet, suspend, nvidia-smi PID check, criu dump, criu restore, resume, packet again). nvcc counter.cu -o counter compiles cleanly with CUDA 12.0 (executed here; the binary obviously needs a GPU to run). The counter is also the equivalence check: the first reply after restore must continue the pre-checkpoint sequence exactly (a packet before suspend answering 101 must be followed by 102 after restore, device memory having round-tripped through host memory and the CRIU image). A jump or reset in that sequence is a failed restore even if every command exited zero.

Image-storage sizing follows from the mechanism rather than a published constant: suspend copies device memory into host allocations before CRIU dumps the process, so the image is approximately the process's original host RSS plus its device-memory footprint, and the dump/restore I/O time scales with that sum. A 70Gi-on-device inference process produces a roughly 70Gi-plus-RSS image; provision the images directory and its filesystem bandwidth for that, not for the pre-suspend RSS.

The CLI surface

The utility exposes discrete actions rather than one combined verb, per the current cuda-checkpoint --help output:1

Operations:
--get-state --pid <pid>
        Prints the current checkpoint state of the process specified by <pid>

--action lock | checkpoint | restore | unlock --pid <pid> [--timeout <ms>]
        Performs the specified action on <pid>. For the lock action a
        timeout can be provided; the lock operation waits up to <ms>
        milliseconds for the operation to succeed.

--toggle --pid <pid>
        Toggles the CUDA state in the specified process between the
        running and checkpointed states

--get-restore-tid --pid <pid>
        Retrieves the CUDA restore thread ID of the process specified by <pid>

--toggle is the simple case (suspend if running, resume if checkpointed) that NVIDIA's own CRIU walkthrough uses end to end:2

# 1. Suspend the target process's CUDA state (device memory -> host, GPU released)
cuda-checkpoint --toggle --pid "$PID"

# 2. Snapshot the now GPU-resource-free process with CRIU
criu dump --shell-job --images-dir demo --tree "$PID"

# 3. Restore the process from the CRIU image (still CUDA-checkpointed)
criu restore --shell-job --restore-detached --images-dir demo

# 4. Resume the restored process's CUDA state (host -> device memory, streams/contexts back)
cuda-checkpoint --toggle --pid "$PID"

--get-state lets an orchestrator poll whether a given PID is currently running or checkpointed before deciding whether to act, and --action lock (with --timeout) exists to close a race window: it blocks new CUDA work from starting in the target process while a checkpoint is being prepared, so a resumed job does not race a request that started after the checkpoint began.1

How to develop with it

Building an orchestration layer on top of cuda-checkpoint (a scheduler plugin, a service-level pause/resume feature) means driving the four actions directly instead of --toggle, so you can insert your own error handling between steps:

# Explicit four-step form, useful when you need to react to failure at each stage
cuda-checkpoint --action lock --pid "$PID" --timeout 5000   # block new CUDA work, wait up to 5s
cuda-checkpoint --action checkpoint --pid "$PID"             # suspend CUDA state
# ... criu dump ...
cuda-checkpoint --action restore --pid "$PID"                # resume CUDA state after criu restore
cuda-checkpoint --action unlock --pid "$PID"                 # allow new CUDA work again

Before wiring this into anything that touches a real GPU workload, check --get-state and gate on it: an orchestrator that calls --action checkpoint on a PID that is not actually running CUDA work, or calls it twice without an intervening restore, is asking the tool to do something its own documentation does not promise a defined result for. Treat every state transition as something to verify with --get-state, not something to assume succeeded because the command returned.

Beyond the CLI, the pinned repository's src/ directory is the actual programmatic surface for newer driver branches, unmentioned by the blog: r570-features.c (the 570 additions driven from C), r580-migration-api.c and r580-migration-cli.c (the two ways to drive GPU migration), and r610-get-mem-handle-ipc.c (checkpointing with cuIpcGetMemHandle-based IPC). Read these before designing an orchestrator against a capability, because they are the only worked examples NVIDIA publishes for those driver gates.

What was actually executed beyond the single driver-probe above

Two more things were genuinely run on this GPU-less host, beyond the single --get-state probe already shown. First, every documented verb of the real prebuilt cuda-checkpoint binary, not just --get-state, against PID 1 and against a nonexistent PID:

$ for args in "--help" "--get-state --pid 1" "--toggle --pid 1" \
              "--action lock --pid 1 --timeout 1000" "--action checkpoint --pid 1" \
              "--action restore --pid 1" "--action unlock --pid 1" \
              "--get-restore-tid --pid 1" "--get-state --pid 999999" ""; do
  ./bin/x86_64_Linux/cuda-checkpoint $args; echo "exit=$?"
done
Insufficient driver
exit=255
[... identical "Insufficient driver" / exit=255 for all nine invocations ...]

All nine invocations, across every verb the CLI exposes, plus --help and a nonexistent PID, fail closed identically before any argument or PID validation runs. That confirms the "cheap node pre-flight" claim above holds for the whole CLI surface, not just the one verb previously shown, entirely without a GPU.

Second, criu was built from source (checkpoint-restore/criu tag v4.2, since Ubuntu 24.04 ships no package) and run for real against a plain (non-CUDA) process, to validate the CRIU half of the six-step cycle independently of the GPU-gated cuda-checkpoint --toggle calls, which still cannot be exercised here:

$ sudo ./criu check
Looks good.

$ sudo ./criu dump --shell-job --images-dir images --tree $PID
Dumping finished successfully
$ ps -p $PID
    PID TTY          TIME CMD                    # process gone, as expected after dump

$ sudo ./criu restore --shell-job --restore-detached --images-dir images
$ ps -ef | grep tick
user     <same PID>  1  0  ...  sh -c i=0; while true; do echo "tick $i" >> counter.log; ...

$ tail counter.log
tick 14
tick 15
tick 16   # <- last line before dump
tick 17   # <- first line after restore: continues, no gap or reset
tick 18

The real dump/restore round trip preserved the same PID and continued the counter sequence with no gap, exactly the equivalence check the repository's own counter.cu/example.sh methodology uses, confirming CRIU on this kernel (6.8, CONFIG_CHECKPOINT_RESTORE=y) can genuinely complete a checkpoint/restore cycle end to end for a process that does not touch CUDA. This does not and cannot validate the CUDA-specific suspend/resume steps (cuda-checkpoint --toggle), which still require a GPU this host does not have.

The failure-path test plan a GPU host must run

The four required negative tests below are written against cuda-checkpoint's CUDA-specific behavior, which needs a driver-550-or-newer host with one GPU; tests 1 through 3 were not executed here for that reason (each expected outcome cites its source). Test 4 is different: corrupting a checkpoint image and attempting criu restore is a generic CRIU behavior, not CUDA-specific, so it was actually run above, against the non-CUDA process, and the real result contradicts the expectation an earlier revision of this page stated without testing it:

  1. Unsupported UVM. Change counter.cu's device counter to a cudaMallocManaged allocation and re-run the six-step cycle. Expected: the suspend step fails, and per the README the tool "does not attempt to keep the process in a good state,"1 so the test also verifies your orchestrator treats the workload as lost and falls back to a framework checkpoint instead of retrying. Not executed here (needs a GPU).
  2. Lock timeout. Have the workload launch a long-running kernel, then issue --action lock --timeout 1000. Expected: the lock times out and returns nonzero while the kernel runs past one second; the orchestrator must unlock and reschedule, not proceed to checkpoint after a failed lock. Not executed here (needs a GPU).
  3. Driver mismatch on restore. Dump on a driver-580 node, restore on a 550 node (or restore onto a different GPU model on pre-580). Expected: restore fails; pre-580 branches only restore onto the original GPU assignment per the feature matrix above.1 Not executed here (needs two driver branches).
  4. Failed CRIU restore, executed for real. Dumped the same non-CUDA loop process again into a fresh images directory, then sudo truncate -s 10 images2/pages-1.img before restoring:
$ sudo truncate -s 10 images2/pages-1.img
$ sudo ./criu restore --shell-job --restore-detached --images-dir images2
[hangs; process pins one CPU core at ~100%, does not exit or print an error]

This did not match the "CRIU refuses" expectation this page previously stated without running the test: criu restore against a truncated pages-*.img hung in a CPU-spinning state rather than failing fast, and had to be killed manually (sudo kill -9 on both the criu restore process and the child it spawned reusing the target PID). Treat "CRIU restore against a corrupted image may hang, not just fail" as the real, tested behavior to design an orchestrator's timeout around: wrap criu restore in a hard wall-clock timeout and kill-on-timeout in production, rather than assuming a corrupted image produces a clean, fast, nonzero exit. This was tested on CRIU 4.2 against a plain shell-loop process on a 6.8 kernel; the CUDA-specific restore path may behave differently and was not reachable here.

Independently re-verified in a second session (2026-07-17): CRIU v4.2 was rebuilt from source on the same GPU-less host (git clone --branch v4.2 https://github.com/checkpoint-restore/criu.git, make, the same build dependencies this page already lists), criu check passed, and the same non-CUDA shell-loop dump/restore reproduced the same PID and the same gap-free tick continuation. Repeating the truncated-image case surfaced one operational detail the paragraph above does not yet cover: naively wrapping the restore in the standard timeout command is not sufficient by itself. timeout 20 sudo criu restore --shell-job --restore-detached --images-dir images2 returned once timeout's own tracked child exited, but the actual restored process, which --restore-detached reparents to PID 1, kept spinning at 100% CPU under the reused original PID for over 30 minutes afterward, invisible to timeout because it is no longer that command's descendant. It only stopped after a manual sudo kill -9 targeted at the reused PID directly, the same manual step the paragraph above describes. An orchestrator's kill-on-timeout logic must track and signal the restored target PID itself, not just wait on the criu restore invocation's own exit status.

Record all four transcripts next to the happy-path transcript; a pause/resume feature that has only ever seen the happy path has not been commissioned.

The following runnable model validates the driver gates and the only four legal orchestration transitions. It does not exercise NVIDIA hardware or CRIU.

FEATURE_MIN_DRIVER = {
    "checkpoint": 550,
    "process_tree": 570,
    "gpu_migration": 580,
    "arm": 595,
    "cuda_ipc_handle": 610,
}


def supports(driver: int, feature: str) -> bool:
    if feature not in FEATURE_MIN_DRIVER:
        raise ValueError(f"unknown feature: {feature}")
    return driver >= FEATURE_MIN_DRIVER[feature]


TRANSITIONS = {
    ("running", "lock"): "locked",
    ("locked", "checkpoint"): "checkpointed",
    ("checkpointed", "restore"): "locked",
    ("locked", "unlock"): "running",
}


def transition(state: str, action: str) -> str:
    try:
        return TRANSITIONS[(state, action)]
    except KeyError as exc:
        raise ValueError(f"illegal transition: {state} -> {action}") from exc


assert supports(550, "checkpoint") and not supports(550, "gpu_migration")
assert supports(580, "gpu_migration") and not supports(580, "arm")
assert supports(595, "arm") and supports(610, "cuda_ipc_handle")

state = "running"
for action in ("lock", "checkpoint", "restore", "unlock"):
    state = transition(state, action)
assert state == "running"

for invalid in (("running", "checkpoint"), ("checkpointed", "unlock")):
    try:
        transition(*invalid)
    except ValueError:
        pass
    else:
        raise AssertionError(f"accepted illegal transition: {invalid}")

print("cuda-checkpoint support/state validation: all asserts passed")

Executed output:

cuda-checkpoint support/state validation: all asserts passed

How to maintain it

  • Pin the driver version. The tool is only supported on display driver 550 and newer; confirm the node's driver version before assuming it is available.1
  • Audit for UVM and IPC memory before relying on this in production. Since the tool can leave the process in a bad state on encountering either, scan the target workload's allocator configuration (does it use cudaMallocManaged, or cuMemExportToShareableHandle) as a pre-flight check, not an assumption.
  • Keep NCCL/multi-rank jobs off this path until NVIDIA documents support; treat any observed success on a multi-rank job as accidental, not a supported configuration, per the CRIUgpu findings above.3
  • Track the feature matrix by driver branch. Migration requires 580, ARM requires 595, and cuIpcGetMemHandle support requires 610. Re-check UVM and cuMemExportToShareableHandle() support before each driver rollout.1

How to run it in production

  • Gate every node by display-driver branch and architecture before scheduling a checkpoint or migration. A fleet containing 550, 580, and 595 nodes does not expose one uniform capability set. The executed pre-flight from above is cheap enough to run in a node health check: cuda-checkpoint --get-state --pid 1 exiting 255 with Insufficient driver disqualifies the node outright.
  • Drive lock, checkpoint, CRIU dump/restore, restore, and unlock as an externally persisted state machine. Record the PID, source GPU identity, driver branch, CRIU image path, and last verified --get-state result so an orchestrator restart does not guess which transition completed.
  • Time and export every phase separately (lock wait, suspend duration, dump duration, restore duration, resume duration). Suspend and resume scale with device-memory footprint and dump/restore with image size, so per-phase latency regressions localize the problem (device-to-host copy versus storage I/O) in a way one end-to-end number cannot. Alert on lock timeouts specifically: they mean the workload submits work faster than the orchestrator can quiesce it.
  • Define the recovery action per failed phase before enabling the feature: a failed lock leaves the process running (unlock and walk away); a failed checkpoint with UVM/IPC present leaves the process in an undefined state (kill it and fall back to the framework checkpoint); a failed criu dump leaves a suspended-but-alive process (resume CUDA with --action restore then unlock); a failed criu restore against a clean, complete image typically leaves no process, but the executed corrupted-image test above shows a partially failed restore can instead leave a reparented target-PID process alive and spinning under PID 1 (invisible to a timeout-wrapped restore call), so recovery must poll for and kill that specific PID before assuming the slot is free and restoring again elsewhere; a failed CUDA restore after a successful CRIU restore leaves a host-only process that must be killed and re-restored. Each of these is a different runbook branch; a single generic "retry" is wrong for at least four of them.
  • Manage image lifecycle explicitly: images contain full process memory (including copied device memory), so they are both large and sensitive; store them on capacity-planned storage with the same access controls as a core dump, verify a restore from a copied image before deleting the source process's slot, and garbage-collect images only after the restored process has passed its own application-level health check.
  • Rollback for the feature itself is disabling the pause path and letting workloads run to completion or restart from framework checkpoints; nothing about cuda-checkpoint changes the workload binary, so rollback is purely an orchestrator-side switch plus cleanup of any suspended processes (resume or kill each PID recorded in the state store).
  • Validate migration and process-tree restore on the exact driver/CRIU/container combination before admitting a workload; the reference templates above were not hardware-tested in this knowledge base.

Failure modes

  • Checkpointing a process that uses UVM or IPC memory. The tool does not gracefully decline; it can leave the process in a bad state, since it "does not attempt to keep the process in a good state if an error ... is encountered."1 Pre-flight check for these allocation types instead of discovering the failure at checkpoint time.
  • Checkpointing one rank of an NCCL job and expecting the others to tolerate it. No documented NCCL support exists as of this writing; the peer ranks' communicators have no coordinated way to handle a suspended participant.3
  • Racing new CUDA work against an in-progress checkpoint. Skipping the lock action (or not honoring its --timeout) lets new CUDA submissions start while a checkpoint is being prepared, undermining the consistency CRIU's snapshot is supposed to provide.
  • Assuming every 550+ node supports migration. Migration requires driver 580 or newer. Restore also preserves GPU virtual addresses; test the intended source/target GPU and container combination rather than inferring portability from the base 550 requirement.1
  • Confusing this with a framework checkpoint. A cuda-checkpoint/CRIU image is not a portable, framework-readable model checkpoint; it captures OS-process and CUDA-driver state, not a state dict. Do not use it as a substitute for your training framework's own checkpointing when the goal is resuming training from a specific step.

References

  • NVIDIA/cuda-checkpoint, README.md at commit 00d5cce84c628088d6caa203fc4af40c1538b6f7 (CLI reference, driver feature matrix, limitations). https://github.com/NVIDIA/cuda-checkpoint/blob/00d5cce84c628088d6caa203fc4af40c1538b6f7/README.md
  • NVIDIA Technical Blog, "Checkpointing CUDA Applications with CRIU" (end-to-end workflow, state preserved on suspend/resume). https://developer.nvidia.com/blog/checkpointing-cuda-applications-with-criu/
  • CRIUgpu: Transparent Checkpointing of GPU-Accelerated Workloads, arXiv 2502.16631 (NCCL and multi-node limitations). https://arxiv.org/html/2502.16631v1
  • CRIU project, "GPU Checkpointing." https://www.criu.org/GPU_Checkpointing
  • criu(8) manual page (capabilities: root or CAP_CHECKPOINT_RESTORE minimum plus CAP_SYS_PTRACE/ptrace_scope=0; kernel CONFIG_CHECKPOINT_RESTORE). https://man.archlinux.org/man/criu.8.en
  • checkpoint-restore/criu releases (source builds; latest tag v4.2 as of 2026-07-17). https://github.com/checkpoint-restore/criu/tags
  • Shipped demo workload and end-to-end script at the pinned commit: src/counter.cu, src/example.sh, and the r570/r580/r610 feature examples. https://github.com/NVIDIA/cuda-checkpoint/tree/00d5cce84c628088d6caa203fc4af40c1538b6f7/src

Related: Persistence mode · GPUDirect Storage (GDS) · Checkpoint Recovery / Resume (runbook) · GPU Software Stack and Node Administration · Glossary

  1. NVIDIA/cuda-checkpoint README at commit 00d5cce84c628088d6caa203fc4af40c1538b6f7: base utility support starts at display driver 550; 570 adds NVML, CRIU 4 process trees, Driver API parity, and a separate timed lock; 580 adds GPU migration; 595 adds ARM; 610 adds cuIpcGetMemHandle IPC. The same source says UVM and IPC created with cuMemExportToShareableHandle() remain unsupported, submitted CUDA work completes before checkpoint, and an error can leave the process in a bad state. https://github.com/NVIDIA/cuda-checkpoint/blob/00d5cce84c628088d6caa203fc4af40c1538b6f7/README.md ↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩

  2. NVIDIA Technical Blog, "Checkpointing CUDA Applications with CRIU," published July 2, 2024: documents the driver-550 suspend/resume sequence and CRIU workflow. Its x64, single-process, IPC, and migration statements describe driver 550 at publication time; use the pinned README feature matrix above for current driver milestones. https://developer.nvidia.com/blog/checkpointing-cuda-applications-with-criu/ ↩↩↩

  3. CRIUgpu, arXiv 2502.16631: "at the time of writing, the cuda-checkpoint tool does not support checkpoint/restore operations with NCCL," expected "in a future release of the CUDA driver"; CRIUgpu itself "supports applications running on a single-node with multiple GPUs," not multi-node. https://arxiv.org/html/2502.16631v1 ↩↩↩

  4. criu(8) manual page: CRIU runs as root, or non-root with CAP_CHECKPOINT_RESTORE (kernel 5.9+, the documented minimum; CAP_SYS_ADMIN also suffices) plus CAP_SYS_PTRACE or kernel.yama.ptrace_scope=0 to seize the target; the kernel must be built with CONFIG_CHECKPOINT_RESTORE. https://man.archlinux.org/man/criu.8.en ↩