Markdown

cuda-checkpoint: process-level GPU state checkpoint and restore¶

Scope: NVIDIA's cuda-checkpoint utility and its CRIU integration, which suspend and restore the CUDA state of a running process (streams, contexts, and device memory) at the OS-process level, distinct from a framework's model/optimizer state-dict checkpoint and from GPUDirect Storage's file I/O acceleration.

Reference templates and driver-behavior claims below are transcribed from NVIDIA's own repository and technical blog; they have not been hardware-tested in this knowledge base. Verify the driver version and command syntax on your target node before relying on any of it.

What it is¶

cuda-checkpoint toggles the CUDA state of a running process, identified by PID, between running and checkpointed. NVIDIA's own description: a running-to-checkpointed transition is a suspend, the reverse is a resume.¹ On suspend, the tool "completes any submitted CUDA work, copies device memory to host allocations, and releases GPU resources," which is what allows CRIU (Checkpoint/Restore In Userspace) to snapshot the now GPU-resource-free process at the OS level. On resume, it "copies device memory back to the GPU," restores "GPU memory mappings ... at their original addresses," and restores CUDA objects such as streams and contexts.²

This is a different layer from two things it is easy to conflate it with:

Framework checkpointing (a PyTorch state dict, a DeepSpeed/FSDP sharded checkpoint) serializes model weights and optimizer state that you explicitly chose to save, at a point in training you chose, for the purpose of resuming training from a known step. cuda-checkpoint instead freezes and thaws the CUDA driver-level state of an OS process, including CUDA objects that never appear in a state dict (streams, contexts, in-flight kernels), for the purpose of suspending and resuming the process itself, whether or not it is doing training.
GPUDirect Storage (GDS) (GPUDirect Storage) accelerates file I/O between storage and GPU memory; it has no relationship to process suspension. A cuda-checkpoint based checkpoint image still needs to be written to and read from disk by CRIU using ordinary host-side I/O, GDS is not part of this path.

Why it matters¶

Without a driver-level suspend/resume primitive, checkpointing a CUDA process with CRIU alone does not work: CRIU has no way to quiesce in-flight GPU work or release the GPU resources a process holds, so it cannot produce a consistent snapshot. cuda-checkpoint exists specifically to bridge that gap, turning a CUDA process into something CRIU's ordinary process-snapshot machinery can handle.² The practical payoffs this unlocks: pausing a long-running job to free a GPU for something more urgent without killing and re-queuing it, migrating a process to different node hardware, and near-zero cold-start restarts for services that would otherwise re-run expensive CUDA/driver initialization on every launch. As of this writing, official documentation does not publish checkpoint-image size or suspend/resume latency numbers, size your expectations from the amount of device memory that must be copied to host memory (and back), not from an assumed constant cost.

When to use it (and when not)¶

Use it:

To pause and later resume a single CUDA process (one PID) without losing its in-flight CUDA state, for example to free a GPU for a higher-priority job.
As the mechanism underneath a higher-level snapshot/migrate feature (a scheduler or platform that wants to relocate or preempt a GPU process) rather than as an end-user checkpoint format.
On driver 550 or newer, x86-64 only.¹

Do not reach for it when:

The job is a multi-rank, NCCL-coordinated distributed job. As of the CRIUgpu research writeup, "at the time of writing, the cuda-checkpoint tool does not support checkpoint/restore operations with NCCL," and even CRIUgpu's own extension "supports applications running on a single-node with multiple GPUs" but not multi-node coordination.³ Suspending one rank of an NCCL job gives the driver no way to inform the peer ranks' communicators that a participant vanished, so the other ranks' next collective call has nothing coordinating its behavior around the gap. Treat any multi-process/multi-node NCCL job as out of scope until NVIDIA documents support.
You want to resume training from a step boundary. That is what framework checkpointing is for; cuda-checkpoint does not know about your model, optimizer, or dataloader position; it only knows about the CUDA driver state of the process.
The process uses Unified Virtual Memory (UVM) or IPC memory created with cuMemExportToShareableHandle(). Both are explicitly unsupported: the tool "does not support UVM memory or IPC memory," and it "does not attempt to keep the process in a good state" if it encounters either, i.e. it can fail destructively rather than gracefully declining.¹
You need GPU migration across different physical GPUs mid-checkpoint. Not supported as of driver 550; NVIDIA states this and UVM/IPC support are expected in future driver releases, not the current one.²

Architecture¶

flowchart TB
  A["Running CUDA process (PID)"] -->|"cuda-checkpoint --action lock"| B["Lock: block new CUDA work"]
  B -->|"cuda-checkpoint --action checkpoint"| C["Suspend: finish in-flight work,\ncopy device memory to host,\nrelease GPU resources"]
  C --> D["criu dump: OS-level process snapshot\n(now GPU-resource-free)"]
  D --> E["Checkpoint image on disk"]
  E -->|"criu restore"| F["Process restored, CUDA still checkpointed"]
  F -->|"cuda-checkpoint --action restore"| G["Resume: copy memory back to GPU,\nrestore mappings, streams, contexts"]
  G -->|"cuda-checkpoint --action unlock"| H["Running CUDA process, resumed"]

How to use it¶

The utility exposes discrete actions rather than one combined verb, per the current cuda-checkpoint --help output:¹

Operations:
--get-state --pid <pid>
        Prints the current checkpoint state of the process specified by <pid>

--action lock | checkpoint | restore | unlock --pid <pid> [--timeout <ms>]
        Performs the specified action on <pid>. For the lock action a
        timeout can be provided; the lock operation waits up to <ms>
        milliseconds for the operation to succeed.

--toggle --pid <pid>
        Toggles the CUDA state in the specified process between the
        running and checkpointed states

--get-restore-tid --pid <pid>
        Retrieves the CUDA restore thread ID of the process specified by <pid>

--toggle is the simple case (suspend if running, resume if checkpointed) that NVIDIA's own CRIU walkthrough uses end to end:²

# 1. Suspend the target process's CUDA state (device memory -> host, GPU released)
cuda-checkpoint --toggle --pid "$PID"

# 2. Snapshot the now GPU-resource-free process with CRIU
criu dump --shell-job --images-dir demo --tree "$PID"

# 3. Restore the process from the CRIU image (still CUDA-checkpointed)
criu restore --shell-job --restore-detached --images-dir demo

# 4. Resume the restored process's CUDA state (host -> device memory, streams/contexts back)
cuda-checkpoint --toggle --pid "$PID"

--get-state lets an orchestrator poll whether a given PID is currently running or checkpointed before deciding whether to act, and --action lock (with --timeout) exists to close a race window: it blocks new CUDA work from starting in the target process while a checkpoint is being prepared, so a resumed job does not race a request that started after the checkpoint began.¹

How to develop with it¶

Building an orchestration layer on top of cuda-checkpoint (a scheduler plugin, a service-level pause/resume feature) means driving the four actions directly instead of --toggle, so you can insert your own error handling between steps:

# Explicit four-step form, useful when you need to react to failure at each stage
cuda-checkpoint --action lock --pid "$PID" --timeout 5000   # block new CUDA work, wait up to 5s
cuda-checkpoint --action checkpoint --pid "$PID"             # suspend CUDA state
# ... criu dump ...
cuda-checkpoint --action restore --pid "$PID"                # resume CUDA state after criu restore
cuda-checkpoint --action unlock --pid "$PID"                 # allow new CUDA work again

Before wiring this into anything that touches a real GPU workload, check --get-state and gate on it: an orchestrator that calls --action checkpoint on a PID that is not actually running CUDA work, or calls it twice without an intervening restore, is asking the tool to do something its own documentation does not promise a defined result for. Treat every state transition as something to verify with --get-state, not something to assume succeeded because the command returned.

How to maintain it¶

Pin the driver version. The tool is only supported on display driver 550 and newer; confirm the node's driver version before assuming it is available.¹
Audit for UVM and IPC memory before relying on this in production. Since the tool can leave the process in a bad state on encountering either, scan the target workload's allocator configuration (does it use cudaMallocManaged, or cuMemExportToShareableHandle) as a pre-flight check, not an assumption.
Keep NCCL/multi-rank jobs off this path until NVIDIA documents support; treat any observed success on a multi-rank job as accidental, not a supported configuration, per the CRIUgpu findings above.³
Track driver release notes for UVM, IPC, and multi-GPU migration support, all called out as work NVIDIA has said is coming in a future driver, not the current one.²

Failure modes¶

Checkpointing a process that uses UVM or IPC memory. The tool does not gracefully decline; it can leave the process in a bad state, since it "does not attempt to keep the process in a good state if an error ... is encountered."¹ Pre-flight check for these allocation types instead of discovering the failure at checkpoint time.
Checkpointing one rank of an NCCL job and expecting the others to tolerate it. No documented NCCL support exists as of this writing; the peer ranks' communicators have no coordinated way to handle a suspended participant.³
Racing new CUDA work against an in-progress checkpoint. Skipping the lock action (or not honoring its --timeout) lets new CUDA submissions start while a checkpoint is being prepared, undermining the consistency CRIU's snapshot is supposed to provide.
Assuming GPU migration works. As of driver 550 the tool does not support migrating the restored process to different GPU hardware; restore is documented as restoring "at their original addresses" on the same class of device, not an arbitrary target.²
Confusing this with a framework checkpoint. A cuda-checkpoint/CRIU image is not a portable, framework-readable model checkpoint; it captures OS-process and CUDA-driver state, not a state dict. Do not use it as a substitute for your training framework's own checkpointing when the goal is resuming training from a specific step.

References¶

NVIDIA/cuda-checkpoint, README.md (CLI reference, driver requirement, limitations). https://github.com/NVIDIA/cuda-checkpoint/blob/main/README.md
NVIDIA Technical Blog, "Checkpointing CUDA Applications with CRIU" (end-to-end workflow, state preserved on suspend/resume). https://developer.nvidia.com/blog/checkpointing-cuda-applications-with-criu/
CRIUgpu: Transparent Checkpointing of GPU-Accelerated Workloads, arXiv 2502.16631 (NCCL and multi-node limitations). https://arxiv.org/html/2502.16631v1
CRIU project, "GPU Checkpointing." https://www.criu.org/GPU_Checkpointing

NVIDIA/cuda-checkpoint README: --get-state, --action lock|checkpoint|restore|unlock (with --timeout on lock), --toggle, --get-restore-tid; supported on "display driver version 550 and higher"; "does not support UVM memory or IPC memory created with cuMemExportToShareableHandle()"; "waits for already-submitted CUDA work to finish before completing a checkpoint"; "does not attempt to keep the process in a good state if an error ... is encountered." https://github.com/NVIDIA/cuda-checkpoint/blob/main/README.md ↩↩↩↩↩↩↩
NVIDIA Technical Blog, "Checkpointing CUDA Applications with CRIU": suspend "completes any submitted CUDA work, copies device memory to host allocations, and releases GPU resources"; resume "copies device memory back to the GPU" and restores mappings "at their original addresses" plus streams and contexts; example workflow cuda-checkpoint --toggle, criu dump --shell-job, criu restore --shell-job --restore-detached, cuda-checkpoint --toggle; x64 only, single-process, no UVM/IPC, no GPU migration as of driver 550, with these called out as addressed in a future driver release. https://developer.nvidia.com/blog/checkpointing-cuda-applications-with-criu/ ↩↩↩↩↩↩
CRIUgpu, arXiv 2502.16631: "at the time of writing, the cuda-checkpoint tool does not support checkpoint/restore operations with NCCL," expected "in a future release of the CUDA driver"; CRIUgpu itself "supports applications running on a single-node with multiple GPUs," not multi-node. https://arxiv.org/html/2502.16631v1 ↩↩↩